Go Back   Winamp & Shoutcast Forums > Shoutcast > Shoutcast Discussions

Closed Thread
Thread Tools Search this Thread Display Modes
Old 20th November 2013, 10:44   #121
DrO
 
Join Date: Sep 2003
Posts: 27,873
i don't know if it's DNAS or not. it needs a specific DNAS build with extra debugging to be able to better determine what might be happening. as the more this goes on, the more i think it's the setup but i don't know as i don't know what errors are actually being thrown internally to know who / what / why it's limiting you to 250 (which is at complete odds with the issue everyone else had been having with the earlier 2.0 builds).
DrO is offline  
Old 20th November 2013, 11:03   #122
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Yeah, I don't know how to get a handle on exactly what is running afoul of what, where. But I am glad that if set the limit to 250 it runs along quite nicely, and apparently the various aggregators / directories which are used for things like "tabletop WiFi radios" are coping OK with having a primary and when full the secondary server. Obviously the listen.pls files I've got have both servers, but getting the directories / apps / aggregators to follow along is sometimes a pain and sometimes just a complete mystery. I find that almost no one is using Winamp on a PC, most of my listeners seem to be using some kind of Android device; iTunes is also popular on many platforms, but iTunes is no problem... then there are listeners using Nokia Symbian devices and NokiaRadio and on and on....

And, by the way, the instance of sc_serv has TWO streams (two separate stations, from two instances of sc_trans) one limited to 250, the other to 50, and so the magic number is probably 300 in fact, not 250.

The second instance of sc_serv carries only the "big" stream, the other stream has few fans so it's single instance of sc_serv limted to 50 listeners has never filled up. Typical load is 20-30.

I often forget about that other station, so few people listen (and fewer still send money!)
milosz is offline  
Old 20th November 2013, 12:21   #123
Cande
Junior Member
 
Join Date: Nov 2013
Posts: 15
Milosz,
during the last weeks, I've been encountering a lot of stability problems with sc_serv.
The service, which streams 15..20 different external sources (not using sc_trans) kept on crashing (mainly due to segfaults) several times per day, until I discovered version 2.2 of DNAS.
After the update, everything started to be much more stable, so the improvements and fixes introduced with the new DNAS are indeed effective, although some halts were still happening.

By studying the DNAS' logfiles I discovered weird entries, that I suspect are kind of "attacks": some IPs kept on connecting and disconnecting hundreds of times in seconds and some others even tried to use the server to start their own stream (SRC bad password errors).
To test if this was the cause of the crashes I started to ban the attackers' IPs (and in some cases entire netmasks since they were changing IP) via linux firewall and… boom, stability improved noticeably!

My last change to the installation was to rise the max open files limit to 4096, but I can confirm that during a test with "nofile limit" set to 65536 sc_serv crashed, so the standard 1024 value may be enough for DNAS to operate correctly (real time monitoring during an average load of about 200 connected clients shows 800..900 open files).

I have set a global limit of 500 users, but never go over 300 in the last days, so I cannot confirm your problem.

Hope that helps…

Cheers!
Cande
Cande is offline  
Old 20th November 2013, 12:34   #124
DrO
 
Join Date: Sep 2003
Posts: 27,873
Quote:
Originally Posted by Cande View Post
By studying the DNAS' logfiles I discovered weird entries, that I suspect are kind of "attacks": some IPs kept on connecting and disconnecting hundreds of times in seconds and some others even tried to use the server to start their own stream (SRC bad password errors).
To test if this was the cause of the crashes I started to ban the attackers' IPs (and in some cases entire netmasks since they were changing IP) via linux firewall and… boom, stability improved noticeably!
which is already noted in the known issues -> http://forums.winamp.com/showthread.php?t=373139#known

under high loads, i know what the solution is and have most of the code written to do what's needed, but under testing it was not 100% stable and so was not included as part of the 2.2 release (as a still running DNAS which just stops processing connection requests is not useful - at least that's what it was doing on some test setups we had). however, as the vast majority of stations barely go over a hundred concurrent listeners, having something which is stable for the majority was the primary aim for the 2.2 release (and to ensure we've got something that in terms of general functionality is able to replace the v1.x DNAS at last as supporting v1.x listings is becoming a pain in a number of areas).
DrO is offline  
Old 20th November 2013, 12:52   #125
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Yeah I see things trying to connect then closing 3 seconds later then connecting again and closing after 8 seconds.... I think this is actual client behavior in some cases however. Occasionally I have seen what looks like a "storm" of connection/disconnections coming from a single IP, but not lately. The most common thing I see connecting every few seconds is the Shoutcast Metadata Puller. Guess I don't want to ban THAT! :-)

I don't know that I could prove it, but I think I see fewer clients connecting/staying connected a short time/ disconnecting / connecting again.... than I used to see with the previous release of DNAS 2.

I also see random sources trying to connect from time to time, and failing password. I assume these ARE some kind of attack.

I have to say that, in general, things seem more stable with this newer release as long as I keep users-per-instance of sc_serv below 300. I mean, the older sc_serv couldn't handle more than 300 either but it also had other things it would sometimes do, like segfault, which this new one just does not do.

Also, the older version used to sometimes cause init to go kinda wacky, burning up CPU cycles. Haven't seen that with this new one. (I have init start my instance of sc_serv, 'cause the old one used to fall over often enough that having it automatically re-start was a good thing. )
milosz is offline  
Old 20th November 2013, 14:07   #126
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
DrO.

I know I keep talking about the useucy=0 but I would like an understanding and actually know whats happening so all your explanations are taken and trying to be stored as knowledge.

previously as you know we spoke about useucy=0 and the 480 error. Well my stream has been up and running along with live shows for a few days now on the new 2.2 shoutcast.

this morning when I woke up I noticed very low listeners and then I noticed I wasnt able to connect to the stream. So I looked at the DNAS and saw that my stream was no longer public.

Did the useucy=0 cause this too? If not what could have caused the stream to be public for so long and then not be public?

Bryon Stout is offline  
Old 20th November 2013, 14:11   #127
DrO
 
Join Date: Sep 2003
Posts: 27,873
having set useicy=0 should not have done anything to cause that - the only reason for setting that is to invert what the DNAS is incorrectly trying to do.

as you're running it under a control panel, maybe that's done something which has messed up the access (which i'm pretty sure you've had happen previously), but unless there's something specific in the logs to work against, i've no explanation for why it just randomly happened as nothing has been changed on the YP side for the last few days.
DrO is offline  
Old 20th November 2013, 14:26   #128
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
I have yet set useicy=0

I havent gotten around to it.
Bryon Stout is offline  
Old 20th November 2013, 14:34   #129
DrO
 
Join Date: Sep 2003
Posts: 27,873
the prior post seemed to imply that you had set it. not having set it then explains the failure.
DrO is offline  
Old 20th November 2013, 14:37   #130
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
sorry.. I should take more time and make sure my explanations are clear.

Thanks for the info.
Bryon Stout is offline  
Old 20th November 2013, 19:30   #131
Cande
Junior Member
 
Join Date: Nov 2013
Posts: 15
Quote:
Originally Posted by DrO View Post
which is already noted in the known issues -> http://forums.winamp.com/showthread.php?t=373139#known
Thanks for pointing out.

Please note that, right before the crashes I could closely monitor, I did never notice a high number of concurrent connections (Max Listeners never exeeded 250).

On the contrary, there were multiple and continuous connection attempts which disconnected after 0..2 seconds.

As I repeat, stability greatly improved by updating to version 2.2, but also the way sc_serv crashes changed.
Version 2.0 usually exited after a segfault (as seen in server's syslog), while 2.2 hangs/freezes and gives no reasons for that.

Thanks a lot for your great support!
Cande
Cande is offline  
Old 20th November 2013, 19:36   #132
DrO
 
Join Date: Sep 2003
Posts: 27,873
oh ok, wasn't aware it was just hanging (unless i missed that from the posts above) and assumed it was segfaults that were happening. as that's interesting as i only saw the hangs with the re-done networking core and not the current implementation so that gives me a few other ideas on things to check.
DrO is offline  
Old 21st November 2013, 19:17   #133
nwgator
Member
 
Join Date: Oct 2008
Location: Jacksonville, FL
Posts: 72
Hi DrO,

I saw the recent announcement of Winamp's coming closure, does this affect the SHOUTcast YP as well?
nwgator is offline  
Old 22nd November 2013, 17:17   #134
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
Does this also affect DrO releasing 2.2.1?
Bryon Stout is offline  
Old 24th November 2013, 17:22   #135
Cande
Junior Member
 
Join Date: Nov 2013
Posts: 15
I discovered something "new" (for me at least), that may help you...

Since Friday afternoon, my server started to be VERY unstable, with sc_serv freezing/halting every 5 minutes or so.
Ubuntu's and sc_serv's logfiles didn't give any hint, so I tried everything, from reinstalling Shoutcast DNAS, to replacing config files, changing install location, modifying resources and so on, but... nothing helped!

What I realized is that for sure it was not a "DST client" side problem, because the listeners count never reached more than 10..20 simultaneous connections between freezes.

Therefore, I checked the "SRC clients" and found that one radio station (running on a Windows XP virtual machine) was partially blocked due to "virtual memory full" error.
Basically, connection to the server was established and a "no data flow" (or maybe corrupted data) kept on feeding sc_serv, which shortly after stopped working.

After fixing the virtual machine, sc_serv is running again like a charm!

So, it looks like data transfer problems/errors on the source side cause sc_serv to become very unstable, much more than destination clients' (bad/repeated) requests management issues described in my previous posts.

Cheers!
Cande
Cande is offline  
Old 27th November 2013, 09:14   #136
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
Hey DrO.

Trying to deal with the useicy=0 issue now. Which config file is that setting in? Cant seem to find it.
Bryon Stout is offline  
Old 28th November 2013, 15:00   #137
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
Am I blind? I dont see "useicy" anywhere




; ShoutCast DNAS 2.0 configuration file
; Generated by Centova Cast
;
;
; LOGGING OPTIONS
;
; should we do any logging Default = 1
log=1
; log file name Default = c:\sc_serv.log or /tmp/sc_serv.log
logfile=var/log/error.log
; 1 to log to console Default = 1
screenlog=0
; 1 to enable w3c logging Default = 1
w3cenable=1
; name of w3c log file Default = sc_w3c.log
w3clog=var/log/access.log

; 1 to activate debug logging for web client connections Default = 0
webclientdebug=0
; 1 to activate debug logging for yp connections Default = 0
yp1debug=0
; 1 to activate debug logging for yp2 connections Default = 0
yp2debug=0
; 1 to activate debug logging for shoutcast source connections Default = 0
shoutcastsourcedebug=0
; 1 to activate debug logging for shoutcast 2 source connections Default = 0
uvox2sourcedebug =0
; 1 to activate debug logging for shoutcast streaming clients Default = 0
shoutcast1clientdebug=0
; 1 to activate debug logging for shoutcast 2 streaming clients Default = 0
shoutcast2clientdebug =0
; 1 to activate shoutcast relay debug logging Default = 0
relayshoutcastdebug =0
; 1 to activate shoutcast 2 relay debug logging Default = 0
relayuvoxdebug =0
; 1 to activate common relay code debugging Default = 0
relaydebug =0
; 1 to activate common streaming code debugging Default = 0
streamdatadebug =0
; 1 to activate debug logging for http style requests Default = 0
httpstyledebug =0
; 1 to activate debug logging for statistics Default = 0
statsdebug =0
; 1 to activate common debug logging for server activity Default = 0
microserverdebug=0
; 1 to activate debug logging for thread manager Default = 0
threadrunnerdebug=0

;
; FLASH SECURITY
;
; name of file that has flash crossdomain policies Default = crossdomain.xml
flashpolicyfile=var/lib/crossdomain.xml

;
; BUFFERS
;
; 0 = fixed 1 = adaptive Default = 0
buffertype=0
; buffer size in bytes if buffer is fixed Default = 1024 * 1024
fixedbuffersize=1048576
; buffer size in seconds if buffer is adaptive Default = 1
adaptivebuffersize=1
; never bigger than this (in bytes) no matter what Default = 16777216
bufferhardlimit=16777216

;
; SHOUTCAST 1 ONLY
;
; metadata transmission interval in bytes. Default = 8192
metainterval=8192

;
; SHOUTCAST 2 ONLY
;
; key used to obfuscate initial handshaking with source Default = foobar
uvoxcipherkey=

;
; NETWORKING
;
; 1 to do reverse DNS look ups on incoming IP addresses Default = 0
namelookups=0
; port that clients and sources connect to (note, shoutcast 1 sources only connect to portbase+1) Default = 8000
portbase=8006
; how long before an idle source is dumped (in seconds). Zero means no timeout Default = 30
autodumpsourcetime=30
; maximum size of an HTTP header line Default = 2048
maxheaderlinesize=2048
; max headers lines in HTTP style exchange Default = 100
maxheaderlinecount=100
; password for broadcaster to connect
password=xxxxxxxxx
; administrator password
adminpassword=xxxxxxxxx

;
; YP
;
; 1 to use yp2 protocol Default = 0
yp2=1
; address of yp server Default = yp.shoutcast.com
ypaddr=yp.shoutcast.com
; port of yp server Default = 80
ypport=80
; path to yp2 services on server Default = /yp2
yppath=/yp2
; Timeout, in seconds, for yp requests Default = 60
yptimeout=60
; Max times a yp request will be retried Default = 10
ypmaxretries=10
; yp reporting must happen at least this fast (in seconds) Default = 300
ypreportinterval=300
; yp reporting cannot happen faster than this (in seconds) Default = 10
ypminreportinterval=10
; Can be used to override "public" flag from source. (default - use flag from source; always; never) Default = default
publicserver=always

;
; BANNING
;
; file in which to store banned IP addresses Default = sc_serv.ban
banfile=etc/sc_serv.ban
; 1 to rewrite banfile on application exit Default = 1
savebanlistonexit=1

;
; RIP (reserved IP)
;
; file in which to store list of reserved IP addresses Default = sc_serv.rip
ripfile=etc/sc_serv.rip
; 1 to rewrite ripfile on application exit Default = 1
saveriplistonexit=1
; 1 to only allow connections from reserved IP addresses Default = 0
riponly=0

;
; CLIENT BEHAVIOR
;
; max clients Default = 32
maxuser=1000
; max time in minutes a client can listen. 0 means no limit Default = 0
listenertime=0
; 1 to disconnect clients if source disconnects Default = 0
autodumpusers=0
; server side binding address for sources
srcip=198.xxxxxxxx
; server side binding address for clients
dstip=198.xxxxxxxxx
; modifies icy-name
titleformat=%s
; modifies icy-url
urlformat=%s

;
; SPECIAL FILE HANDLING
;
; file to play when a client first connects
introfile=
; file to play if source disconnects
backupfile=
; place to store intro and backup files uploaded by sc_trans Default = /tmp/ (*nix only)
specialfiletmpdir=var/tmp/
; maximum size, in bytes, of backup or intro files Default = 30000000
maxspecialfilesize=30000000

;
; MISC
;
; 1 to have sc_serv rewrite the config file on exit Default = 0
configrewrite=0
; max song history to preserive Default = 10
songhistory=10
; non-zero overrides cpu count calculated by software. Default = 0
cpucount=0
; substitution string for file names Default = $
unique=$
; include file
; include=

;
; RELAYING
;
; 1 to allow a relay to connect Default = 1
allowrelay=1
; 1 to allow relays to list themselves in yp Default = 1
allowpublicrelay=1
; seconds to reconnect on relay failure Default = 30
relayreconnecttime=30
; max times we can redirect when relaying Default = 5
maxhttpredirects=5
; legacy, use stream configs instead
;relayport
; legacy, use stream configs instead
;relayserver

;
; STREAM CONFIGS
;
; 1 to requires stream configs for any sources Default = 0
requirestreamconfigs=0

;---- MUTLI -------
; ID of stream
;streamid=
; authorization hash for yp2 registration
;streamauthhash=
; path clients use to access stream
;streampath=
; url of source to relay (if this is a relay)
;streamrelayurl=
streampath_1=/stream
streamid_1=1
streamrelayurl_1=
streamauthhash_1=WRNbDk6Ixxxxxxxxxxxx
Bryon Stout is offline  
Old 28th November 2013, 15:01   #138
DrO
 
Join Date: Sep 2003
Posts: 27,873
you'd just add it to the file. Centova doesn't know about the option and the general rule is options which need to be left at the default aren't in the config file. though it won't matter much once i finish off the 2.2.1 update i'm working on at the moment.
DrO is offline  
Old 28th November 2013, 15:21   #139
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
" general rule is options which need to be left at the default aren't in the config file."

ahh thanks for this info.


Well then screw it. Ill just deal with it and just wait for you to be done.

As always, thanks for everything.
Bryon Stout is offline  
Old 28th November 2013, 15:32   #140
DrO
 
Join Date: Sep 2003
Posts: 27,873
aim is there will be a fixed build available tomorrow.
DrO is offline  
Old 28th November 2013, 16:08   #141
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
Thats great news.

Cheers DrO.
Bryon Stout is offline  
Old 29th November 2013, 17:45   #142
DrO
 
Join Date: Sep 2003
Posts: 27,873
closing this now so anything about v2.2.1 can be discussed in http://forums.winamp.com/showthread.php?t=373139

for milosz's reported issue, am going to follow-up on that with a different build.
DrO is offline  
Closed Thread
Go Back   Winamp & Shoutcast Forums > Shoutcast > Shoutcast Discussions

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump