Old 5th June 2013, 09:40   #1
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
sc_serv using 100% CPU cycles

Machine is Ubuntu 12.04.2 server, 2 GB RAM, 2 CPU Opteron 6128, dual core 2 GHz, 50 Mbps up / 50 Mbps down internet line

Stream server starts and runs along fine, sc_serv at like 9%, sc_trans at 7%, then at some point sc_serv shoots up to 100% CPU utilization ( as seen using top in the terminal) and at that point no new listeners can connect and the Shoutcast YP directory listing for the stream goes away, presumably because YP can't connect either 'cause the CPU is too busy

Not seeing anything in the log file that stands out as causing this; one of the problems is GETTING the log file while the CPU is at 100% with sc_trans - just establishing an SSH session takes minutes.

Yesterday, I did see multiple attempts to log in as admin with failed password, if these were coming at the server fast enough I guess it could clog things up. A denial-of-service attack? But I'm not seeing those in logs today, and around 11 PM server time the CPU went to 100% for sc_serv.


Any suggestions? What can make sc_serv take up 100% of the cpu cycles?

Normally, with ~300 listeners, sc_serv is 10% or less. 500 max listeners, so I don't think it can be listener-count related. (stream is 48 kbs)
milosz is offline   Reply With Quote
Old 5th June 2013, 15:45   #2
tyral
Junior Member
 
Join Date: Jun 2013
Posts: 7
Is there any way you can have a terminal on the box tailing the logfile so you can see when this happens? Additionally, if you can have an open session, you may be able to strace the process to see where it's consuming that much CPU time.

Without debugging information I don't think it'll be much help, an strace may point out the issue though.

http://linux.die.net/man/1/strace

you'll likely want to use strace -vvvvvvvvv -p $PID
tyral is offline   Reply With Quote
Old 5th June 2013, 18:33   #3
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Yeah I've got two SSH sessions open right now to the machine, one is tailing the log file, the other showing CPU % using top. Of course this won't help for temporary excursions by sc_trans to 100% CPU, but so far my experience is that when CPU % goes crazy it stays that way. Maybe this will give me a clue.

I don't think the problem is sc_serv itself, I think some connection, connection attempt(s), login attempt(s) or other activity coming in to sc_serv from the Internet is causing the issue. I think that WOULD show in the logfile.

Of course it could somehow be related to something wacky in the MP3 stream or metadata that sc_trans is sending sc_serv that drives sc_serv to 100% CPU utilization, I think this also might be visible in the logfile.

So anyway I'll keep these SSH sessions open and maybe I'll see something. So far, running 8 hours with no issues that I can see. sc_serv and sc_trans each running around 7~9%
milosz is offline   Reply With Quote
Old 6th June 2013, 08:09   #4
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
OK I figured it out. I left SSH terminal window open where I started sc_serv, and as more and more listeners connected, the terminal filled with.....

"msg:[MICROSERVER] Could not call accept() on socket because Too many open files"

I hit a limit in my Linux build. Doing ulimit -a shows me

open files (-n) 1024

So, I did ulimit -n 10240; now, and ulimit -a shows me

open files (-n) 10240

So, 10 times the limit for open files. I'll try that, see if it works.
milosz is offline   Reply With Quote
Old 6th June 2013, 14:55   #5
tyral
Junior Member
 
Join Date: Jun 2013
Posts: 7
The CPU usage could be attributed to that, since it would cause a FS lockup due tot he kernel being unable to load things, and then eventually a Kernel panic.

Hope that fixes it for you 8D
tyral is offline   Reply With Quote
Old 6th June 2013, 22:54   #6
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
....so far, so good.

I think a general thing to take away from this for anyone running Shoutcast server on Linux- and ESPECIALLY those on hosted virtual servers- is to check the max number of open files allowed, and increase it to some sensible figure if it seems low.
milosz is offline   Reply With Quote
Old 7th June 2013, 09:33   #7
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Well... sc_serv EXCEEDED 10,240 open files .... HOW MANY OPEN FILES DOES sc_need???

I set it to 102,400 - maybe that is enough???
milosz is offline   Reply With Quote
Old 8th June 2013, 21:09   #8
tyral
Junior Member
 
Join Date: Jun 2013
Posts: 7
I believe it depends on the number of people streaming. One thing you can do to see why Shoutcast is using so many files (or what files it's using) is to do this:

lsof -c sc_serv

or whatever your shoutcast binary name is. Right now the only thing I see that would attribute to usage would be pipes, though I'm not seeing many in use on mine.

Hope this helps.
tyral is offline   Reply With Quote
Old 9th June 2013, 00:03   #9
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Even with 1,024,000 open files permitted I am still running into 100% CPU on sc_serv

I have to keep looking to find the cause....

Memory use is stable, it's not a leak.
milosz is offline   Reply With Quote
Old 10th June 2013, 12:27   #10
DrO
 
Join Date: Sep 2003
Posts: 27,873
rather than me re-typing things (note: 3 threads effectively about the same thing isn't a good way to get a consistent official answer as i'm now having to check back in the other threads), http://forums.winamp.com/showpost.ph...81&postcount=5 basically explains it's a known v2 DNAS issue at high listener numbers due to how it's currently coded (in relation to some other bugs).
DrO is offline   Reply With Quote
Reply
Go Back   Winamp & Shoutcast Forums > Shoutcast > Shoutcast Technical Support

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump