Strangest "audio glitches" with sc_trans

[djsuck]

Hi everyone,

I'm having a problem with sc_trans (I guess) and it's really killing me.

Here is my Setup:
Server:
Core i7-4770 QuadCore
32GB RAM
2x2TB SATA 6Gb/s 7200 rpm
Guaranteed Bandwidth: 200 Mbit/s

Bare metal I installed a Citrix XenServer 6.02

The first virtual machine (where Shoutcast-Server/sc_serv and sc_trans runs on)
is a Debian 7.2 (fully updated) with 8GB of RAM.

The second VM is a Win7 with 8GB of Ram, Winamp 5.65 and the DSP-Plugin (2.3.3)

The third VM is a Sophos UTM Firewall, which is "in front" of everything.

The Win7-VM streams the audio signal to sc_trans on the Debian-machine
and connects to it as a "DJ" over Port 8505.

sc_trans is set up to relay the stream to two seperate instances of sc_serv on the same machine (192K AAC and 32K AAC).

So far everything works like a charm:
- Playlist (on the debian machine) works
- Different DJs can connect (priorities work)
- pretty much everything works

BUT:
After ~ 20-40 minutes of continuous streaming from the Win7-VM to sc_trans, "audio glitches" occur.
The sound "skips" for like a tenth of a second every couple of seconds and "blips" and "blobs" can be heard.

I spent > 50 hours troubleshooting these audio glitches, but I can't solve that problem.

Here is, what's NOT causing it:
- Hardware/Virtualisation (Had the same issues on a bare metal Debian machine)
- Firewall (The traffic from Winamp over sc_trans to sc_serv is internally without any Firewalls between)
- sc_serv (When I stream directly to sc_serv, everything is fine)
- DSP-Plugin (I tried 2.2.1, 2.3.2 & 2.3.3, also the stream written to disk sounds fine)
- OS (Same issues on two different Debian 5.0 machines)
- Encoder (I tried 192K, 128K and 96K both on mp3 and aac)
- Defective Installation (I (re-) installed the whole infrastructure 3 times)
- User (tried root and my shoutcast-User on every installation)
- Ports (Same Issues on port 8000, 8010, 8080...)
- Streaming Win7-machine (Same issues with other sources)

In my opinion, that pretty much only leaves sc_trans, right?
When the stream is faulty and I kill the sc_trans process and start it right afterwards
everything is fine... for another 20-40 minutes.

Here are my configs (URLs, PWs etc. obfuscated):

sc_trans.conf

PHP Code:

logfile=logs/sc_trans.log
calendarrewrite=0

encoder_1=aac
bitrate_1=192000
encoder_2=aac
bitrate_2=32000

outprotocol_1=3
serverip_1=127.0.0.1

outprotocol_2=3
serverip_2=127.0.0.1


serverport_1=8080
serverport_2=8000

password_1=password
password_2=password

streamid_1=1
streamid_2=1

endpointname_1=/Endpoint1
endpointname_2=/Endpoint2_64K

streamtitle=Cool title
streamurl=http://www.example.com
genre=House

adminport=7999
adminuser=admin
adminpassword=admin_password

unlockkeyname=My Name
unlockkeycode=XXXXXXXXXXXXXXXXXXXXXXXX

djport=8500
djport2=8505

djlogin_1=dj1
djpassword_1=pw1
djpriority_1=10

djlogin_2=dj2
djpassword_2=pw2
djpriority_2=5

djlogin_3=dj3
djpassword_3=pw3
djpriority_3=3

djlogin_4=dj4
djpassword_4=pw4
djpriority_4=1

calendarfile=calendar/dj_calendar.xml
playlistfile=playlists/playlist.lst 


1 of the sc_serv.conf:

PHP Code:

MaxUser=150
logfile=logs/sc_serv.log
w3clog=logs/sc_w3c.log
banfile=control/sc_serv.ban
ripfile=control/sc_serv.rip

publicserver=always
streamauthhash_1=XXXXXXXXXXXXXXXXXXXXXXXXXXXX

portbase=8080

password=password

adminpassword=admin_password

streamid=1
streampath=/streamname.mp3 


Can you guys PLEASE help me with this issue.
Has anybody experienced this kind of audio glitches an got rid of them?

Thank you so much in advance!

- Benjamin -

[DrO]

anything is possible when using beta software. i don't have anything to offer as a solution other than to see if an updated build of sc_trans which may come within the next few weeks (though there's no guarantees it will happen) and see if that helps with the issue as the AAC libraries being used have seen a number of resource optimisations made to them since the the last sc_trans build was released which may help (or may not, i just don't know).

[djsuck]

Thank you for your reply, DrO
The one thing I had not tried out (because the whole Setup is in a datacenter) is, trying to stream to sc_trans from a "normal" non-virtualized Workstation.

With a non-virtualized Workstation as the streaming source, there are no hiccups/skips when streaming to sc_trans.

So if anyone runs into the same problem...
After countless hours I've come to the conclusion:
Virtualized Source -> sc_trans2 = NO
Virtualized Source -> sc_serv2 = Yes

Non-Virtualized Source -> sc_trans2 = Yes
Non-Virtualized Source -> sc_serv2 = Yes

This is kinda sad, but I can't do anything about it

I hope that that'll be fixed in the future, but I also understand if sc_trans-development says: "Not our problem - must be your VM-Software's fault".

With cloud computing being trendy it would be nice, if sc_trans would support virtualized sources.

For me - I changed my approach and switched over to Centova Cast to manage the streams and except for the fact, that the playlists don't work correctly, it's a really great tool.

[DrO]

and every VM is different as we've run sc_trans in VMs for ages without such an issue. who knows what else is running on the rest of what's running on the underlying VM host. but sc_trans is not light on resources due to what it's trying to do i.e. re-encode on the fly and get it to the DNAS as fast as possible and not all setups like / can cope with that without issues.

maybe the newer build may help or it may not (is hard to tell as it appears to be a bit lighter on resources after changing the aac library and with a few other fixes), though the irony of you last comment is that they tend to use sc_trans for things (especially with v3).

[djsuck]

I don't think the ressources are the problem here (see server hw above).
Monitoring the systems ressources shows, that the server and the individual VMs are ideling > 95%

Also sc_trans is doing just fine, when the source is a non virtualized machine.

The problem is, that the hiccups occur, when the source is virtualized - in my case a virtualized Win7-machine with Winamp + DSP-Plugin.
Also not a matter of ressources at that point.

This server is fully under my control and I installed the underlying XenServer, the Debian machine hosting sc_serv and sc_trans and also the Win7-machine as the streaming source with Winamp and DSP-Plugin myself - so I know pretty much exactly whats going on the system - and everything ist just perfect - except for the fact, that it doesn' work *lool*

The customer for which I'm building this setup likes to control the stream and it's playlist via a windows machine, rather then a webinterface - this is why I set up the Win7-VM.

As I mentioned before:
- Streaming from a Win7-VM directly to sc_serv (in a Debian-VM) is no problem
- Streaming from a non virtualized Machine to sc_trans (in a Debian-VM) is no problem

But streaming from a Win7-VM to sc_serv IS a problem.
This seems strange to me, as I virtualize Firewalls, Servers, Desktops, Applications, telephone systems and many more things for a living - but a simple Winamp+DSP Plugin is too much?
I had a really hard time suspending this setup and starting a new approach