Listener disconnections, rebuffering, relay server collapse etc

[MrBlades]

Hi guys,

I've spent some time trawling through these forums in an attempt to find a solution to a rather irritating problem and after hours of following false leads I've given in and decided to post here. Apologies if this has been detailed elsewhere and I've missed it but Im at a loose end here.

I help maintain a community driven Radio Station for an Onling game (EvE Online) and over the past 20 months we have built a fairly respectable listener base. We exist totally on donations from listeners in order to fund our operations and put an awful lot of time, energy and effort into running our station.

Due to our success, we have at peak times anywhere from 200 - 500 concurrent listeners on both a 96kbps 44hrz stereo and 24kbps 22hrz mono and broadcast 24/7 with around 30 live DJ's covering every time slot and an automated jukebox when not broadcasting live.

Until fairly recently we have never had any trouble with dropping the current live DJ and having another pickup the stream within a couple of seconds resulting in a nearly unbroken hand off from one user to another. This meant that there was no noticeable interuption in the stream from one DJ to the next and users seldom had to reconnect in order to continue listening. As we all broadcast to the master server at the same bitrate theres none of the usual mixed bitrate glitches which can occure - basically under normal circumstances things run perfectly.

However, we now have an issue whereby whenever a live DJ drops his connection for a hand off and the next DJ's connects to take over, listeners rebuffer the last 30 seconds of the audio over and over for around 4 or 5 minutes. Each time the buffer is read and comes to an end the connection is dropped and the listener needs to reconnect. The listener rebuffers last 30 seconds of braodcast again. This repeats for a few minutes until it clears.

Even more irritatingly, the number of active listeners (according to Radio Toolbox) begins to jump dramatically (presumably as listeners are disconnected then reconnect) effectivly duplicating or ghosting (as we like to call it) the listener numbers.

The worst thing that hapens is that after 4 or 5 minutes of this, all the relay servers die, dropping all active connections which then need to reconnect.

People being lazy or listen to the stream in the background may not immediately reconnect, which means we suffer around a 60% - 70% loss of listeners which takes upwards of 45 minutes to get back - if they come at all.

All in all, really frustrating and despite our best efforts we have been unable to find a solution.

A little bit about the setup :-

We have a master broadcast server that we stream to. This server is set to only accept a few connections and relays additional connections to one of five relay servers. Shortly before the troubles began we updated to the most recent version of shoutcast and had no issues. After a while, these issues appeared and even after rolling back to a previous version which gave us no trouble the problem remained.

Sadly I dont have exact configuration etails to hand for scrutiny, but we have examined, tweaked and tested the backend config as much as possible and although we're by no means experts, we know enough not to overlook something simple (at least we believe so). Oh, and we run the broadcast and relays on linux boxes.

So anyway, that's our story. It is one of woe and we really need to get this sorted as we are suffering horribly due to it.

If anyone knows of this issue and has a solution I would greatly appreciate any advice. If there are any questions that need asking please do so here and I will answer them all to the best of my ability.

Many thanks in advance.

MrBlades
EvE Radio
www.eve-radio.com

[bored_womble]

A quick thing to check, which springs to mind, as I have had something similar in the past.

in the config

AutoDumpUsers=0

Which should mean when the source drops, dont disconnect the users, which may help.

BW

[MrBlades]

Hi,

Thanks for the feedback. Unfortunately this hasn't made any difference and we are still having the same issue. Anybody else got any thoughts?

[djSpinnerCee]

I would look at "newer" versions of the player software being used by your listeners -- for example -- RealPlayer One can now cache and allow queueing [FF/RW] of a live stream -- this "smart" player behavior could cause what you see.

Sometimes upgrades to player packages [super buffering algorythms for example], can cause side effects that are almost impossible for you to "tune out" on the DNAS end.

Basically, what I'm saying is there may be nothing you can do.

You may also want to consider turning "AutoDumpUsers" ON to force the listener to be dumped when the source change is made, because I think most newer players are designed to quickly reconnect when disconnected. This is the proper behavior for a player [and TCP/IP], and it will probably do so immediately if it recieves a proper "TCP SigTerm" command from your DNAS.