Media Library memory leak?

[damenace]

used mp3val on a bunch of folders and they seemed to be ok.
so that shouldn't be the cause :-(

and yes, it's the fresh scanning that's causing the problem, so as DrO said, not the artwork.

any other tests I can run for you to circle in on the problem? anything?

[Aminifu]

Originally Posted by MrSinatra

i think we should be careful in our terms. to me, "tags" are what an app like mp3tag shows you, (and some tags can be "bad" or corrupted) while "headers" are what apps like mp3val are meant to address, mostly fixing audio issues.

MP3Diags is more comprehensive than MP3Val, it tries to fix headers, tags, streams, and other stuff. I know you tried it; didn't like it and don't trust it because it misidentified some files. All I can say is it has worked for me and like I said, no app is perfect.

Originally Posted by MrSinatra

i agree that art itself is not scanned in at scan time, and is only accessed "on demand" ie. while playing or scrolling or viewing in ML. however, if art is in the tag, ie. embedded, it does seem to make this problem worse. since art can be in an ape tag, or a id3 tag, etc... perhaps its worse yet in one or the other, depending on how winamp reads such tags, and then manipulates the data.

also, it wouldn't surprise me if two different yet seemingly similar problems are at work here:

1. seems to be the fresh scanning in problem, while the other
2. seems to be the accessing/browsing of art problem, where the more you have, the bigger it is, if you embed, etc all seem to be part of it.

Given time, DrO will get to the bottom of this.

Originally Posted by MrSinatra

in my usage, i have mostly flacs and mp3s, about 415 gigs worth, and i do not embed, and i delete all extra tag types except id3v2.3 for mp3s, and my art rarely is larger than 500x500, with most being 200x200, and so i think i suffer the problem less seriously than others.

Imo, as long as WA allows use of other tags types, the need to try to support them will exist. After-all, except for artwork, it's just text isn't it?

Other users (myself included) will want larger images (embedded or not) that scale up better for viewing on high res monitors and big screen TVs.

Originally Posted by MrSinatra

nevertheless, the more i scroll and browse artwork, the more my ram usage rises, and it doesn't seem to ever get released.

I see this too. For me, my collection is still small enough that I don't need to worry about exceeding the addressing limit for 32-bit apps. Those in danger of this, will have to paid more attention to memory usage and restart WA to avoid a crash, when usage gets too high. I fear solving this will be harder given WA's early design decisions.

[Aminifu]

Originally Posted by damenace

used mp3val on a bunch of folders and they seemed to be ok.
so that shouldn't be the cause :-(

and yes, it's the fresh scanning that's causing the problem, so as DrO said, not the artwork.

any other tests I can run for you to circle in on the problem? anything?

If that question is partly directed at me, then no. I was just trying to rule out stuff that had been left hanging. Now that one of the best WA devs is looking at this, we need to give him time (since he's doing this 'off the clock').

[damenace]

:-)
the question was directed at whomever had any idea as to what the cause may be or simply knows something we should further rule out.
just trying to help in whatever way i can :-)

[MrSinatra]

Originally Posted by Aminifu

MP3Diags is more comprehensive than MP3Val, it tries to fix headers, tags, streams, and other stuff. I know you tried it; didn't like it and don't trust it because it misidentified some files. All I can say is it has worked for me and like I said, no app is perfect.

i wasn't talking about mp3diags. i was saying that there is an important distinction to be made between tags and headers.

i use mp3tag to spot bad tags, and mp3val for headers.

Originally Posted by Aminifu

Imo, as long as WA allows use of other tags types, the need to try to support them will exist. After-all, except for artwork, it's just text isn't it?

i agree, i was just stating my own usage case and suggesting it as a best practice if avoiding the issue described here is important.

Originally Posted by Aminifu

Other users (myself included) will want larger images (embedded or not) that scale up better for viewing on high res monitors and big screen TVs.

as do i, but i have not found an easy enough way to do it yet. my art is a reflection of how i ripped and the apps i used, more than what i wanted.

still, i'm not sure how big i will want to go. even on a big TV, i doubt i'd want more than orig LP size.

[edit - moderation]
split talk about 64-bit to http://forums.winamp.com/showthread.php?t=344473

[Aminifu]

Originally Posted by MrSinatra

i wasn't talking about mp3diags. i was saying that there is an important distinction to be made between tags and headers.

i use mp3tag to spot bad tags, and mp3val for headers.

Hi MrSinatra,

Sorry I misread your statement. I agree distinctions are important ('the devil is in the details'). Unfortunately, we all get sloppy with terminology from time to time.

Are you saying that you use MP3Tag to just 'look' for bad tags? Does MP3Tag highlight tags with structural errors (things you can't see)? I've only used it to add/replace or remove tags. I thought that MP3Val could also fix some tag structural errors as well as most header problems.

Then I noticed that MP3Diags listed stuff that MP3Val either missed or did not deal with. Sometimes it lists things that even it can not fix.

I should note that almost all of these 'deviations from spec', detected by MP3Val and MP3Diags, did not prevent WA (and other players) from playing the mp3s. That may be because I use "EAC" to make high quality rips and try to only download high quality files. So, in my case, I'm not sure what the 'real benefit' is, other than making the files slightly smaller.

[Batter Pudding]

I assume that Winamp can play the "bad" MP3s fine as it is doing some level of error correction. I'd assume mp3val will be doing an exact "by the book" check against the spec to compare with the perfect MP3 track.

Winamp shows how good it is at handling these kinds of messy errors when it plays back some of the crud that sharing services like Limewire would dish out. Or a Torrent that only got to 98% download leaving those little gaps in the MP3 tracks. (Some of my mates bring round collections filled with some utter trash of "mp3" tracks)


Out of curiosity. If you have a number of "bad" albums that mp3val has cleaned up and corrected, how often have you actually heard any difference?

[damenace]

topic guys :-)

everytime I get a notification you get my hopes up that finally someone found the problem! :-D

I need a solution, not a discussion about MP3 tag quality ;-)

[Aminifu]

Originally Posted by Batter Pudding

Out of curiosity. If you have a number of "bad" albums that mp3val has cleaned up and corrected, how often have you actually heard any difference?

Hi,

When I hear a difference (from mp3val and/or mp3diags clean up) it's not often and it's usually the removal of minimal noise at the beginning and/or end of a song. It seems to be associated with the removal of extra, truncated, and/or invalid data streams and/or frames.

This is not the noise you can get from encoding a wav of a LP or tape. That kind of noise must be cleaned by some other means. I use other apps for that (to clean up the wav), with varying degrees of success.

What I appreciate most often with mp3val and/or mp3diags clean up is not audio. It's the correction of reported song length stemming from bad headers (usually associated with files using variable rate encoding).

[Aminifu]

Originally Posted by damenace

topic guys :-)

everytime I get a notification you get my hopes up that finally someone found the problem! :-D

I need a solution, not a discussion about MP3 tag quality ;-)

Sorry damenace, while its not completely off topic, you are right. If I have anything else to say about mp3 quality (tag or otherwise), I will start a new topic.

[damenace]

not sure how good mp3val and mp3diag really are to be honest.
found three files that, in winamp play fine but in my portable mp3 player produce distortion for the first 2-3sec.
ran them through mp3val and it found nothing. ran them through mp3diag and it found a list of problems.
had them fixed.
still the same distortion. not worth risking damaging my files really. and it doesn't help winamp with the issue discussed here...

sooooo, what might be causing the memory leak? ;-)

[Aminifu]

Originally Posted by Aminifu

When I hear a difference (from mp3val and/or mp3diags clean up) it's not often and it's usually the removal of minimal noise at the beginning and/or end of a song. It seems to be associated with the removal of extra, truncated, and/or invalid data streams and/or frames.

Hi damenace,

Since you opened the door.

The noise I'm referring to is not distortion. It's more like a squeak, crackle, or pop. Sometimes I use "MP3 DirectCut" to snip off the portion with this noise, when it's a little before or after the song and mp3diags doesn't clean it up.

Distortion can have many causes and be present in some players and not others. I have had a few variable rate mp3s sound fine in some players and sound distorted in others. I assumed it was some fault in the player and not the file when this happened. It could also have been different error correction schemes for the audio data, which would not be a fault of the player.

I don't think mp3val or mp3diags tries to correct possible bit errors in the audio data. Songs with distorted audio (in all the players I use), I either re-rip or download from a different source.

[MrSinatra]

Originally Posted by Aminifu

Hi MrSinatra,

Sorry I misread your statement. I agree distinctions are important ('the devil is in the details'). Unfortunately, we all get sloppy with terminology from time to time.

np. the idea is just to educate someone else who might not know.

Originally Posted by Aminifu

Are you saying that you use MP3Tag to just 'look' for bad tags? Does MP3Tag highlight tags with structural errors (things you can't see)? I've only used it to add/replace or remove tags.

mp3tag can report tags as "Bad" meaning that it may be able to read them, or not, but either way the tag is somehow malformed and not the way it should be in the file, like in a data structure way.

i would try to eliminate those if looking to minimize this issue. its the "tag" column with this equation: %_tag_read%[ (%_tag%)] where it will say "Bad" in red if it doesn't like how the tag is done.

Originally Posted by Aminifu

I thought that MP3Val could also fix some tag structural errors as well as most header problems.

i believe it can and does, but i don't use it for that. i've never had it do it wrong that i know of, but mp3val is unfortunately aging, and i trust mp3tag to be better at tag evaluation.

Originally Posted by Aminifu

Then I noticed that MP3Diags listed stuff that MP3Val either missed or did not deal with. Sometimes it lists things that even it can not fix.

yep, prob true. but i find diags too dangerous to use.

Originally Posted by Aminifu

I should note that almost all of these 'deviations from spec', detected by MP3Val and MP3Diags, did not prevent WA (and other players) from playing the mp3s. That may be because I use "EAC" to make high quality rips and try to only download high quality files. So, in my case, I'm not sure what the 'real benefit' is, other than making the files slightly smaller.

i have had files that when "fixed" by mp3val, sounded unquestionably better. mp3s are by their nature very robust and error tolerant, but to get the best out of them, you want sensible headers. headers allow the playback to know what the data is truly encoded at, esp vbrs or those files with multiple audio specs in the file.

i have also had files with "garbage" at the end, that when cleaned would keep the file from popping.

[Uwe Fissenebert]

Hello,

I read this thread right from the beginning. I use Winamp Pro for several years now and I had the problem of "crashing" when adding new files to the library from starting using winamp until now.

My first machine was a slow Pentium single core, 1 Gb of RAM and Windows ME. My current machine is an Intel quad-core 2500, 8 GB and Win7 64 Home. The scan-problem with winamp is still the same, no changes.

My library is about 50k of tracks in more than 8k of albums (folders). Every track has cover art (jpg, gif). All tracks have ID3-Tags, well edited with mp3tag, no errors. All tracks are verified with mp3val, no errors. All tracks are normalized by mp3gain, no errors. The structure of the file-storage is format/artist/album/track (format = mp3, ogg, ...). Only mp3-files were used for my winamp-library.

All other programs I tried out had never ever any problem scanning the files, showing a media-library, displaying cover-art etc. - but Winamp.

The scan-problem is not a new issue, it exists for years now. You can find several threads about this in the winamp forums. I tried all workarounds ever posted here - without success.

So I can not understand, why this issue still exists, because it's a major function of winamp. Sorry, I can not understand any more why the dev-section argues the problem is unknown and can not be reproduced. It seems, the dev-section does not know about reported problems in forum threads. If so, please open a special issue-tracker for reported problems, maybe filled by moving forum-threads to that tracker. Why do we post detailed technical descriptions of problems here if they are not recognized by the dev-section?

If the problem is reported again and again over years, why is it still not accepted as a major problem (even when it can not be reproduced at Nullsoft)? The pure existance of the problem with winamp for a long time is a good reason to think about the winamp code and think about a safe recoding of the library-section to avoid known problems under almost all conditions. Maybe this is done for version 6. Many users are happy to help you to solve such problems.

For the moment I "solved" the problem this way: I use winamp only for internet-radio and manage my audio-library with another program. I don't want to do so, but I have to.

Kind regards.

[Aminifu]

Originally Posted by Uwe Fissenebert

Hello,

I read this thread right from the beginning. I use Winamp Pro for several years now and I had the problem of "crashing" when adding new files to the library from starting using winamp until now.

So I can not understand, why this issue still exists, because it's a major function of winamp. Sorry, I can not understand any more why the dev-section argues the problem is unknown and can not be reproduced. It seems, the dev-section does not know about reported problems in forum threads.

Reproducing a problem is key to solving it. As you have read, a dev has finally been able to partially do this. Please don't piss him off. He's doing this work on his own time. Why that is so is another issue. Let's give him time, after so long there's no need to rush the person who is trying to help.

[Uwe Fissenebert]

Sorry, being unpolite that was not my intention. I only wanted to express my anger about the handling of this problem for years now. This issue runs through my whole winamp history.

Of course, solutions need time. If I understand it right, someone has accepted that it is (or might be) a winamp-problem. That is a big step forward. Thank you for that.

If I can do anything to help the dev-section to solve this problem, please let me know. I wish to have my winamp back to full functionality.

Kind regards

[Batter Pudding]

@Uwe - I can understand your frustration.

I'm an ex-programmer. And as you see in the thread, have also tried to assist with this issue. Trouble is, if they cannot reproduce an issue reliably in the lab, they can't fix it. Also notice that this issue only happens to a small minority of Winamp users. Which again makes it harder to trace. The puzzle of "what is different in their systems?"

Post comments about the kind of tracks you have, the way you organise your collection, if artwork is generally embedded or loose files, quality (compression and size). Add in OS version and memory sizes. Maybe even the odd note of extra programs you have running on the PC. Little bits of information like that may eventually make sense. A common theme may then start to be spotted among those people who are suffering from this issue.

As you can see above, attempts have been made to look into the code, but nothing has appeared as obvious yet. Also remember, Winamp has a very small dev team. So work is rarely that fast. Well - it is a free product. The "pro" costs only really cover licensing of MP3, H.234, etc. (And I don't think a huge percentage pay for that going by comments from people in this forum)

[Uwe Fissenebert]

OK, this is the machine:

Intel i2500 quadcore
8GB RAM
Win7 Home 64bit
USB-disk 1TB

Winamp Pro, latest version, no extra plugins, Bento-skin.

Audio-pool 1 (290 GB):

25k tracks, every track is

mp3
variable bitrate (196 - 320)
ID3-Tags V 2 and 3
cover-art (JPG, GIF, up to 512x512 pixel)
normalized (mp3gain)
tested with mp3val (no error)
ID3-Tags and cover art were added by mp3tag

structure on disk:
mp3/artist/album/track
filename of track: artist - album - track# - title.mp3
playlist of album in album-folder (artist - album.m3u)

Scanning pool 1 causes error. The scan terminates, Winamp is closed. This can happen any time as described by other users in this thread. When starting winamp again, the scanned files may be present in the library, but most times all is gone. If the results of the terminated scan are still available and a new scan is started, the scan starts at the beginning of the pool-structure (first folder). Some times the new scan dies earlier, some times later than the first one. There is no fixed point the crash is initiated. There are no additional information from me about that behavior. All is said in this thread.

This behavior is independend of hardware and software running during scan. This behavior is present since years on different machines (all Win) and on different circumstances (system, devices, background-processes, ...).

Today I made a new "pool 2" for testing winamp (130 GB), just for this thread:

30k of tracks, every track is

mp3
static bitrate (128, some 196, little 96)
ID3-Tags V 2 and 3
cover-art (JPG, 256x256 pixel)
normalized (mp3gain)
tested with mp3val (no error)
ID3-Tags and cover art were added by mp3tag

structure on disk:
mp3/static/track
filename of track: artist - album - track# - title.mp3
no playlists in folder

So all the tracks of pool 2 are in one folder, there are no playlists or other stuff in this folder, just tracks as described. This is the opposite structure of "pool 1" in many cases.

I cleared the library and started a new scan on pool 2. The scan lasted about 6 minutes, was successful, no errors, all tracks were in the library. I watched the scan with "Process explorer". During the scan there was an average of free RAM of about 2,5 GB, rising and falling. Normally I have an average of free RAM of 3,5 GB (yes, the machine has always work to do).

I hope this helps to find the problem.

Kind regards

[Batter Pudding]

Here is a little suggestion to try (it is also included earlier in this thread)

With "Pool 1", instead of using the Automatic Scan to scan the whole folder tree. What happens if you "drag and drop" groups of folders in? For example - open Windows Explorer looking at the mp3 folder in "Pool 1" and "drag and drop" all folders from A to E into the main media library window. If that works, drag and drop F to M. And so on. Does this method let you get to the end?

If the above test still crashes, try again but this time exit Winamp after you get to M. Allowing the Media Library to be written fully to disk. Watch Task Manager after you quit to make sure the Winamp process goes away. Then restart Winamp and drag and drop N to S, and so forth up to Z.

If your bug follows the pattern of other people, this seems to get you a completely scanned library.

A downside of this method - you have to go back into the Preferences \ Media Library \ Watch Folders and clean up the list. That list is now full of every individual folder name you have been dragging and dropping. It is worth clearing that out now back to just the root of Pool 1 as usual.

I have also seen cases where people report that once the library is in place, subsequent scans to update the library then seem to work.


Note - as mentioned earlier in the thread, watch out for the crash always happening on the same file. Sometimes corrupted files have crashed Winamp. Also do a disk check on your hard disk as a failing hard drive will also cause a crash. (I doubt this is the case with your example as you say other programs are handling this fine)

[MrSinatra]

just curious what everyone else thinks... but have we not established that you are more likely to run into this problem if the following two things are true:

1. you embed art in most/all tracks, and
2. you use "big" high quality, high res art?

i believe this is the root of the issue. i think you could test it by making a large library, say of 500k tracks, but no art at all of any kind, and see if the problem persists.

[Batter Pudding]

Originally Posted by MrSinatra

just curious what everyone else thinks... but have we not established that you are more likely to run into this problem if the following two things are true:

1. you embed art in most/all tracks, and
2. you use "big" high quality, high res art?

i believe this is the root of the issue. i think you could test it by making a large library, say of 500k tracks, but no art at all of any kind, and see if the problem persists.

If you skim back through the thread, you will see that once DrO started digging into the code, the artwork idea soon dropped out of the running.

Also notice that Uwe has a "Pool 1" that crashes and he is listing artwork as only 512x512 and JPG \ GIF. So reading between the lines there, as he mentioned JPG\GIF then that is a single loose file per album. Not embedded.

@Uwe - can you confirm that please? Are your MP3's just music with the artwork not embedded? Also what is the file size of an average piece of artwork you are using?

[MrSinatra]

Originally Posted by Batter Pudding

If you skim back through the thread, you will see that once DrO started digging into the code, the artwork idea soon dropped out of the running.

i did see those, but i also saw this:

Originally Posted by Batter Pudding

What I was suggesting is the in_mp3 code could be loading all the MP3 TAGs before working out what they actually are. So this would not trigger any calls to the image library. It would just cause something to allocate some memory for the data being read from the .mp3 file.

You may need to use something like mp3tag to insert some huge 1MB JPGs into every mp3 file of an example album to see the memory changes. Personally I'd be tempted to make an example album with stupidly big MP3 tags for testing using 5MB images. But then I always did like evil test cases when coding.

and i keep seeing people who embed having this issue:

Originally Posted by Batter Pudding

Also notice that Uwe has a "Pool 1" that crashes and he is listing artwork as only 512x512 and JPG \ GIF. So reading between the lines there, as he mentioned JPG\GIF then that is a single loose file per album. Not embedded.

@Uwe - can you confirm that please? Are your MP3's just music with the artwork not embedded? Also what is the file size of an average piece of artwork you are using?

i took away a different impression reading this:

Originally Posted by Uwe Fissenebert

My library is about 50k of tracks in more than 8k of albums (folders). Every track has cover art (jpg, gif).

[Batter Pudding]

Yeah, I remember that I was one of those nagging at the images. I just thought that DrO had ruled them out after looking in the source code.

I also can't guess at what Uwe meant about the artwork. I don't know enough about how things are embedded. When he was talking of GIFs and JPEGs I assumed that was talking about files. That is why I asked Uwe for clarity at the end of my post.


As you understand these tags better than me - what format is the artwork when it is embedded? Can it be either GIF or JPEG? I had previously just assumed it was only JPEGs embedded and anything not JPEG was converted. But that was me guessing and not reading the tag spec.

I understand that different levels of compression and different sizes can be used in the artwork. So what image types are supported? Maybe the differing types of images could be sparking different results?

[MrSinatra]

Originally Posted by Batter Pudding

Yeah, I remember that I was one of those nagging at the images. I just thought that DrO had ruled them out after looking in the source code.

he did, but your post i quoted above was well after that, and i agree with it. DrO has been wrong before, and i simply can't discount the seeming correlation between these problems and embedding, however anecdotal.

Originally Posted by Batter Pudding

I also can't guess at what Uwe meant about the artwork. I don't know enough about how things are embedded. When he was talking of GIFs and JPEGs I assumed that was talking about files. That is why I asked Uwe for clarity at the end of my post.

when you embed, you are literally embedding an image file into the tag. so it makes sense to say "i embedded a jpeg into the file" b/c you can extract a jpeg from it. in other words, you actually have jpegs (or whatever) in your tags. the id3 frame is called APIC i think. not sure how its done in vorbis.

Originally Posted by Batter Pudding

As you understand these tags better than me - what format is the artwork when it is embedded? Can it be either GIF or JPEG? I had previously just assumed it was only JPEGs embedded and anything not JPEG was converted. But that was me guessing and not reading the tag spec.

i don't embed, mainly b/c i think its inefficient and leads to more problems (like this) than it solves. however, i think the format is unchanged. whatever you put in, remains that way, and its up to the app to support a jpg, gif, png, or whatever.

so to be clear, i'm no expert regarding embeds. i could be way off.

Originally Posted by Batter Pudding

I understand that different levels of compression and different sizes can be used in the artwork. So what image types are supported? Maybe the differing types of images could be sparking different results?

i think that depends on the app. but i would say jpg is most universal, followed by gif and then png. i would not use anything other than those three, and really, i'd try to stick to jpg exclusively. nearly all my folder art, is jpg.

i def think its some ratio of number of art files x resolution x art file size. i know DrO says this is barking up the wrong tree, and he may well be proven right. but i would like to see a test case, where a user has say 100k files all with big, HQ art embedded, vs the same 100k files with no art all of any kind, and then do the scan in new files, like 20k more. there has to be a way, like that, to start testing out some of these ideas or theories.

EDIT: BMPs used to be common as well, not so much anymore.

[DrO]

what i read of the code indicated that the tag is read and then the memory allocated for that is freed. that that aspect of things does not care if there is embedded artwork in the tag or not, the tag is read as a whole and then processed after that.

yes i could be wrong but as i can only base things on what i'm seeing in the code and on the machine in front of me then that was what i said at the time. so maybe the large tags is triggering something, i know i was seeing something funky (as said in my earlier replies) but there was nothing glaring going on and it could even be things are better with the internal builds vs the current build but i just don't know.


anyway, i've not done anything else since and as things are going, i don't think i'm going to find any time soon - i've already had to officially kill development on my plug-ins due to the lack of time.

-daz

[MrSinatra]

just so its clear, you're still the man imo DrO. i just can't get over the correlation but i know that doesn't prove causation. still, when BP says:

"What I was suggesting is the in_mp3 code could be loading all the MP3 TAGs before working out what they actually are. So this would not trigger any calls to the image library. It would just cause something to allocate some memory for the data being read from the .mp3 file."

it just seems to me that there must be something to that.

i think the onus is now on the users, specifically those affected, to create a reproducible example. i think my test situation above would in fact reproduce the issue.

[DrO]

Originally Posted by MrSinatra

still, when BP says:

"What I was suggesting is the in_mp3 code could be loading all the MP3 TAGs before working out what they actually are. So this would not trigger any calls to the image library. It would just cause something to allocate some memory for the data being read from the .mp3 file."

as i said, that is what Winamp is doing, it indentifies the tag block, loads it and then processes it. what then happens will depend on how and what is requested from the tag by the other plug-ins doing things. but everything in there is setup to free that memory at least from the tag load and read side of things - maybe the leak is elsewhere bur like i said i don't know for certain.


really the best thing to confirm if it is definitely a tag related issue or not is to do as was done earlier and disable the different tag reading options to see how much of an effect that has on the additional collections being mentioned (i've not read all of the new posts, just cba at the moment).

-daz