Unfortunately, running the remove APE tags still. (from noon)

:( i'd hoped you'd not have started doing that to be able to see if the ape tag fixes i'd made improved things with the new build.

since it runs since yesterday afternoon, I could not even try, but I have a backup and I'll try the new version immediately.

Hi DrO,

I don't feel jumped down on. To the contrary, I feel somewhat honored that you took time to explain some things in detail. My problem is reading too much of the fud and not really knowing what the code is doing.

With very large collections some tradeoffs are inevitable, but I'm confident the scanning crashes will be solved in time. As to the memory usage due to displaying artwork. It is easily managed, for the time being, by paying attention to the number of images displayed (especially those with very high resolutions) during any single session of Winamp.

User "NoPlayBack" provided a file with ID3v2.2 tags and embedded art in post # 22 of the following thread.

http://forums.winamp.com/showthread.php?t=350911

If that link no longer works, I've provided the file with the following link. It should be good for the next 7 days.

http://wikisend.com/download/507560/1-17 Dreadlock Holiday.mp3

well even i don't know what all of the code is doing, heh. the main thing really is media db and artwork db's being thought off as the same when they're not (which is an implementation detail as such but i can see how it'd be viewed that way externally).

though there is the case that we could populate the art db's on media import as well though that would slow it down noticeably for such collections hence doing it like the current way on display when enabled in the views. like you say, there's trade-offs.


thanks, knew i'd seen it recently, was just buried so it wasn't coming up in the searches i'd tried. i've grabbed your copy as the original is no longer there, so will try to get that working.

full scan without APE

well the crash is different this time which is a (welcome) surprise. i assume you're not a registered Winamp Cloud user?

as it looks like the Cloud plug-in is trying to process the files which are being added to the library which is what it's meant to do but not if you're not logged in as that isn't going to help with memory issues either.

so, if you can remove ml_cloud.dll from the plug-ins folder and try again, that will remove that from the equation (as i know it leaks memory when logging events and is something i need to get fixed asap).

First I deleted the ml_cloud.dll from the plugin folder. Then a total scan, and after a long time is a title, name stood still (up to this point about normal memory behavior 480000k) then the memory has risen to about 900000k and it has terminated. Then I started winamp again and the contents of the media library is available (624432k).

thanks for re-running things. well it's doing a lot better than it was before as it's no longer failing during the main bulk of the import process as was happening with some of the crashes (when we're getting the data and initially storing it).

the doubling of the memory is due to us attempting to re-create the database whilst it's tidied up for saving to disk and it's still in that stage when things aren't working correctly.

the memory size on loading afterwards seems pretty reasonable at around 2.5-3kb per file in the library (working against 250k of files as you'd mentioned previously).


just had a thought now that you've got things to import (despite the crash), could you upload a copy of the main.dat and main.idx files from %appdata%\Winamp\Plugins\ml as i might be able to do something against the data in there.

ok is to large for the forum 23,7MB

thanks, getting it from the pm link received.

that really helped with testing of things, thank you! as i finally got it to crash on my side and have fixed the reason for the crash.

it doesn't help with the fact that we run out of memory to allocate during the import / rescanning process and will very likely mean not everything will be imported (though will see what else i can do towards that).

so we'll need to look into what we can do to help with all of that what with libraries getting ever larger but it should at least not crash now which should mean the import in general should work as expected (at least i hope so based on what i've seen when trying to combine my library into the one provided and how things were failing).

That's great to hear!

Now if unixgeri would be kind enough to run the test version against his collection with the APE tags, to rule out their possible involvement.

well, it'd need another test build being provided for that to be attempted. plus i've a few other tweaks which may help with the memory fragmentation issue (am a bit doubtful but it helped do a merge of unixgeri's library and mine without allocation failures which i'd not been able to do previously).

Hello DrO,

I have been using winamp on my XP PC for years and just upgrade to 5.64 on my Windows 7 64 Bit PC as my XP is about to die :(... I found that 5.64 will not scan my Music Directory at all fails very quickly. Deleted the version I have on XP and re-scanned no problems.

I have removed the ml_cloud.dll as suggested above with no effect.. removes all addin's and still fails. I am about to do a bug trace on the running program to see where it fails and obtain a dump at the fail point.

All my MP3's are clean as far as I am aware for Tags as I keep it up to date for cataloging my Music collection from LP's etc,. I have transcoded to MP3.

If there is anything I can look for to assit please let me know and I will look out for it.

OzBrown
Music the life of the Relaxing

see http://forums.winamp.com/showthread....84#post2950984 (which ironically was the top thread until i replied to this).

as you're hitting the obvious bug we could fix (which is 5.64 specific), and not the obscure memory failures which unixgeri's very large library is triggering (which is likely present going back to when the library feature was added in v2.9).

My issue and solution
 

I had the same problem, and then decided to import the file in batches by moving them to the permanent directory, importing them, and then moving them to a temp location to be moved back once the full 1700 songs were downloaded.

Started working great, and then it crashed. I found that one of the albums i purchased digitally had a video included with the purchase. I removed the video from the directory, and all the files imported.

Is there a new version to test? or anything I can help? (a renewed full scan with and without ape tags) :-)

I assume you read posts #53 annd 54 above. From what I hear, version 5.65 or a new beta should be released very soon.

*tada* -> http://forums.winamp.com/showthread.php?t=356750 (newer public 5.7 build #3434 at the time of this reply).

Great first rescan without problems, Will now add all remaining directories, and if ok then another test with the APE TAGS (I have a complete backup) many thanks.

:up:

that's good to know. there's still a lot of other things which could be done but am hoping the changes so far should help for the time being (as the rest involves fundamental changes which cannot be commiitted to currently).

btw thanks for all of the testing done, is much appreciated (especially the large library files as that's helping to see if there's anything obvious which can be done to improve large library handling).

after the scan (without ape), I noticed that about 1/3 of my files were not in the library. Did I was thinking to delete my entire library exit winamp restart it and do a re-scan. Unfortunately, the same result-it seems to be random what is missing. if I then added a single folder has been missing, a par title coming to the library with an incorrect number eg instead of 20 titles in the album appears to be 145499 and also the title repeated often. All scans were without crash. (also in the original library that I have posted in many things I missed unfortunately only now noticed.)

that is sort of expected as the main aim was to get it not crashing. when we're hitting the memory limits with a large library we now try a few things to keep going but its very likely that it will stop being able to add newer files.

to better handle that we need a complete change of the library handling which just isn't possible at the the time.

at the time my best suggestion is switch to a classic skin, don't use a language pack, disable any plug-ins not used and do the import as soon as Winamp is started. as that should lower the chance of the memory being badly fragmented (which is the main reason for the import issues).

Or set a bigger Pagefile.sys, this should probably also help (?)

that won't have any effect on things as the issue is down to the memory needed and how the available memory in the Winamp process space is fragmented which can lead to all sorts of weirdness.

the main aim of the changes has been to better cope with memory allocation issues so we're not crashing and attempting a few ways to try to get the memory needed (but that can and does fail still - just not crashing in the process).

it effectively requires a large overhaul of how memory is allocated by the library which just isn't something that can be done at the moment (the fact we're no longer crashing for most installs should help with things, it's just extremely large ones like unixgeri's where things will get to a point and then stop working as expected).

I'm back to version 5.63, I have not found a way with the new version to create a valid library ('ve even tried to split into 50 parts and individually scan winamp of course always ends in between) in the version 5.63 I can scan a complett make a passage (different to earlier is all I have APE s away) but the version from crashes after scanning but it will then contain all the title. If DrO need to test a library I make them happy again for disposal. Anyway thanks for the efforts.

I have re-read the thread, but I remain confused. a chronic condition for me it seems.

tell me if I have this right:

when scanning, winamp attempts to load everything into memory, and only writes out the DB after scanning everything. is that correct?

b/c it does it this way, it has a limit as to how many items can actually be in the library, although that limit will vary according to what OS and hardware you have. is that true?

the "fix," discussed in this thread, isn't to actually address the problem and correct the issue, but rather to mask it or hide it, by keeping winamp from crashing. the idea is that its better for winamp not to crash, if that's all that can be done at the present time. is that an accurate assessment?

obviously, the danger is that someone won't know things are "missing" from the DB, should this happen. sure, its obvious if its a third of your stuff, but what if its only 100 items out of 100k? given that libraries are only ever growing larger, this issue will probably need fundamentally addressed sooner rather than later. couldn't a warning be flashed to a user that winamp encountered an issue during scan, so the problem isn't silent and masked?

what I don't understand is why the DB isn't written out as it scans, so that memory can be released? surely the DB itself is capable of much bigger numbers? but I could easily be mis-stating the problem/how things actually are.

I am also curious if certain tags are being blamed here. does the problem become more evident if there are multiple tag types in a given file? like id3, lyrics, ape, etc? are certain types more prone to provoke the issue, like ape? are certain vers, like id3v2.2 more likely to provoke the issue? (I saw the fix for apic in id3v2.2 for example)

I am also perplexed why, after a certain point, doing the scans in "chunks" doesn't work? that would seem to suggest my understanding above is faulty, b/c its working with an existing written out DB at that point, so what then is the problem? (unless its the case the entire DB is loaded in order to load in the new chunk?)

a brief chronological presentation of the up event

1) about 3 years before I even get out of plates ript mp3 from my brother about 2000 when I added this to ml (before ca 220000) crashed during the scans from winamp. I thought well it's more a limit is exceeded, searched forums and found a note that could be the problem already included album arts. then I then made the new 2000, all album arts removed (using mp3tag.exe) has brought nothing more forums search resulted in lumps you want to scan. I have done this and also monitored the task manager, the memory size and found that the problem-at about 700 000 kb memory requirements begins. if one breaks off and then scans again everything works.

2) Since any version (I dont know it) is then no longer update after scann winamp
message without it crashed but were getting all the titles available.

3) on June 22 I then installed 5.64 and ran winamp not start anymore and I have written to the forum.

4) the exact sift through my files, I noted that in the 2000 ape tags in addition to the id3v2 inside are I have also sought to big or faulty embedded artworks (jpg and all 300 * 300 72dpi) and have found some apparently large (200kb, the large part of the other is less than 50kb) have this removed (ape and large jpgs).

5) Dr.O ml of me dat and idx and if he could to reproduce the error.

6) several new beta versions tested but never the whole collection, and you've seen that's not the memory requirements increased. I have not noticed the number of entries. it was always the cloud plugin deleted because otherwise it is very fast to crash (memory)

6) and last beta for existing ml added another folder and it has struck me that the number is no increased.

7) library deleted and completely re-scann only about 96000 title


8) 50 Individual parts added (and always end with restart) is also approximately the same result ca 96000th

9) back to version 5.63 and a complete scan Directorate same as before except that after the scan it was still useable ready to crash And arrived just all about 270000 titles are indoors (difference to the look are 3 files in explorer).

these tests were very expensive since a scan takes approximately 8 hours.

I would not criticize I'm a winamp fan, but only help to eliminate errors. If I can help something, please let me know.

excuse the bad english it also it comes from the google translator, but I can read everything and understand what you write to me.

Therefore I think that the APE tags are a problem, and on the other site, a new error has been added,

As DrO stated in post #40 above, Winamp tried to load the entire metadata DB (sans artwork) into memory in order to enhance performance when working with it. If this memory usage, plus what is needed for other things, causes Winamp to attempt to allocate more than a 32-bit app is allowed to use, then various problems occur (depending on what was being attempted when the memory allocation was exceeded) which would ultimately end in a crash.


Partial quote of post #40 below:

"the media import goes into the main.dat/main.idx files and are separate to the artwork cache (art*.dat/art*.idx). the db is loaded into memory when we get a request to make use of the db and is like that due to the performance effects it has over always reading from disk and it does make use of virtual memory as that's how everything goes through. and it's mapped into memory for the performance it offers over the alternative.

the issue with this is irrespective of what we do with the current implementation, as a 32-bit process we are fundamentally limited to what can be done within those confines due to limits on virtual memory and all that. so if we try doing things with large libraries and need to re-build things in memory (since it's a hell of a lot faster than doing in pieces and working on disk), when we do an import and then re-build things for saving out (as we do since it provides a better experience post import), if we've already pushed for memory availability then we are going to have issues and random failures as we cannot access the memory needed.

there are things which can be done relating to all of this but it is not a simple thing to do."


The current Winamp version is trying some of the things DrO alluded to in order to complete a scan of very large collections without crashing.

the tags are not the issue now (there were a few minor leaks on a large library which did not help but there were bigger issues elsewhere), its just down to how we allocate the memory needed to be able to store what is read during the import process where we have the issue in conjunction with how memory is available at the time in the large blocks which are needed during the whole import and saving / re-reading process.

there's a few things I can try and will be following up on that in the coming weeks and is just a matter of time to make changes and test things which is the issue at the moment. as we need to be better in general in how we use memory for smaller libraries all the way up to being able to import large ones like yours.

[edit]
post above wasn't there when I'd started. the 5.65 changes are a starting point and for most having the memory issue it resolves the crash at the risk of not adding more files. that is not a final fix but changing an entrenched way of doing things since 2002 is not simple to change without causing other issues e.g. we broke ml_ipod as part of the first internal beta sent out with all of the changes so far from the library changes.

and due to the stream of complaints about the 5.64 import crashes, a 5.65 was released as is to solve things for those people once we'd gotten the key memory related crashes resolved. next comes optimisation and getting unixgeri's library to import correctly.

I understand the issue better now, and will not reignite the 32bit/64bit debate. however, until such time that the issue can be properly addressed, I do think some kind of mention in the form of a pop-up dialog or something should exist, so the issue is not masked and silent.

I agree that keeping winamp from crashing is better, but I never think masking an issue, especially one that can easily not be noticed, is a good solution, even if only temporary.

it would also be helpful to know around what the threshold is? meaning, 150k items? or what?

if we knew that, perhaps at that point winamp could be triggered to use a different method that included disk swapping if ultimately necessary?

the threshold is at what point we cannot allocate a continuous block of memory needed for the items being added / updated against however much metadata they have read from them. that is what happens when available memory is fragmented.

yes, I understand it is variable depending on the actual data presented.

but lets set some assumptions and then get actual numbers.

would 100k mp3s that each had 4kb of metadata in them (not including artwork) cause the issue reliably? 125k? 150k?

(is 4kb the max amount of metadata for id3?)

as you understand it's variable depending on the actual data, you'll understand that there is no fixed / quantifiable answer.

that's why I gave 4kb as the per file metadata size for this experiment. is there another variable to consider? (and/or a more likely value size to use? I just went with what I think is the max per file value)

Since Winamp needs a lot of contiguous memory to successfully scan large collections, it may help, for now, to close all apps and re-open Winamp to a UI view that does not cause an artwork cache to be loaded into memory.

Maybe Winamp should unload any artwork caches when a scan is requested (until the scan is finished and the associated memory is released), but I don't know if that would help with the memory fragmentation issue. Once an artwork cache is loaded, currently the only way to unload it is to shutdown Winamp.

My reasoning follows, sorry for the length:

When I open the Bento skin (maximized) to the media library's "Local Library" panel, the 'Peak Working Set (PWS)' memory as shown in the Task Manager rises to about 257 MB (I use a lot of plug-ins). When I switch to (or open with) a panel showing multiple (60 when maximized) large icon album art images (which causes the art_120.dat file to load), the PWS jumps to about 566 MB.

I currently have a small collection (only 6,268 songs) and my art_120.dat file is about 200 MB. 566 - 257 = 309, so I have no idea what the extra 109 MB is for. Maybe it is partially used to actually display the 60 images.

Anyway, when I open Winamp to a view that does not cause an artwork cache to load, select all the songs, and command a re-read of all metadata, the PWS rises from about 257 MB to about 264 MB. This small 7 MB increase is because I have a small collection.

Now, assuming the memory increase is linear, increasing my collection to about 300,000 would use about 350 MB (about 50 times as much and assuming the average metadata usage to stay the same). Since Winamp needs twice the memory for it's processing, that need would be about 700 MB. Large, but 32-bit Winamp currently should be able to handle it, especially if it is the only memory hog running. ;)

However, assuming the art_120.dat file will also increase linearly to about 10,000 MB, no way could I hope to successfully complete a scan of 300,000 with an artwork cache of that size also loaded in memory at the same time.

Of course, the working set memory is equal or lower than the peak working set memory. Sometimes it is substantially lower. But, I assume the instantaneous peak demand would be enough to cause a crash. It is should also be noted that my MB values are million bytes not megabytes.

closing other programs has zero effect on things - it's the fragmentation of the memory inside of the Winamp process which is the issue - what another program is doing is not going to make a difference in the general scheme of things.

how the artwork cache is handled might help with things on updates / later imports depending on what is currently being viewed though that is an area i've not really looked into though from a clean import (which is what this thread is covering), there isn't going to be an art cache and the cache won't even be built until after the import and a view is loaded (which requests such artwork).


it takes more memory to display artwork than it does to store it (when it's compressed rather than using the raw bitmap data) so displaying is going to use more memory (just the same as how a modern skin engine uses more memory since it has more images to be loaded which in turn uses more memory for the decoded image data).


all of the caches should ideally be able to be flushed / unloaded after a time when they're not being used and is something i want to look into / ensuring we're not loading things if not needed or not configured to be loaded until they're actually needed. as well as making sure that for things like artwork, we are definitely not duplicating / are freeing memory on view closes and all of that fun.

but it's a lot of work and with other things going on, time is somewhat pushed so unfortunately this is going to be something that's going to take a while to follow through and improve as needed (unless i don't sleep, have no weekends and don't have to read through tomes of posts). which is why just getting Winamp to not crash was all that could be done for v5.65 rather than nag / warning prompts, etc which maybe should have been done but considering i was heavily restricted on time to do this due to the need for a non-crashing update for v5.64, what we've got now is all that could be done.

Sorry, I realize it will take time to deal with these kind of difficult issues. I know you and whatever other dev help available are pushed by what is currently going on.

I don't want to add to the stress, so I'll slow down my comments on these kind of issues and redirect them to trying to help those with issues due to operator error or getting Winamp to do what it is currently capable of doing (at least as far as I better understand those kind of things :) ). I also did not realize this thread was only about 'virgin' scans.

A lot has been accomplished this year, including the correction of some long standing bugs. Respect for that work and the effort it took should not be ignored or taken lightly because of these few remaining difficult issues.

You know how it is, the more we users get, the more we want. Winamp will never be prefect, but it is very, very good and consistently getting better, imo.