Skip to main content

Notice

If you are using a Hotmail or Outlook email address, please change it now, as Microsoft is rejecting all email from our service outright.
Topic: Loading properties for 250K files takes much longer and often crashes (Read 649 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Loading properties for 250K files takes much longer and often crashes

After upgrading to version 1.5 Foobar takes now more than 30 minutes to load the properties for 250K files... Before 1.5 there was no progress bar and the operation used to take 2-3 minutes max...
When the operation eventually completes I often get a blank screen, and just from time to time I get the proper information displayed.... However, when getting the right info, Foobar is crashing when I close the window and then I have to start over again loading the info for another 30 minutes or more...
Is there anything I can do to improve this or do I just have to go back to the old version which was working?

Thanks

Re: Loading properties for 250K files takes much longer and often crashes

Reply #1
And the metadata screen came up empty once again... Really can't work with this I'm afraid... :(

Re: Loading properties for 250K files takes much longer and often crashes

Reply #2
Did you try to increase the size limits in LargeFieldsConfig.txt
You can find LargeFieldsConfig.txt in C:\Users\YOURNAME\AppData\Roaming\foobar2000.
Or hold shift in foobar2000, then click file, then browse configuration folder

Re: Loading properties for 250K files takes much longer and often crashes

Reply #3
I did do any change to the file, it's the same as before the upgrade:

# Size limit for meta fields that appear in the basic fields list
basicMetaMax=2000
# Size limit for meta fields that do not appear in either list
defaultMetaMax=1000
# Size limit for tech info fields
infoMax=200

I have already added some large fields to exclude several years ago when I reached the max size of the library files that foobar could load in memory, so large fields should not be loaded:

fieldSpam=ACCURATERIPDISCID
fieldSpam=ACCURATERIPID
fieldSpam=ACCURATERIPRESULT
fieldSpam=ACOUSTID_FINGERPRINT
fieldSpam=ACOUSTID_ID
fieldSpam=ANALYSIS
fieldSpam=DISCSUBTITLE
fieldSpam=FINGERPRINT
fieldSpam=FILE
fieldSpam=MUSICIP_PUID

Re: Loading properties for 250K files takes much longer and often crashes

Reply #4
My library is already split into more than 100 sub-folders in order to increase speed and concurrency, as each folder metadata is stored in a separate DB...




Re: Loading properties for 250K files takes much longer and often crashes

Reply #5
After upgrading to version 1.5 Foobar takes now more than 30 minutes to load the properties for 250K files... Before 1.5 there was no progress bar and the operation used to take 2-3 minutes max...

What version did you have before upgrading? I cannot open the properties dialog for my whole 250K-items library since at least v1.3.
I'm late

Re: Loading properties for 250K files takes much longer and often crashes

Reply #6
I was running 1.4.7 but I ran through many versions before as I've been running foobar for more than 15 years now... The only time I ever had problem was when reaching the limits of memory addressed by foobar because the library was getting too large and I had a lot of large fields to load. But after the introduction of the LargeFieldsConfig.txt file a few years ago, I could exclude those from the library and all was perfect again.
Last summer I was still running on an 11-years old PC with 2 cores and loading properties for 250K files used to take about 4-5 min. Since then I have upgraded to a new PC with an i7-8700 and it went down to just a couple of minutes (of course this involves a lot of disk access too, it's not just CPU based as all changes in headers have to be detected).
Every week when importing new music to my library I open properties for all files to be sure there are no new strange tags included because I like to keep things under control. Until last week when I upgraded to 1.5 it was all working fine.
I am thinking of going back to 1.4.x now because I need this functionality but before, if there is any info I can send to help solve this, I'd rather do that than get stuck on an old version indefinitely....

Re: Loading properties for 250K files takes much longer and often crashes

Reply #7
You are right. For me the only way to open the properties dialog for the whole library is with a perfectly clean foobar2000 instance, with no playlists, no plug-ins, etc.. I tested this on the most recent foobar2000 versions and up to v1.4.8 it takes only a few minutes. v1.5.1 is an agony.
I'm late

Re: Loading properties for 250K files takes much longer and often crashes

Reply #8
Well as a workaround I have installed version 1.4.8 portable (to avoid losing my config settings) and opened all my folders in it. Opening the properties window for all files takes exactly 10 seconds... Yes, I know it's a fresh clean installation with no playlists and extra plugins but still, 10 seconds vs. more than 30 minutes in 1.5.1 can't really be compared in any way... Also 1.5.1 keeps on crashing most of the time for me and I don't get the info even after 30 minutes...

Re: Loading properties for 250K files takes much longer and often crashes

Reply #9
I can also confirm this.

I used the 1.5.1 setup to overwrite the files of my previous 1.4.3 installation. On 150k tracks the properties window went from opening in ~5 seconds to a whopping 36 minutes (I timed it). Waiting for a complete load, closing the properties window, then reopening it was not any faster the second time either (at the hopes that maybe it has to cache all tracks first, but over time it'll be faster).

During the loading I have not seen any increased memory usage (<450 mb), disk usage was pretty low (<1 mbps) as well as CPU usage (~1-7%). Unlike others in this topic, I also never had crashing/instability problems due library size before, nor did I have any after the update. It's just that loading the properties window is extremely slow for some reason on 1.5.1.

For what it's worth, I heard (what I think is) the HDD head seeking or writing a lot more during the loading process on 1.5.1 as opposed to 1.4.3.

Did not test again with clean installs.

Re: Loading properties for 250K files takes much longer and often crashes

Reply #10
https://hydrogenaud.io/index.php?topic=118217.msg975681#msg975681
The old Properties dialog would show only cached information when opening a large group of tracks - it always had some information to show immediately.
This behavior was dropped. Complete info (with lyrics and whatever other huge fields) is always shown.

Re: Loading properties for 250K files takes much longer and often crashes

Reply #11
Then I'm really screwed, I have to rollback to previous version and stay on that one for ever because the new one is unusable for me  :o

Re: Loading properties for 250K files takes much longer and often crashes

Reply #12
Thanks for finding that Rollin, I recalled the change, but I could not find it at: https://foobar2000.org/changelog. And indeed choosing 'Tools > Load complete info' behavior on 1.4.x is roughly identical to 1.5.x default (seeking sounds, loading times and all).

It seems like the recommended practice with 1.5 going forward for large libraries is to increase the cached metadata using LargeFieldsConfig.txt?

This to me sounds like a suicide, no? More stuff cached = more RAM used, and some people here are already hitting out of memory type errors with the LargeFieldsConfig.txt set to drop whatever it can. Wasn't the entire point of that setting to reduce the memory footprint while also increasing search performance? Why exactly is everything force-loaded with the new properties panel? What are we getting in exchange for potentially losing the ability to open the properties panel (within a reasonable timeframe) at all?

I don't consider my library huge, at least certainly haven't had any issues with it before performance-wise, but the way 1.5 works is pretty much a deal breaker for me, for benefits I can't quite see yet. Is there something I'm missing? Is there something in the works and this is just a stop-gap solution for now? Is performance for small libraries the primary target now and this solution works much better for them? Can we get some insight from Peter on why this change was made and what are the benefits of it?

Re: Loading properties for 250K files takes much longer and often crashes

Reply #13
I am using software like MusicIP Mixer which is storing acoustic signatures in the file tags and those fields are HUGE. Before the introduction of LargeFieldsConfig.txt I was hitting the limit of addressable memory for foobar, which still remains a 32-bit program to date... After allocating about 1.8 GB of RAM, foobar couldn't save its library anymore and all changes would be lost upon every program shutdown. With the introduction of LargeFieldsConfig.txt I could add those large fields to the exclusion list and foobar memory went down to 6-700 MB RAM with the library loaded...
If I remove those large fields from the exclusion list in order to cache the metadata and have the properties window working again, then I will hit again the memory limit, so this can't really work for me...
One of the main reason I've been using foobar for so many years was its ability to handle large libraries better than any other software... it seems that ability has been lost now... maybe there is a possibility to add a configuration option somewhere in order to revert to previous behavior for those unfortunate enough to have such large libraries?

Re: Loading properties for 250K files takes much longer and often crashes

Reply #14
Maybe for a quick fix/workaround it will help to install the classic properties component ?

 

Re: Loading properties for 250K files takes much longer and often crashes

Reply #15
Maybe for a quick fix/workaround it will help to install the classic properties component ?

Yes, it does help, thank you very much, I was not aware of that component!
At least now I can can still open the properties window same as before and it does load as fast as it used to ;)

 
SimplePortal 1.0.0 RC1 © 2008-2020