Skip to main content
Topic: Speed tips for users with large libraries (Read 18026 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Speed tips for users with large libraries

I'm running Foobar 9b2 on a Fujitsu Lifebook S6210 with an Intel Centrino 1.6 chip and 768 megs of RAM. Between my laptop and three external hard drives, I have over 50,000 MP3s. I started using Foobar because it was better with large collections but as my collection has grown as of late (maybe it has something to do with the upgrade to 0.90 as well), I've noticed significant slowdown. It's at the point now where I have to wait at least 3-4 minutes for Foobar to load.

I realize that 50,000 is a ridiculously extravagant library but I'm hoping that everyone here can give me some tips to help make things more manageable.

Here's a screen capture of my setup:



You can't see it from here, but I'm running the Browser, Album Art, Track Info, Explorer and Playlist tabs extensions along with the latest Columns UI. In addition I'm also running a number of components in the background such as Audioscrobbler, etc.

So here are my questions. I'm hoping this can turn into a general help thread for Foobar users with large collections.

1. Does the number of components in my components directory affect the overall footprint, even if I'm not necessarily using them? Would it make sense to go through that directory and remove the ones I never use? Also, are any components known as being particularly bad on memory?

2. Are there any other tweaks I can perform within Foobar to bring the load down?

3. Are there any software utilities/memory managers I can use to help lessen the load?

4. Would adding another 768megs of RAM result in a noticeable difference?

Thanks in advance guys -- any expertise/advice you can offer would be huge.

Speed tips for users with large libraries

Reply #1
..., but I'm running the Browser, Album Art, Track Info, Explorer and Playlist tabs extensions....


Try removing those one by one and see if and when the problem gets fixed.

Speed tips for users with large libraries

Reply #2
Try removing the "#title" browser window, that could take alot of ram to filter 50 000 songs there.

Speed tips for users with large libraries

Reply #3
I have 20 000 tracks and I never had any special problem with foobar, but i think that removing especially panels plugins that you never use would help.

The modularity of foobar is here to do that, store them apart, and if you ever find the occasion to use them, put them back in your component folder.

I'm using similar plugins and my foobar only takes 13 Mo of memory and less than 4% of my cpu (between 1 and 2 usually with some peaks at 13%)

You can also try to make a test speed of your UI columns config, with such tracks, having a heavy UI columns config won't help..

Speed tips for users with large libraries

Reply #4
1. Does the number of components in my components directory affect the overall footprint, even if I'm not necessarily using them? Would it make sense to go through that directory and remove the ones I never use? Also, are any components known as being particularly bad on memory?
The number of components is less than important than the exact nature of the components. Components like album art that work on a single item do not contribute much as the ML or playlist size grows, but any kind of ML browser (album list, foo_browser, etc.) do.

2. Are there any other tweaks I can perform within Foobar to bring the load down?
If you regularly use large playlists with tens of thousands of entries, you should consider disabling the total selection time display (status bar) or the total playlist length display (Columns UI playlist switcher). These features require extra memory that notably increases the total memory usage of foobar2000 with playlists of that size, though the effect is still quite small unless your playlists contains hundreds of thousands or even several million entries. I recently did some tests with playlist of that size, it was rather "enlightening".

Another thing that only depends on the total number of unique items in your ML and all playlists is the size of the metadata cache. The current version of foobar2000 does not use a dictionary to store tag values, so if all 20 tracks from an album contain the same 10 KB short story in the comment tag, that will be 200 KB memory for the whole album. So the advice here would be to keep your tags reasonably short.
Binary tags like cover art are not affected by this, since foobar2000 does not cache them in memory; in fact, it currently does not use them at all.

4. Would adding another 768megs of RAM result in a noticeable difference?
It might, but reducing the memory usage by removing components or switching to components with more efficient algorithms (well, not much choice in reality, I admit) generally works better than throwing more hardware at the problem.

Speed tips for users with large libraries

Reply #5
1. Does the number of components in my components directory affect the overall footprint, even if I'm not necessarily using them? Would it make sense to go through that directory and remove the ones I never use? Also, are any components known as being particularly bad on memory?


foo_ui_trackinfo needs some tweeking. BIG TIME. Also make sure you have the latest version. My foobar was using 33% cpu and with configuring ONLY foo_ui_trackinfo i reduced it to 4%. i forget the exact changes needed, but make sure you look into it.

Speed tips for users with large libraries

Reply #6
foo_ui_trackinfo needs some tweeking. BIG TIME. Also make sure you have the latest version. My foobar was using 33% cpu and with configuring ONLY foo_ui_trackinfo i reduced it to 4%. i forget the exact changes needed, but make sure you look into it.


The cpu usage went from ~30% to 1-2% as soon as I set Scroll speed and Scroll end delay to 50000ms

Speed tips for users with large libraries

Reply #7
I have a playlist on a similar scale. I'm using 11 autoplaylists, 2 track info panels, 1 album art, 1 Album list (sorting = path). Using about 75 megs of physical and virtual ram, and about 5 page faults per second. Performance is fine for me, except for the official playcount component, which I've seen take 14 seconds to update one song. I even put a bunch of random effects into my track info panel and modified Navigator fcs.


(slight obfuscation via crap compression)

Maybe when Autoplaylist gets done there'll be a built in speed test for that too?

Speed tips for users with large libraries

Reply #8
The cpu usage went from ~30% to 1-2% as soon as I set Scroll speed and Scroll end delay to 50000ms


Wow. Really great tip. Tried it and the cpu-usage went from 60% to 3% . 
Thanks a lot.

Speed tips for users with large libraries

Reply #9
Also, try setting scroll steps to 1.

Anybody have any idea what scroll steps is and why default is set at 60?  If it eats up CPU like that... It'd make sense to not include it......... or at least default it at a low value.

Speed tips for users with large libraries

Reply #10
I was getting ~60% usage on my Athlon64 3000+ machine with a 45000 length playlist. Turning off the auto metadata update thing (can't remember the exact title from the top of my head) reduced this to essentially zero.

I still notice that the first time I click on something it takes ~10 seconds to spring to life, in which time the UI is completely frozen. After that it's fine. I assume that unlike 0.8.3 it only parses the database properly when some sort of playlist action is performed, as opposed to on startup?

Speed tips for users with large libraries

Reply #11
First of all, thanks to everyone for such amazing tips. Per Foosion's advice, I've removed the memory-intensive panels from Columns UI and noticed an improvement. My Foobar isn't as pretty but it's certainly a little more nimble.

That said, I'm still experiencing pretty significant slowdowns at times, especially when I open and close Foobar. When I open the program it only takes a couple of seconds for the Foobar interface to show up, but I get an hourglass cursor for about 45-60 seconds after that, during which CPU load hovers at around 50% and mem usage hangs around 55k. Similarly, the act of closing Foobar is maddeningly unresponsive -- it often takes upwards of a minute for the program to shut down.

As Seanyseansean noted up top, I also experience pretty major moments of unresponsiveness when surfing around playlists for the first time. It's almost like they need to be worked in every time in order to respond fluidly.

So: are these things cause for alarm/further investigation or should I just consider them an occupational hazard that comes with having such a large library?

Also, Foosion, is there anywhere to check the size of the metadata cache? Do you have any idea for what it should be for a library of my size?

And not to belabor the point, but I am interested in knowing if anyone has had any success speeding up Foobar with the help of any software treaks/memory managers, etc. Thanks guys.

Speed tips for users with large libraries

Reply #12
I'm still experiencing pretty significant slowdowns at times, especially when I open and close Foobar. When I open the program it only takes a couple of seconds for the Foobar interface to show up, but I get an hourglass cursor for about 45-60 seconds after that, during which CPU load hovers at around 50% and mem usage hangs around 55k. Similarly, the act of closing Foobar is maddeningly unresponsive -- it often takes upwards of a minute for the program to shut down.

I had the exact same issues, and when I removed everything except $extra(foobar2000_version) from my "Columns UI/Title formatting/Main window title", the problem went away.



EDIT: Just noticed that hotzenpl0tz gave the same tip in the 3rd post

It's a great tip

foobar2000 + EAC + Burrrn = Happiness

Speed tips for users with large libraries

Reply #13
Thanks Lex. That seems to have helped a bit but it's still lagging pretty hard..

Speed tips for users with large libraries

Reply #14
If you want to get some numbers where a part of the memory or CPU time is used, you can get the Playlist Loader Benchmark (foo_plbench) and the Metadata Cache Statistics (foo_metadbstats) components from my 0.9 page.

foo_plbench can measure the time it takes to read or write a playlist in a given format. If you run it from the main menu ("File/Playlist Loader Benchmark..."), it will use the items in the media library as playlist contents, or if you use the context menu version ("Utils/Playlist Loader Benchmark..."), it will use the selected items.
Warning: Do not try to benchmark M3U or M3U8 reading performance with a non-small number of items unless you have a lot of patience.

foo_metadbstats can compute the estimated memory required to hold textual information for items in the media library and the open playlists. To keep the implementation simple, the computation cannot be aborted currently. The user interface will be unresponsive until the computation is complete.
The most interesting (and easiest to interpret) values are "References", "Strings" and "Estimated Memory". "References" refers to the number of "logical strings", while "Strings" is the number of actually "physical strings". The difference is simple: if you have an ARTIST field on two different files, you will have two logical strings with the value "ARTIST", though they might be stored as a single physical string. Note that this is merely an optimization, two physical strings can have the same value, so if the component displays that there are 700 info name strings, this does not imply that there are actually 700 different info names in the cache. (For programmers: "References" are pointers and "Strings" are chunks of memory containing the characters.)
If I can be bothered to enhance this component sometime in the future, I'll probably think of some more descriptive names for some of the values.
Note that the number of path references is equal to the number of songs.

Speed tips for users with large libraries

Reply #15
Thanks for the tips, good to have for us with a big libary (have one of 120k myself).

Speed tips for users with large libraries

Reply #16
Quote
EDIT: Just noticed that hotzenpl0tz gave the same tip in the 3rd post

It's a great tip


Nah, that tip was yours, mine was regarding the "#title" query he uses with his foo_browser panels

Speed tips for users with large libraries

Reply #17
speeding up the tagz script would be a good idea i think...
does anybody know what's fast?
i think using lot of globals should be faster, but i don't really know.
is using $meta(field) faster than %field%? is there a difference?

Speed tips for users with large libraries

Reply #18
Here are my PLB results for a playlist containing my entire library in FPL. Is this good? I have no idea:

^ writer ^ storage ^  filesize ^  count ^  write ^  read ^
| fpl | memory |  17257564 |  47960 |  0.69256 |  0.22829 |
| fpl | memory |  17257564 |  47960 |  0.68822 |  0.23933 |
| fpl | memory |  17257564 |  47960 |  0.71397 |  0.30153 |
| fpl | file |  17257564 |  47960 |  1.48580 |  0.34418 |
| fpl | file |  17257564 |  47960 |  1.31965 |  0.40997 |
| fpl | file |  17257564 |  47960 |  1.47650 |  0.34762 |

Speed tips for users with large libraries

Reply #19
This is a great topic as I have currently been wondering what I might have done to my foobar.  All I am running is explorer tree, playlist tree, playlist switcher, track info and album art, and my foobar 0.9.1 opens and closes considerably slower than 0.8.3.  While it is not as bad as some other people are reporting here, it is clearly noticeable.  I've tried some of the tips on here and for me, none of them have changed much.  So, I am wondering what is so different with 0.9.1 than 0.8.3 that would cause this?

Speed tips for users with large libraries

Reply #20
hiscores:
I don't know your CPU or harddrive, but those values are within the expected range. Below are the results from my PC, you will notice that my ML is about one seventh the size of yours, and the filesize and I/O speeds have about the same ratio.

^ writer ^ storage ^  filesize ^  count ^  write ^  read ^
| fpl | memory |  2394166 |  7233 |  0.11181 |  0.06606 |
| fpl | memory |  2394166 |  7233 |  0.11331 |  0.02770 |
| fpl | memory |  2394166 |  7233 |  0.11462 |  0.02884 |
| fpl | file |  2394166 |  7233 |  0.20643 |  0.03346 |
| fpl | file |  2394166 |  7233 |  0.18841 |  0.03310 |
| fpl | file |  2394166 |  7233 |  0.20550 |  0.03492 |

It would be nice to be able to compare results from different users in a more organized way, but I don't have time to set up a website for that or something. If someone else feels like doing that, please go ahead.

Edit: By the way, the format exported to the clipboard is DokuWiki table syntax.

Speed tips for users with large libraries

Reply #21
^ writer ^ storage ^  filesize ^  count ^  write ^  read ^
| fpl | memory |  8612980 |  25279 |  0.31804 |  0.09186 |
| fpl | memory |  8612980 |  25279 |  0.31813 |  0.08769 |
| fpl | memory |  8612980 |  25279 |  0.31519 |  0.09090 |
| fpl | file |  8612980 |  25279 |  0.79058 |  0.10218 |
| fpl | file |  8612980 |  25279 |  0.78900 |  0.10328 |
| fpl | file |  8612980 |  25279 |  0.76831 |  0.10928 |

I also have something linear compared to previous results as I have 22,5k of files.

But my startup time is quite long : between 10 and 20 sec. I will try to see why it's that long (no browser nor album list nor playlsist tree)

edit It seems that it's my autoplaylists, it went from 6sec to 1,8sec the rest comes from UI columns and panels(5sec)..


edit2 Well it's strange, I recreated the autoplaylists I had with one slight modificication and now the startup time is always 5,7 sec. I am not complaining but it's weird. I will try to recreate everything as it was before and see what happens.

edit3 Well it seems that huge autoplaylists are not very nice. with 12K tracks in an autoplaylist, it costs nearly 2seconds on startup while reducing the number to 200 tracks costs nearly nothing

Speed tips for users with large libraries

Reply #22
Great topic.
Foosion, can you explain how to get a benchmark with foo_metadbstats ? THe component is installed, but I see no new entry to trigger it, either in preferences, toolbars, or context menu. Thanks

Speed tips for users with large libraries

Reply #23
It adds a main menu command called "View/Metadata Cache Statistics".

Speed tips for users with large libraries

Reply #24
It adds a main menu command called "View/Metadata Cache Statistics".

thanks

Quote
By the way, the format exported to the clipboard is DokuWiki table syntax.

How do you get the data into the clipboard?

 
SimplePortal 1.0.0 RC1 © 2008-2019