Skip to main content
Topic: How can I speed up autoplaylists initialization? (Read 593 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How can I speed up autoplaylists initialization?

My configuration takes almost 10 minutes to start up, of which about 9'20" just for autoplaylists initialization. Almost all my playlists are autoplaylists. As time goes by my library grows bigger and the number of autoplaylists increases, making the start up time ever longer.

What is there on the hardware side that can be upgraded in order to speed up the start-up and the query process, which I guess is what actually slows down the autoplaylists initialization?

I'm about to upgrade my pc and I'd really like to improve this performance, but I don't really know what benefits most, weather a faster cpu, more cores, more RAM, faster discs, more empty storage for cache memory? Where should I invest the most money in regards to this very issue?

For reference, this is my present set up:
CPU: intel core i3-3220 (2 cores; 4 threads; 3,30 Ghz; no turbo boost)
RAM: 8GB DDR3 1600 MHz
system drive: SSD 850 evo, 250GB
library music drive: WD Caviar Green 2TB
OS: windows 10 pro


I'm late

Re: How can I speed up autoplaylists initialization?

Reply #1
1. You haven't posted foobar version. In v 1.3.17 it loads much slower than in v 1.4 beta. This was listed in v 1.4 release notes and I can confirm it.
2. With WD Green you can't do much. Those are slow disks - got 2 of them and know it from experience. Older 2TB is even slower than newer 3TB.
3. Personally I tried foo_disk_cache but honestly I can't tell if this makes difference or not. It still resides there in plugins...

Re: How can I speed up autoplaylists initialization?

Reply #2
You are speed-limited by the CPU. More cores and core speed helps. RAM speed may have some small effect too but if you have enough to hold the database adding more won't help. Hard drive speed only matters for loading the database, should be pretty much instant. The WD Greens will only be a bottleneck when performing full media library rescan from them.

Re: How can I speed up autoplaylists initialization?

Reply #3
1. You haven't posted foobar version. In v 1.3.17 it loads much slower than in v 1.4 beta. This was listed in v 1.4 release notes and I can confirm it.
I'm running v 1.4 beta 13, but you're right: I noticed a slight improvement since one of the previous beta versions, can't remember which one exactly, something like 30 seconds less, which is almost a 5% better performance.

2. With WD Green you can't do much. Those are slow disks - got 2 of them and know it from experience. Older 2TB is even slower than newer 3TB.
Definitely. Furthermore I had to compress the disc because I was running out of space, which is the main reason I need to upgrade (I ran out of sata ports). Actually, NTFS compression of the music library didn't make foobar2000 noticeably slower, but I get a lot of "HDD wakeup" warnings since then.
The WD green were chosen for budget and noise reasons and it really looks like there is no real alternative to expensive solid state drive storage if you want speed and silence. Perhaps I should deal with it...

3. Personally I tried foo_disk_cache but honestly I can't tell if this makes difference or not. It still resides there in plugins...
I just tested foo_disk_cache (thanks for this, I wasn't aware of its existence) and autoplaylists were initialized in 9' 16", which is only 4 seconds less, well within a margin of error. However jscript panels initialization, which accounted for most of the remaining start up time, is a lot quicker.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #4
More cores and core speed helps.
I'll be definitely going for a last generation multicore, but I can't make my mind up between the two more cores I get from AMD and the few hundreds more of MHz I get from Intel. If I get it right, operating on foobar2000 metadb has the same hardware requirements of any database manager and I've always been told more cores are useful in regards to query complexity, whereas clock speed is useful in regards to the size of the recordsets. Is this true? If it is, I'd say foobar2000 metadb can be very large, but an autoplaylist format is by no mean a complex query, therefore Intel would be the better choice. Any thoughts?

Hard drive speed only matters for loading the database, should be pretty much instant. The WD Greens will only be a bottleneck when performing full media library rescan from them.
I can experience speed disc limits, though, when tagging, which I guess implies a partial rescan of the library. Updating tags, and even opening the properties tab, has become much slower over time and of course even more since I compressed the library. This is not a marginal problem because I spend a lot of time updating my pretty complex tag system. What's even worse, adding, moving or renaming any file or folder in the library can freeze the player for quite a while.
I experienced freezes also when running other applications that scan the library, such as beatunes. Running foobar2000 while a beatunes analysis has been launched in the background is not possible with my present hardware, and beatunes can take a few days to analyze the whole library.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #5
Do you really need that many autoplaylists though? Can't you exchange them for some Columns UI filter panels or Facets panels? I'd imagine theres a great deal of redundancy between them. Marc2003 also had some WSH/Jscript panel mod script where you could basically bookmark queries and recreate them on the fly.

Re: How can I speed up autoplaylists initialization?

Reply #6
Do you really need that many autoplaylists though? Can't you exchange them for some Columns UI filter panels or Facets panels? I'd imagine theres a great deal of redundancy between them. Marc2003 also had some WSH/Jscript panel mod script where you could save autoplaylists queries and basically create pre-set autoplaylists on the fly.

Yes, I do need them. I actually use playlists only, they're more like thematic radio stations to me. I've been using filter panels and I presently use EsPlaylists as a library viewer with many views and layers based on my primary selection parameters, but both of them are not as handy when it comes to complex queries that I want to keep trace of.
I was actually thinking of trying out a jscript solution like the one you said Marc2003 had and I probably will, but I still would like to understand if the autoplaylist initialization actually resembles a usual query process of any DBMS and what hardware should I look after most, when upgrading my PC, in order to make this kind of process as fast as possible.
I'm late


Re: How can I speed up autoplaylists initialization?

Reply #8
I'm curious about the level of complexity you have in your Autoplaylist properties:
do you mind to share a couple of the "complicated" ones?

I neves used Autoplaylists because, at the time, it was quite hard, or even not possible
to change their definitions/properties after they were created.

I was a fan of Playlist Tree (as buggy and slow as it were) because I could create
very powerful queries that would let me easily have many different
views of the content of my library (20k tracks, with an average of 15 tags each,
many of them multivalue). It took two or three minutes to get FB started
but that on an Atom 330 @ 1.6GHz; 1.5GB Ram; W7 32bit SP1.

Now I use primarily "Library tree view" panel (CUI), a partial port
of Playlist Tree: it's quite fast and you can easily twick queries and formatting
of the views. On the same Atom PC, it takes 20secs to start FB with about 50 views.

When I need to change query often, as for "tagging sessions", I tend to use Facets,
as a floating window: I find it really amazing, fast and powerful.

For "rating", "mood" (we call it "situation") and "tempo", and a few of other
"check tags" (Look for sheet music; bad quality; broken; uncorrect tagging; etc)
I set up an ATI RW remote control with Girder 4 that let me add those tags
from my armchair while listening a track, without having to "exit" the listening
state, to go to the PC and add the tags.

When I want to search something very specific, I use Quicksearch: another
piece software ingenuity condensed into such a small GUI.

Thanks.

Re: How can I speed up autoplaylists initialization?

Reply #9
I'm curious about the level of complexity you have in your Autoplaylist properties:
do you mind to share a couple of the "complicated" ones?

I neves used Autoplaylists because, at the time, it was quite hard, or even not possible
to change their definitions/properties after they were created.

Code: [Select]
(GENRE IS garage punk OR %genre_family% IS punk OR GENRE IS punk blues OR GENRE IS noise rock OR GENRE IS post-punk OR GENRE IS gothic rock OR GENRE IS dream pop OR GENRE IS shoegaze OR ARTIST IS velvet underground OR ARTIST IS lou reed OR ARTIST IS pere ubu) AND (%date% PRESENT) AND NOT FAVOURITE IS 1 AND %added% DURING LAST 6 WEEKS SORT DESCENDING BY %added%

Would I know how to create such a selection on the fly, using library search tools or library selectors, filter panels, facets or whatever? Of course I would, and I actually do most of the times, but writing it down as an autoplaylist format means taking notes and being able to pick up the work from where I left. As you correctly pointed out, the ability to edit the autoplaylist format after it is created, is crucial. Library selections are very handy, but they are lost once you change selection or you shut down foobar2000.
For example the above autoplaylist was created at first because I wanted to compare some genres, but as I listened to it, I removed some of the genres and added others, because I realized it made more sense for my thesis. Then I realized it might have been useful adding some artists regardless of the genre tag. Then I decided I wanted to listen to the playlist in chronological order, therefore the date tag was mandatory, etc... This edits did not happen all at once, but over time as I listened to it. Saving the query as an autoplaylist also means keeping track of the listening progress. Most of these playlists last several weeks and even months, but foobar2000 will resume the reproduction from where I stopped, whenever I will reactivate the playlist.

Of course this level of complexity is ridiculous compared to SQL queries you can find in a proper DBMS, but it is actually more complex than it looks, because the %genere_family% field is a customdb field, therefore the query is based on a sort of join clause. Needless to say, most of my autoplaylists are based on genre classification, where customdb is highly involved.
For the record, the above playlist is initialized in 43 seconds.


I was a fan of Playlist Tree (as buggy and slow as it were) because I could create
very powerful queries that would let me easily have many different
views of the content of my library (20k tracks, with an average of 15 tags each,
many of them multivalue). It took two or three minutes to get FB started
but that on an Atom 330 @ 1.6GHz; 1.5GB Ram; W7 32bit SP1.

I wouldn't blame it all on the atom. If I remove all my autoplaylists, start-up time is only 16 seconds, including the initialization of a dozen jscript panels. But that's my point, and I'd really appreciate a developers' hint here: how helpful would a faster processor be? Would I be better off trading some MHz for more cores or viceversa? Should I save money on the CPU and rather invest in memory? memory bandwidth? memory size? Should I look at disc speed instead?

All in all I don't think there's anything wrong with this behavior, foobar2000 works great, it is a lightweight and fast player. Actually foobar2000 can run queries and initialize autoplaylists faster than any other player I tested (most of which can't even handle autoplaylists, let alone tag customization options, which can make autoplaylists really powerful). Fact is I abuse of autoplaylists, I'm aware of it, and I'm looking forward to understand what hardware I need to support at best my bad habits.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #10
Of course this level of complexity is ridiculous compared to SQL queries you can find in a proper DBMS, but it is actually more complex than it looks, because the %genere_family% field is a customdb field, therefore the query is based on a sort of join clause. Needless to say, most of my autoplaylists are based on genre classification, where customdb is highly involved.
And that is most likely the main reason for the long time of initializing the autoplaylists. Queries with using customdb fields are signifantly slower than queries, which are only run against the media library.

Re: How can I speed up autoplaylists initialization?

Reply #11
Of course this level of complexity is ridiculous compared to SQL queries you can find in a proper DBMS, but it is actually more complex than it looks, because the %genere_family% field is a customdb field, therefore the query is based on a sort of join clause. Needless to say, most of my autoplaylists are based on genre classification, where customdb is highly involved.
And that is most likely the main reason for the long time of initializing the autoplaylists. Queries with using customdb fields are signifantly slower than queries, which are only run against the media library.

I'm sure it is. So, what is your advise to speed it up? Do you agree with Case that I'm limited primarily by the CPU? And should I trade cores for clock speed or viceversa (I can't decide between an i7-8700 or a ryzen 7 2700)? Am I right supposing I should look into DBMS hardware requirements?
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #12
You might want to consider that something inherently flawed won't be so easily solved by just throwing more horsepower at it (it will help, but it might not grant you the magnitude of gains you are wishing for).

Comparing clock speeds between different architectures is somewhat pointless (4GHz on an old Pentium 4 is not the same as on a modern CPU), so you can't just assume higher is better. Maybe you meant single threaded performance?

Cores in general is a complicated topic and you shouldn't just be looking at what a single application will be doing. Surely you won't dedicate a high-end CPU to just create your autoplaylist faster? What else will you be doing on that PC? Gaming? Number crunching? How many of your important software support scaling on multiple cores properly? How many of them will be running concurrently? Did you look at the whole platform costs (motherboard, RAM, CPU cooler, potential upgrade path later)? These to me sound like much more important questions to answer. As far as handling your autoplaylists either of those CPUs will do just fine.

If they don't, you will really have to start addressing your setup in the first place. Probably by testing what happens if you only create autoplaylists one at a time on the fly (such as marc2003's method, or facets have queries you can save too). If that doesn't work, I'd reconsider how (and why) you are doing customdb fields in the first place. Third, the example query you have listed implies to me that there might be something specific about that group of elements (genres, genres families and the artists etc) you listed. If there is, you could tag them as such so you can retrieve them quickly later.

The bottom line is that to me your situation seems like a weird edge-case scenario of combination of things not working all that well together, so I wouldn't expect the devs to be able to give you much pointers without them replicating the exact scenario and starting to swap components in and out to see which mitigates it to what extent. The best candidate for that is - unfortunately - you.

Re: How can I speed up autoplaylists initialization?

Reply #13
Daeron, I totally agree with all you wrote, so let me make it clear that I'm not asking for an IT support on the foobar2000 forum. I already thought my hardware through elsewhere and I don't need to go over it here. Neither am I asking assistance for my foobar2000 setup, which I'm actually happy with and is not flawed as you assume, it is just slow on start up.
I'm asking how are computer resources used by the query process and what is the optimal hardware configuration to speed it up.

P.S.
I'm not upgrading my hardware for foobar2000, but since I am upgrading it anyway, I just want to make sure no stone is left unturned to make autoplaylist initialization as fast as possible.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #14
Just to clarify I meant flawed in terms of interfering with your usage (slow). I wasn't questioning whether the functionality provided is something you want.

To sum it up I would personally expect the worst. As I recall foo_customdb is fairly old and not maintained by now, meaning any newer SQLite improvements are not available and what's there might be poorly implemented as well. On top of that many problems might just come from the DB design itself and probably would need a lot of work to work around (if possible at all) to not slow down.

Meaning basically that I wouldn't expect it to scale well at all on multiple cores. I personally think it's generally a good idea to assume that applications won't properly use your additional cores unless proven otherwise. It might also just be limited by your I/O. You could try giving it only one of your CPU cores and see how that affects startup times (and CPU utilization). You could also move a portion of your library to your SSD, benchmark startup times with only those tracks in your database. Then do the same but using your HDD and extrapolate the results. I wouldn't be surprised if that actually net you much better results alone, while more cores not so much.

In any case there are way too many moving parts here and the only thing you can do is poke at it and see what it likes and what it doesnt, then try to explain why. At least that's how I would approach it.

 

Re: How can I speed up autoplaylists initialization?

Reply #15
...I wouldn't expect it to scale well at all on multiple cores. I personally think it's generally a good idea to assume that applications won't properly use your additional cores unless proven otherwise. It might also just be limited by your I/O. You could try giving it only one of your CPU cores and see how that affects startup times (and CPU utilization). You could also move a portion of your library to your SSD, benchmark startup times with only those tracks in your database. Then do the same but using your HDD and extrapolate the results. I wouldn't be surprised if that actually net you much better results alone, while more cores not so much.

Thanks a lot, these very are useful hints!
I'm late

 
SimplePortal 1.0.0 RC1 © 2008-2018