Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How can I speed up autoplaylists initialization? (Read 4602 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How can I speed up autoplaylists initialization?

My configuration takes almost 10 minutes to start up, of which about 9'20" just for autoplaylists initialization. Almost all my playlists are autoplaylists. As time goes by my library grows bigger and the number of autoplaylists increases, making the start up time ever longer.

What is there on the hardware side that can be upgraded in order to speed up the start-up and the query process, which I guess is what actually slows down the autoplaylists initialization?

I'm about to upgrade my pc and I'd really like to improve this performance, but I don't really know what benefits most, weather a faster cpu, more cores, more RAM, faster discs, more empty storage for cache memory? Where should I invest the most money in regards to this very issue?

For reference, this is my present set up:
CPU: intel core i3-3220 (2 cores; 4 threads; 3,30 Ghz; no turbo boost)
RAM: 8GB DDR3 1600 MHz
system drive: SSD 850 evo, 250GB
library music drive: WD Caviar Green 2TB
OS: windows 10 pro


I'm late

Re: How can I speed up autoplaylists initialization?

Reply #1 –
1. You haven't posted foobar version. In v 1.3.17 it loads much slower than in v 1.4 beta. This was listed in v 1.4 release notes and I can confirm it.
2. With WD Green you can't do much. Those are slow disks - got 2 of them and know it from experience. Older 2TB is even slower than newer 3TB.
3. Personally I tried foo_disk_cache but honestly I can't tell if this makes difference or not. It still resides there in plugins...

Re: How can I speed up autoplaylists initialization?

Reply #2 –
You are speed-limited by the CPU. More cores and core speed helps. RAM speed may have some small effect too but if you have enough to hold the database adding more won't help. Hard drive speed only matters for loading the database, should be pretty much instant. The WD Greens will only be a bottleneck when performing full media library rescan from them.

Re: How can I speed up autoplaylists initialization?

Reply #3 –
Quote from: EpicForever on
1. You haven't posted foobar version. In v 1.3.17 it loads much slower than in v 1.4 beta. This was listed in v 1.4 release notes and I can confirm it.
I'm running v 1.4 beta 13, but you're right: I noticed a slight improvement since one of the previous beta versions, can't remember which one exactly, something like 30 seconds less, which is almost a 5% better performance.

Quote from: EpicForever on
2. With WD Green you can't do much. Those are slow disks - got 2 of them and know it from experience. Older 2TB is even slower than newer 3TB.
Definitely. Furthermore I had to compress the disc because I was running out of space, which is the main reason I need to upgrade (I ran out of sata ports). Actually, NTFS compression of the music library didn't make foobar2000 noticeably slower, but I get a lot of "HDD wakeup" warnings since then.
The WD green were chosen for budget and noise reasons and it really looks like there is no real alternative to expensive solid state drive storage if you want speed and silence. Perhaps I should deal with it...

Quote from: EpicForever on
3. Personally I tried foo_disk_cache but honestly I can't tell if this makes difference or not. It still resides there in plugins...
I just tested foo_disk_cache (thanks for this, I wasn't aware of its existence) and autoplaylists were initialized in 9' 16", which is only 4 seconds less, well within a margin of error. However jscript panels initialization, which accounted for most of the remaining start up time, is a lot quicker.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #4 –
Quote from: Case on
More cores and core speed helps.
I'll be definitely going for a last generation multicore, but I can't make my mind up between the two more cores I get from AMD and the few hundreds more of MHz I get from Intel. If I get it right, operating on foobar2000 metadb has the same hardware requirements of any database manager and I've always been told more cores are useful in regards to query complexity, whereas clock speed is useful in regards to the size of the recordsets. Is this true? If it is, I'd say foobar2000 metadb can be very large, but an autoplaylist format is by no mean a complex query, therefore Intel would be the better choice. Any thoughts?

Quote from: Case on
Hard drive speed only matters for loading the database, should be pretty much instant. The WD Greens will only be a bottleneck when performing full media library rescan from them.
I can experience speed disc limits, though, when tagging, which I guess implies a partial rescan of the library. Updating tags, and even opening the properties tab, has become much slower over time and of course even more since I compressed the library. This is not a marginal problem because I spend a lot of time updating my pretty complex tag system. What's even worse, adding, moving or renaming any file or folder in the library can freeze the player for quite a while.
I experienced freezes also when running other applications that scan the library, such as beatunes. Running foobar2000 while a beatunes analysis has been launched in the background is not possible with my present hardware, and beatunes can take a few days to analyze the whole library.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #5 –
Do you really need that many autoplaylists though? Can't you exchange them for some Columns UI filter panels or Facets panels? I'd imagine theres a great deal of redundancy between them. Marc2003 also had some WSH/Jscript panel mod script where you could basically bookmark queries and recreate them on the fly.

Re: How can I speed up autoplaylists initialization?

Reply #6 –
Quote from: Daeron on
Do you really need that many autoplaylists though? Can't you exchange them for some Columns UI filter panels or Facets panels? I'd imagine theres a great deal of redundancy between them. Marc2003 also had some WSH/Jscript panel mod script where you could save autoplaylists queries and basically create pre-set autoplaylists on the fly.

Yes, I do need them. I actually use playlists only, they're more like thematic radio stations to me. I've been using filter panels and I presently use EsPlaylists as a library viewer with many views and layers based on my primary selection parameters, but both of them are not as handy when it comes to complex queries that I want to keep trace of.
I was actually thinking of trying out a jscript solution like the one you said Marc2003 had and I probably will, but I still would like to understand if the autoplaylist initialization actually resembles a usual query process of any DBMS and what hardware should I look after most, when upgrading my PC, in order to make this kind of process as fast as possible.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #8 –
I'm curious about the level of complexity you have in your Autoplaylist properties:
do you mind to share a couple of the "complicated" ones?

I neves used Autoplaylists because, at the time, it was quite hard, or even not possible
to change their definitions/properties after they were created.

I was a fan of Playlist Tree (as buggy and slow as it were) because I could create
very powerful queries that would let me easily have many different
views of the content of my library (20k tracks, with an average of 15 tags each,
many of them multivalue). It took two or three minutes to get FB started
but that on an Atom 330 @ 1.6GHz; 1.5GB Ram; W7 32bit SP1.

Now I use primarily "Library tree view" panel (CUI), a partial port
of Playlist Tree: it's quite fast and you can easily twick queries and formatting
of the views. On the same Atom PC, it takes 20secs to start FB with about 50 views.

When I need to change query often, as for "tagging sessions", I tend to use Facets,
as a floating window: I find it really amazing, fast and powerful.

For "rating", "mood" (we call it "situation") and "tempo", and a few of other
"check tags" (Look for sheet music; bad quality; broken; uncorrect tagging; etc)
I set up an ATI RW remote control with Girder 4 that let me add those tags
from my armchair while listening a track, without having to "exit" the listening
state, to go to the PC and add the tags.

When I want to search something very specific, I use Quicksearch: another
piece software ingenuity condensed into such a small GUI.

Thanks.

Re: How can I speed up autoplaylists initialization?

Reply #9 –
Quote from: fabiospark on
I'm curious about the level of complexity you have in your Autoplaylist properties:
do you mind to share a couple of the "complicated" ones?

I neves used Autoplaylists because, at the time, it was quite hard, or even not possible
to change their definitions/properties after they were created.

Code: [Select]
(GENRE IS garage punk OR %genre_family% IS punk OR GENRE IS punk blues OR GENRE IS noise rock OR GENRE IS post-punk OR GENRE IS gothic rock OR GENRE IS dream pop OR GENRE IS shoegaze OR ARTIST IS velvet underground OR ARTIST IS lou reed OR ARTIST IS pere ubu) AND (%date% PRESENT) AND NOT FAVOURITE IS 1 AND %added% DURING LAST 6 WEEKS SORT DESCENDING BY %added%

Would I know how to create such a selection on the fly, using library search tools or library selectors, filter panels, facets or whatever? Of course I would, and I actually do most of the times, but writing it down as an autoplaylist format means taking notes and being able to pick up the work from where I left. As you correctly pointed out, the ability to edit the autoplaylist format after it is created, is crucial. Library selections are very handy, but they are lost once you change selection or you shut down foobar2000.
For example the above autoplaylist was created at first because I wanted to compare some genres, but as I listened to it, I removed some of the genres and added others, because I realized it made more sense for my thesis. Then I realized it might have been useful adding some artists regardless of the genre tag. Then I decided I wanted to listen to the playlist in chronological order, therefore the date tag was mandatory, etc... This edits did not happen all at once, but over time as I listened to it. Saving the query as an autoplaylist also means keeping track of the listening progress. Most of these playlists last several weeks and even months, but foobar2000 will resume the reproduction from where I stopped, whenever I will reactivate the playlist.

Of course this level of complexity is ridiculous compared to SQL queries you can find in a proper DBMS, but it is actually more complex than it looks, because the %genere_family% field is a customdb field, therefore the query is based on a sort of join clause. Needless to say, most of my autoplaylists are based on genre classification, where customdb is highly involved.
For the record, the above playlist is initialized in 43 seconds.


Quote from: fabiospark on
I was a fan of Playlist Tree (as buggy and slow as it were) because I could create
very powerful queries that would let me easily have many different
views of the content of my library (20k tracks, with an average of 15 tags each,
many of them multivalue). It took two or three minutes to get FB started
but that on an Atom 330 @ 1.6GHz; 1.5GB Ram; W7 32bit SP1.

I wouldn't blame it all on the atom. If I remove all my autoplaylists, start-up time is only 16 seconds, including the initialization of a dozen jscript panels. But that's my point, and I'd really appreciate a developers' hint here: how helpful would a faster processor be? Would I be better off trading some MHz for more cores or viceversa? Should I save money on the CPU and rather invest in memory? memory bandwidth? memory size? Should I look at disc speed instead?

All in all I don't think there's anything wrong with this behavior, foobar2000 works great, it is a lightweight and fast player. Actually foobar2000 can run queries and initialize autoplaylists faster than any other player I tested (most of which can't even handle autoplaylists, let alone tag customization options, which can make autoplaylists really powerful). Fact is I abuse of autoplaylists, I'm aware of it, and I'm looking forward to understand what hardware I need to support at best my bad habits.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #10 –
Quote from: davideleo on
Of course this level of complexity is ridiculous compared to SQL queries you can find in a proper DBMS, but it is actually more complex than it looks, because the %genere_family% field is a customdb field, therefore the query is based on a sort of join clause. Needless to say, most of my autoplaylists are based on genre classification, where customdb is highly involved.
And that is most likely the main reason for the long time of initializing the autoplaylists. Queries with using customdb fields are signifantly slower than queries, which are only run against the media library.

Re: How can I speed up autoplaylists initialization?

Reply #11 –
Quote from: fbuser on
Quote from: davideleo on
Of course this level of complexity is ridiculous compared to SQL queries you can find in a proper DBMS, but it is actually more complex than it looks, because the %genere_family% field is a customdb field, therefore the query is based on a sort of join clause. Needless to say, most of my autoplaylists are based on genre classification, where customdb is highly involved.
And that is most likely the main reason for the long time of initializing the autoplaylists. Queries with using customdb fields are signifantly slower than queries, which are only run against the media library.

I'm sure it is. So, what is your advise to speed it up? Do you agree with Case that I'm limited primarily by the CPU? And should I trade cores for clock speed or viceversa (I can't decide between an i7-8700 or a ryzen 7 2700)? Am I right supposing I should look into DBMS hardware requirements?
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #12 –
You might want to consider that something inherently flawed won't be so easily solved by just throwing more horsepower at it (it will help, but it might not grant you the magnitude of gains you are wishing for).

Comparing clock speeds between different architectures is somewhat pointless (4GHz on an old Pentium 4 is not the same as on a modern CPU), so you can't just assume higher is better. Maybe you meant single threaded performance?

Cores in general is a complicated topic and you shouldn't just be looking at what a single application will be doing. Surely you won't dedicate a high-end CPU to just create your autoplaylist faster? What else will you be doing on that PC? Gaming? Number crunching? How many of your important software support scaling on multiple cores properly? How many of them will be running concurrently? Did you look at the whole platform costs (motherboard, RAM, CPU cooler, potential upgrade path later)? These to me sound like much more important questions to answer. As far as handling your autoplaylists either of those CPUs will do just fine.

If they don't, you will really have to start addressing your setup in the first place. Probably by testing what happens if you only create autoplaylists one at a time on the fly (such as marc2003's method, or facets have queries you can save too). If that doesn't work, I'd reconsider how (and why) you are doing customdb fields in the first place. Third, the example query you have listed implies to me that there might be something specific about that group of elements (genres, genres families and the artists etc) you listed. If there is, you could tag them as such so you can retrieve them quickly later.

The bottom line is that to me your situation seems like a weird edge-case scenario of combination of things not working all that well together, so I wouldn't expect the devs to be able to give you much pointers without them replicating the exact scenario and starting to swap components in and out to see which mitigates it to what extent. The best candidate for that is - unfortunately - you.

Re: How can I speed up autoplaylists initialization?

Reply #13 –
Daeron, I totally agree with all you wrote, so let me make it clear that I'm not asking for an IT support on the foobar2000 forum. I already thought my hardware through elsewhere and I don't need to go over it here. Neither am I asking assistance for my foobar2000 setup, which I'm actually happy with and is not flawed as you assume, it is just slow on start up.
I'm asking how are computer resources used by the query process and what is the optimal hardware configuration to speed it up.

P.S.
I'm not upgrading my hardware for foobar2000, but since I am upgrading it anyway, I just want to make sure no stone is left unturned to make autoplaylist initialization as fast as possible.
I'm late
 

Re: How can I speed up autoplaylists initialization?

Reply #14 –
Just to clarify I meant flawed in terms of interfering with your usage (slow). I wasn't questioning whether the functionality provided is something you want.

To sum it up I would personally expect the worst. As I recall foo_customdb is fairly old and not maintained by now, meaning any newer SQLite improvements are not available and what's there might be poorly implemented as well. On top of that many problems might just come from the DB design itself and probably would need a lot of work to work around (if possible at all) to not slow down.

Meaning basically that I wouldn't expect it to scale well at all on multiple cores. I personally think it's generally a good idea to assume that applications won't properly use your additional cores unless proven otherwise. It might also just be limited by your I/O. You could try giving it only one of your CPU cores and see how that affects startup times (and CPU utilization). You could also move a portion of your library to your SSD, benchmark startup times with only those tracks in your database. Then do the same but using your HDD and extrapolate the results. I wouldn't be surprised if that actually net you much better results alone, while more cores not so much.

In any case there are way too many moving parts here and the only thing you can do is poke at it and see what it likes and what it doesnt, then try to explain why. At least that's how I would approach it.

Re: How can I speed up autoplaylists initialization?

Reply #15 –
Quote from: Daeron on
...I wouldn't expect it to scale well at all on multiple cores. I personally think it's generally a good idea to assume that applications won't properly use your additional cores unless proven otherwise. It might also just be limited by your I/O. You could try giving it only one of your CPU cores and see how that affects startup times (and CPU utilization). You could also move a portion of your library to your SSD, benchmark startup times with only those tracks in your database. Then do the same but using your HDD and extrapolate the results. I wouldn't be surprised if that actually net you much better results alone, while more cores not so much.

Thanks a lot, these very are useful hints!
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #16 –
Quote from: davideleo on
For reference, this is my present set up:
CPU: intel core i3-3220 (2 cores; 4 threads; 3,30 Ghz; no turbo boost)
RAM: 8GB DDR3 1600 MHz
system drive: SSD 850 evo, 250GB
library music drive: WD Caviar Green 2TB
OS: windows 10 pro

I think it's worth updating this old thread because I finally upgraded my PC, which is now so configured:

CPU: intel core i5-8400 (6 cores; 6 threads; 2,80/4,00 Ghz;)
RAM: 32GB DDR4 2400MHz
system drive: WD Black SSD NVMe 500GB
library music drive: WD Red 10TB
OS: windows 10 pro
(Something I forgot to mention is that in my previous set up I was still using sata II ports)

Since the OP the library grew bigger and, right before the upgrade, start up time was over 15 minutes, of which almost 6 declared in the console log for autoplaylist initialization. The extra time was mainly due to foobar2000 unresponsiveness, caused I believe by customdb and panel stack splitter combined.
The improvement with the new set up is amazing: 22 seconds for autoplaylist initialization, according to the console log, and 1,5 minute actual start up time.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #17 –
I saw the reference to foo_disk_cache earlier in this thread.  I also was not aware of this plug-in, and have been unable to find any information about it.  (The only reference Google turns up is this thread.)   Can anyone point me to any documentation or a download location?

Re: How can I speed up autoplaylists initialization?

Reply #18 –
Quote from: fretless on
I saw the reference to foo_disk_cache earlier in this thread.  I also was not aware of this plug-in, and have been unable to find any information about it.  (The only reference Google turns up is this thread.)   Can anyone point me to any documentation or a download location?

Honestly I can't remember trying a component named like that, despite what I wrote 9 months ago. I think I was referring to foo_ramdisk, which I remember installing with no actual improvement (probably because it's meant for something else).
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #19 –
The cache component is called foo_disccache and it won't affect media library scanning or autoplaylist initialization one bit. It tries to allow hard drives to sleep during playback by preloading playing and hopefully next track in advance to memory.

Re: How can I speed up autoplaylists initialization?

Reply #20 –
And I still stand by my initial post where I say disk speed is irrelevant to autoplaylist initialization speed. I actually tested it just now by copying my configuration directory to work where I set up a VPN connection from virtual machine to home and mapped my music drive from home with the same letter as is used in the configuration. foobar2000 starts in 0.6 seconds and autoplaylists are initialized in 0.4 seconds despite the super slow <1 MBps link between the music files and the foobar2000. The player doesn't need to see or read the files to start or initialize autoplaylists. It will slowly verify if there is something new or removed on the background without slowing anything down.

Re: How can I speed up autoplaylists initialization?

Reply #21 –
Quote from: Case on
And I still stand by my initial post where I say disk speed is irrelevant to autoplaylist initialization speed.

It makes sense. My new WD red is bigger, but not really faster than the previous WD greens, even though these were limited by sata II transfer speeds. The task manager shows that the new CPU is doing most of the work during start up, peaking to 100% usage for some twenty seconds and memory usage is actually lower during start-up than during playback.
What puzzles me is that CPU usage was much lower with the previous one, which was a lot less powerful. This suggests that it was bottle-necked by some other hardware component.
I'm late

Re: How can I speed up autoplaylists initialization?

Reply #22 –
Case, thank you for the link.  Satisfied my curiosity.   8)

Re: How can I speed up autoplaylists initialization?

Reply #23 –
What is autoplaylist initialization, by the way? I never really got it. Is it like running the autoplaylist format as a search and than creating a standard playlist out of the search results?
I'm late