Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: (big) library database search speed (Read 9005 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

(big) library database search speed

OK, it is tens of thousands of tracks. And it is on a netbook, but hey, it is still more powerful than an ordinary computer was when fb2k was first released (2 gigabytes RAM). CPU spends 8 seconds on > 90 percent. Is that about what one would expect from a decent database solution?

(All versions I've tried since 1.0. I suppose the UI is not relevant? Using facets nowadays.)

(big) library database search speed

Reply #1 –
Quote from: Porcus on
I suppose the UI is not relevant?


i don't know if you are referring to DUI/CUI or library viewer...if you are referring to the latter, it does make a difference
for example the media library search and album list are considerably faster than facets and
furthermore, lagging seems to be much less if the library selection playlist is not shown

if i were to guess i'd say that most of the lag is due to caching of album art

(big) library database search speed

Reply #2 –
Certainly not the UI, searching is equally slow using a fresh install with minimal configuration (and without art displayed).

I accessed a few computers to try.  A portable with a six-year old Mobile Turion X2 (dual-core) TL-56 @1.8 Ghz and 3GB RAM has acceptable speed, a single-core Pentium Prescott @ 3Ghz with 2GB RAM is not too good, and the Atom N270 @1.6 Ghz appears abysmal. Although fb2k routinely gets allocated 256 MB-esque memory, I just cannot believe the third GB makes the difference.

What are the limiting variables here? :-/

(big) library database search speed

Reply #3 –
fb2k library search is not exactly efficient as it does not rely on any prebuilt indexes; it goes over every item in your library. However, any modern multi-core CPU ought to handle it well enough (within reasonable library sizes) as the process scales up to as many CPU cores as you can throw at it.
Microsoft Windows: We can't script here, this is bat country.

(big) library database search speed

Reply #4 –
Quote from: Peter on
fb2k library search is not exactly efficient as it does not rely on any prebuilt indexes; it goes over every item in your library. However, any modern multi-core CPU ought to handle it well enough (within reasonable library sizes) as the process scales up to as many CPU cores as you can throw at it.


Good to know that I can increase # of cores. In principle good to know that performance could be improved, although I am not sure if that optimization is high on your radar. I didn't expect fb2k to be such a resource hog, though – and I wouldn't be surprised if fb2k users have more files than the average user, more tags per file, and are more prone to use low-power (less noisy) hardware.

(big) library database search speed

Reply #5 –
Quote from: Peter on
fb2k library search is not exactly efficient as it does not rely on any prebuilt indexes; it goes over every item in your library. However, any modern multi-core CPU ought to handle it well enough (within reasonable library sizes) as the process scales up to as many CPU cores as you can throw at it.

I have merely 7k tracks in my library and the search via the Media Library Search window is pretty slow. That's on a reasonable system (i7-920 Nehalem quad core; 6GB RAM; OS, fb2k, and %appdata%\foobar2000 on an SSD; music folders on a pretty fast dedicated HDD).

If I make a typo when putting in my query, it takes several seconds until there's a response to my pressing of the <Backspace> key. Of course, I still love fb2k given the query syntax is so much more powerful than in other media players. but search with other media players is waaaay faster. that's not surprising since most of the competition uses databases (e.g., amarok uses mysql, rhythmbox uses XML, banshee uses SQLite, etc.)

EDIT: also, queries in fb2k seem *very* CPU-intensive. i frequently hear the fan spinning up when running a query.

(big) library database search speed

Reply #6 –
Quote from: rgtb on
that's not surprising since most of the competition uses databases
fb2k uses a database. Just not one that uses prebuilt indexes.


Your problems seem like it is probably unrelated to the rest of the thread.
elevatorladylevitateme

(big) library database search speed

Reply #7 –
Quote from: shakey_snake on
Your problems seem like it is probably unrelated to the rest of the thread.

I certainly didn't post with the intention to hijack. Honestly, I still don't understand in what sense I missed the point of Porcus' thread. I I merely wanted to second that search can be frustratingly slow. (In my case, this applies to all kinds of searches, including simple <any string> searches--the above was merely an example.)

(big) library database search speed

Reply #8 –
Quote from: rgtb on
I have merely 7k tracks in my library and the search via the Media Library Search window is pretty slow. That's on a reasonable system (i7-920 Nehalem quad core; 6GB RAM; OS, fb2k, and %appdata%\foobar2000 on an SSD; music folders on a pretty fast dedicated HDD).


I have 4300 tracks. That's in the same ballpark as your 7K, as opposed to the 80K mentioned earlier.
All searches are instantaneous, and cold startup time is 2 seconds.
Some things:
- I don't use embedded album art (no idea if this is relevant).
- my queries are always simple keywords; never complex.
- no autoplaylists

So foobar is definitely not slow by default for reasonably sized libraries and enormous playlists, and I expect it to be highly performant with at least 15K items, if not far more. There must be factors that severely increase the load, and some of those factors apply to you.

(big) library database search speed

Reply #9 –
Running foobar2000 DUI in WINE on Linux, on an old 2.8GHz P4 with 1GB RAM, I don't have any lag problems on a library with mid to high tens of thousands of tracks, if that's any sort of useful data-point.

(big) library database search speed

Reply #10 –
Quote from: Porcus on
Quote from: Peter on
fb2k library search is not exactly efficient as it does not rely on any prebuilt indexes; it goes over every item in your library. However, any modern multi-core CPU ought to handle it well enough (within reasonable library sizes) as the process scales up to as many CPU cores as you can throw at it.


Good to know that I can increase # of cores. In principle good to know that performance could be improved, although I am not sure if that optimization is high on your radar. I didn't expect fb2k to be such a resource hog, though – and I wouldn't be surprised if fb2k users have more files than the average user, more tags per file, and are more prone to use low-power (less noisy) hardware.



This is getting so tiring that I consider fragmenting my music collection in order to be able to run fb2k on my fanless Atom-based computer. The library search function is really slow.

Could I suggest a proper database? SQLite is public domain. Used in the MIT-licensed Banshee player.

(big) library database search speed

Reply #11 –
I have to ask, why aren't you using Banshee then?

(big) library database search speed

Reply #12 –
Quote from: Case on
I have to ask, why aren't you using Banshee then?


I think you actually know a good reason or two to use fb2k, although database efficiency is certainly not one of them. Hell, I am even considering crippling my library just in order to be able to use it.

Actually, I am using a codec which is known for (and optimized for) decoding saving resources ... and a player that wastes them on an algorithm known to be inefficient.

(big) library database search speed

Reply #13 –
Foobar2000 would need to be reinvented to make use of an SQL database. For starters, the current system stores each track's data as a tree. Each track contains a list of meta key strings, each with one or more values. Then each track also has a list of technical info key strings, each with a single value. Then each track contains ReplayGain adjustments and peaks, stored in binary floating point format.

An SQL database could easily store the ReplayGain data, since tables are typically designed to store a fixed number of columns. However, the technical information supports fields of arbitrary names, so a scheme would need to support storing key/value pairs associated with each track. I suppose it doesn't get much worse with actual metadata, since then you just need to be able to store multiple entries with the same track ID and key name.

Still, I don't see how arranging that data into an SQL database would make querying against the data much faster, except through abuse of mechanisms typically employed by such databases for data organization, such as binary trees.

Foobar2000 indexing components such as the Album List already organize the data into trees for faster and easier searching, but I suppose what you want is for the Media Library Search prompt and its generated auto playlists to be able to reference data faster. For this, appropriate internal use of binary trees could be employed to make queries run faster, all without having to shove all of the data into an SQL database.

(big) library database search speed

Reply #14 –
kode54: thx for the explanation.

I do not really care for what kind of database, only about the end result. (Which as of now is annoying me.)

(big) library database search speed

Reply #15 –
The search is also a bit annoying for me; though I do have a library of unreasonable size. But for me the problem is not the speed of the search itself. But the "search as you type" feature. That makes foobar unresponsive as you start typing the next word since it is still searching for the first word. To me it would be very nice to be able to disable this "search as you type" so that you don't experience this lag until you actually search for what you intended.

(big) library database search speed

Reply #16 –
I suppose one could build a plugin based on the Apache Lucene search library (or rather one of its C++ ports like CLucene or lucene++).

(big) library database search speed

Reply #17 –
The type-ahead-find requires fast fingers yes. Of course you can just start with a quotation mark, but that does not completely resolve the problem when you have multiple phrases.


A couple of different loose ideas:
- multiple libraries. An option to choose with say, @1 or @2 at the beginning of search.
- An option to define a few fields as 'primary' and only enter the secondary if asked. Say, foo looks only in the primary while * HAS foo searches all.

(big) library database search speed

Reply #18 –
Quote from: rgtb on
I have merely 7k tracks in my library and the search via the Media Library Search window is pretty slow. That's on a reasonable system (i7-920 Nehalem quad core; 6GB RAM; OS, fb2k, and %appdata%\foobar2000 on an SSD; music folders on a pretty fast dedicated HDD).


foobar2000 search indexing design is pretty low-tech (always walks over the whole list of tracks), but 9000 tracks on an Intel Atom D525 machine isn't quite enough to cause an annoyance here (worst case 0.1s) - other than UI lag interfering with typing, the next version will deal with this.

It's like you people are doing something special to get such extreme lag; perhaps base64 encoded album art in your tags or really complicated queries?
Microsoft Windows: We can't script here, this is bat country.

Re: (big) library database search speed

Reply #19 –
Resurrecting this thread, as I upgraded my hardware (still fanless, so still not too speedy).  I noticed that "blanking" the search takes a lot of time: 
Now 2.2 seconds, compare to 2.5 (more!) for the single letter 'e';  1.1 for the single letter 'z'; and 0.6 when pasting with ctrl-v the word 'nothing'.
Is it searching or displaying that takes so much time? (Does it really search algorithmically for the empty string? That could be a quick fix?)

Would it by the way be "easy" to implement a "restrict to subset" if one more letter is entered at the end?
(Requires the restriction to be implemented for the "last part" in case of OR/NOT, and must of course check whether the last letter makes the search syntactically valid.)

Quote from: Peter on
It's like you people are doing something special to get such extreme lag; perhaps base64 encoded album art in your tags or really complicated queries?

No complicated queries, but: would it help to enter album art field names in the "Simple search - exclude fields" - and if so, what are those field names?

Re: (big) library database search speed

Reply #20 –
Quote from: Porcus on
Would it by the way be "easy" to implement a "restrict to subset" if one more letter is entered at the end?
Only for simple queries which do not contain keywords. With keywords it gets tricky even though it should still be easier than figuring out whether there is a subset relationship between two arbitrary queries. It only requires the abstract syntax trees of the two queries and a little interest in formal logic. ;)

Quote from: Porcus on
Is it searching or displaying that takes so much time?
Those are not the only stages in the search process. Of course it depends on the concrete search function or plugin and your settings. Two common ones you didn't mention are sorting and grouping. Both of them take longer the more search results there are, and both evaluate a title formatting script for each found track.

Re: (big) library database search speed

Reply #21 –
Quote from: Jan S. on
To me it would be very nice to be able to disable this "search as you type" so that you don't experience this lag until you actually search for what you intended.
Did you try foo_quicksearch? It has an autosearch delay setting. Someone else also asked for only searching in specific fields, which it does as well.