I'm experiencing very frequent crashes since my library grew past 200K tracks, but I usually don't ask for support because of the four well known buggy components I use. Today I managed to crash the cleanest possible installation, though. I needed to delete two fields from all library tracks. My configuration is so memory greedy that I wouldn't even be able to open the properties dialog when the whole library is selected, therefore I set up a clean installation for this purpose only. I knew it would take several minutes, so after hitting OK in the properties dialog, I switched to other activities. When later I turned back to check if it was finished, foobar2000 had crashed. Attached is the report.
I feel like we went through this in some other topic already, but:
CPU type, RAM, average RAM usage by foobar when it's not crashing, average free RAM, OS type. You could check if increasing virtual memory/pagefile size helps at all.
The log to me seems like it indeed ran out of RAM, but there are people more qualified than me to answer that.
I feel like we went through this in some other topic already, but:
We actually did, but it was before renewing my PC. I was concerned about start up time then, not crashes. Now I have a more powerful computer and start up time is dramatically reduced, but foobar2000 is crashing a bit too often.
CPU type, RAM, average RAM usage by foobar when it's not crashing, average free RAM, OS type. You could check if increasing virtual memory/pagefile size helps at all.
CPU is an i5-8400 with 32GB of DDR4 RAM, much higher than foobar2000's threshold and I rarely exceed 30% usage. My custom theme requires about 2630 MB of RAM on start up. The clean installation I used to edit the tags, 780 MB. By the time the whole library was processed, in the first attempt to save the changes, it was using almost 3600 MB, but it was not able to write tags on a bunch of files, therefore I clicked on "cancel" and than "save changes", but eventually foobar2000 crashed before completing the second round.
At the end of the day only 32 out of 224000 tracks were not affected by the edits, so foobar2000 was almost finished when it crashed.
The log to me seems like it indeed ran out of RAM, but there are people more qualified than me to answer that.
I cannot understand the report, but I'm pretty sure it ran out of RAM. When this happens on my custom configuration I can blame the extended variables and the custom database plugins, the many autoplaylists based on custom tags, the EsPlaylist panels with multiple layers, my amateurish spider monkey scripts, etc., but this was an installation with no extra components and no autoplaylists, just the library and some core custom settings (I use over 100 standard fields, which I guess require a lot of memory).
What I need to know is if something's wrong with my settings and configuration or if I'm simply hitting foobar2000's memory threshold because of the library size. I have at least as many files to be added to the library as soon as they're tagged and organized properly, but I wonder if I will be able to still use foobar2000 with 500K tracks.
What we have here is a case of foobar2000 running out of 32-bit address space.
No matter how much RAM you have, foobar2000 is a 32-bit application and the amount of address space that it can possibly operate on is physically limited to 4GB, some of which is reserved by the operating system so effectively you have even less.
I'm afraid that current foobar2000 design just doesn't cope with extreme amounts of media well, and it's not something that can be fixed without completely breaking compatibility with all currently available add-on components.
A 64-bit version of foobar2000 could be compiled but would not work with any of the available components, and all it would accomplish would be the mitigation of memory limits, I'd rather work on a total rewrite to deal with this problem completely (SQLite backed database/playlists, instant startup with millions of tracks) if I was to break compatibility, but it's not happening anytime soon.
Thanks for the explanation, Peter. Fact is that despite this limit, foobar2000 is still the fastest player I've tested, as far as library scanning and massive tagging is concerned. Furthermore, no other player comes even close in terms of tags customization, which is just as important for browsing and managing huge libraries. So, from great power... you know :))
BTW, attached is one of the last crash reports I usually get from my custom configuration. Is it also a case of address space exhaustion or is there something else going on that I can take care of?
By the way, after some optimization AndreaT is running foobar2000 with library of 600 000 tracks - https://hydrogenaud.io/index.php/topic,117331.0.html
Even on 32 bit OS
I am running it on Win7 32bit.
Thanks for pointing me to that discussion. I tried editing the LargeFieldsConfig text file, but I didn't have the same luck as AndreaT. Still, I'd like to understand a little better how the LargeFieldsConfig settings affect memory usage. First of all, how does foobar2000 use the cached information? If I set the size limit to 0 for every field, foobar2000 seems to work as usual.
First of all, how does foobar2000 use the cached information? If I set the size limit to 0 for every field, foobar2000 seems to work as usual.
All metadata is normally kept in memory and can be searched and used in titleformatting. The config file allows setting maximum length limits to fields and anything exceeding specified bytelength will get replaced by a single placeholder dot in the memory cache. That can drastically reduce memory consumption under normal operation.
But a component can request the core to return the full metadata. For example properties dialog will always load the full info. So the original scenario where you open properties dialog for all tracks in order to remove a tag will require just as much memory as always.
The config file allows setting maximum length limits to fields and anything exceeding specified bytelength will get replaced by a single placeholder dot in the memory cache. That can drastically reduce memory consumption under normal operation.
But what are the advantages of setting the length limits to other than 0? What reasons do I have not to reduce memory consumption as much as possible?
Case already told you this? Things like being able to search as normal and values not showing as simple dots in lets say your playlist or library viewer? I don't know if a value of 0 is counted as unlimited or not, but try setting it to 1 and you might notice the differences.
My understanding is that said config file was mainly a way to mark spammy fields (logs, lyrics, biography, anything with way too long values etc) that usually have no real business kept in memory 24/7 for every single track in your library under normal usage. If you didn't have an abundance of these to begin with the setting will not help much (and Case already told you the Properties window will load all of these regardless).
I would also guess (admittedly based only on my personal experience) that for the majority of people memory is not an issue and yours is one of the few far edge cases. So setting this extremely low would be a hindrance to most people with no benefits.
For example I have lets say half of your tracks and foobar idles at around 180 MB. This is with a fair amount of plugins, custom layout etc. If I select every track in the library and open the properties window its a whopping 300 MB. Still very far from hitting the 3.6-4GB limit and I haven't even touched the largefieldsconfig.txt or optimized my layout. In that sense I'm somewhat puzzled what's going on in your layout that would already hammer at the limits so hard.
properties dialog will always load the full info
But it only loads full info for N tracks, which is set in File->Preferences->Advanced->Display->Properties dialog->Always preload complete info up to N tracks.
Case already told you this? Things like being able to search as normal and values not showing as simple dots in lets say your playlist or library viewer?
No, what Case said is that dots replace values in the cache memory. From my experience foobar2000 behaves the same whether the cached field-size limit is high or low. I'm sure there must be some scenario where a high cache limit makes a difference, but I haven't figured it out yet.
On the other hand neither did I experience memory usage reduction with low limits, so I guess my configuration simply bypasses those LargeFieldsConfig settings all along.
Fresh install with the numerical values in LargeFieldsConfig.txt set to 1:
(https://i.imgur.com/KsJNdzX.png)
If you don't see this I would suspect that you didn't edit it properly or did not restart, and would only suspect the display elements bypassing/forcing the core to load all values directly last.
In any case the Properties window would show it. And Rollin is right, it actually only preloads up to 100 tracks by default. So as long as you are looking at more than that many tracks, the dots should definitely be visible there if you successfully edited them.
In case someone is curious, memory consumption went down to 60 MB idle and 150 MB with every track selected in the Properties window for me in this scenario.
Now that's weird, I never saw anything like that. Attached you can see the screenshot of a clean DUI installation, with no extra components, nor custom settings. Just to make sure I edited it properly, this is the edited LargeFieldsConfig text:
# Size limit for meta fields that appear in the basic fields list
basicMetaMax=1
# Size limit for meta fields that do not appear in either list
defaultMetaMax=1
# Size limit for tech info fields
infoMax=1
I'm not really sure what to respond to this. My screenshot is definitely how it's supposed to work, so my gut reaction is that you are doing something wrong. But obviously I have no way to confirm or deny it.
That means random questions like, did you not accidentally save it as LargeFieldsConfig.txt.txt, or a complete step-by-step reproduction of what you are doing (e.g. when do you create/edit said file, given a fresh install at first launch will create it, possibly overwriting what you have created/edited).
Here's what I did, step by step (and I'm doing it once more as I'm typing):
- install a new portable application of foobar2000 v1.4.3 and launch it when finished
- apply the main layout of choice (album list + properties because it's listed as first) and close foobar2000
- trash the library folder and replace it with a copy of the library folder from my main set up (this is to skip a new library scan)
- open the LargeFieldsConfig file with notepad
- edit the numeric values as posted above
- close notepad and click on "save" when prompted
- start foobar2000 and watch it behave as usual, with no value replaced by dots
- check if the edits in the LargeFieldsConfig were actually saved (they were)
- trash the library folder and replace it with a copy of the library folder from my main set up (this is to skip a new library scan)
This is why the file has no effect for you. You need to fully rescan the library for the filtering to take effect. In your situation you need to remove the monitored dirs and readd them to see the effect.
Ok, thanks. I just started a new library scan, and the first data already shows up as dots. Will experiment with my custom configuration then.
So, after playing around a bit with the LargeFieldsConfig settings, I managed to spare no more than 200 MB, which is better than nothing, but not safe enough (total memory usage is still well over 2000 MB in idle state).
For example I have lets say half of your tracks and foobar idles at around 180 MB. This is with a fair amount of plugins, custom layout etc. If I select every track in the library and open the properties window its a whopping 300 MB.
@Daeron, are you sure of these numbers? 223742 tracks on a clean DUI installation, with no plugins, no playlists, default core settings and all LargeFieldsConfig values set to 0, require 540 MB on my PC (1440 MB with all music selected and properties dialog open). Your memory usage would be a lot less than proportional.
I have seen it going as low as this in the maybe half an hour I kept my eyes on it (while playing in the background):
(https://i.imgur.com/Ac37Nhk.png)
With tracks selected:
(https://i.imgur.com/Bol6qyM.png)
I think part of the reason is that you have a lot more RAM, so the operating system is less agressive at taking away unused/idle memory from apps. If I actually tinker around in the UI the normal usage would be around 250 MB on average.
A minute after startup I have seen it go up to 400-450 MB territory but that, I suspect, is due foo_dynfil or similar things initializing (and perhaps me opening the properties window).
I don't think you will see any notable improvement until you hammer down on whatever is eating so much memory in your layout (I'm guessing multiple things). Any small foothold you gain via the config tweaks will be quickly undone by what? 1000 tracks added at the current memory/track ratio? If you are predicting your library doubling in size, that is 100% unfeasable using your current setup.
I also have a handful of custom tags but definitely not anywhere close to a hundred like you mentioned.
(https://i.imgur.com/Bol6qyM.png)
Well, if the items numbered are tracks, they are closer to one third than half of my library, which at least makes the memory values a little more consistent.
I don't think you will see any notable improvement until you hammer down on whatever is eating so much memory in your layout (I'm guessing multiple things). Any small foothold you gain via the config tweaks will be quickly undone by what? 1000 tracks added at the current memory/track ratio? If you are predicting your library doubling in size, that is 100% unfeasable using your current setup.
500K tracks is more than a prediction, I already have the files on my computer, they are just not in the library. My prediction is 1 million tracks in the next one or two years. And that's the real issue. I know what components and settings require the most memory in my custom configuration, and I already have some ideas on how to replace them. Basically with javascript, which for a non-professional developer like I am means a lot of work and studying. That's ok, I actually enjoy it and I'm willing to invest my time in it if it's worth it. But if foobra200 can't handle such a big library even without customization, than I probably should look somewhere else, I just have no clue where.
I think I tried every free music player available for windows and nothing comes close to foobar2000 in terms of customization. f I have to stick to a standard player, I'm better off with MusicBee, which handled my 500K tracks easily and it's fully packed with a lot of nice features, but it's still just a music player. I can customize the theme or the layout in most other players, but what I'm trying to build is rather a sort of multimedia music encyclopedia, where the most important side is the metadatabase customization. Unfortunately I think there is no alternative to foobar2000 in this regard, at least no alternative music player. Perhaps I should turn to some other kind of software, but once more I have no clue.
If you have any suggestion, it's welcome.
At the scale and purpose you mention I would probably wonder if you might be better off with just a database that is independent of the files themselves (think musicbrainz, discogs, freedb or other similar sites). But I'm guessing you would also like to listen to them, so that doesn't make it any easier.
You could try something like the m-tags component (foo_tags), but I'm guessing it'll load the values to memory the same or worse as usual and I doubt other players support it. I haven't used MusicBee in ages so don't really remember how customizable it is, but maybe you could pair that with something like MP3Tag for editing. Again, don't remember if it's any good, as foobar for me solves both problems and I don't have the memory issues, so haven't looked much elsewhere.
Otherwise I'm not sure how aggressively your caching has been set up in foobar to save that 200 MB, but maybe you can strip it to the bare minimum (just enough to find the artist, but all details would need to be revealed manually) to save more. If this is intended to be an encyclopedia and with the growth you are describing I'm assuming you don't exactly keep going back to update tags constantly. So maybe all you really need is 1) being able to find artists 2) leaving a simple marker whether it is considered 'finished' or not.
It's quite the opposite. I'm constantly checking and editing tags as I listen on. Not only track by track, but on large subsets of the library, checking for consistency of related data. Relating music through tags and being able to browse the library through these relations is my main purpose. The more articulate and refined these relations are, the better. Roon Labs is probably the closest to what I have in mind that I have come across, but besides being pretty expensive, it relies on their own database, whereas I have to be in total control of the data. I don't want to rely on external databases and their way to understand and classify music, because I know better. That's the whole point of my project, I want to be the author of said encyclopedia and I'm trying to build a tool to help me write it.
What could make sense, though, is separating the database from the player. A proper database manager would give me the freedom to articulate, query and display the data the way I want. I have been looking for a multimedia database manager, by which I mean a database manager with built-in features for reading and writing metadata as well as scanning and indexing audio files, but they are usually client/server based and pretty expensive. I might go down that route once I realize it's the only possible one, but for now I'm still searching for more friendly and possibly free solutions. Multimedia features can be implemented in some of the most common DBMS for local use, but it really feels like reinventing the wheel. The integration of music player and metadatabse you get with a software like foobar2000 is hard to trade off.
Just to clarify I meant discogs and similar sites as working examples of the database-only approach being executed, not necessarily in terms of using their site for your needs. (Although the question of trying to unify efforts into a single database vs everyone is just doing their own thing is an interesting debate in itself, that I'm not planning to get into: https://xkcd.com/927/)
I personally would be pretty hesitant to buy any of the expensive custom made things unless you can try them beforehand and it's exactly what you want (which would be very rare). The sites I mentioned are probably just using an SQL-based database server and some basic frontend website that can query and show the data. In that sense you could probably find a lot of tutorials/working examples on how to build a simple site like that, which you could slowly expand to suit your needs. Obviously this means basically taking on a web development project by yourself, but if you eventually want to publish this for others to see, it might coincide with your plans pretty well. And it's probaly how the other, now better known websites started out as well.
The sites I mentioned are probably just using an SQL-based database server and some basic frontend website that can query and show the data. In that sense you could probably find a lot of tutorials/working examples on how to build a simple site like that, which you could slowly expand to suit your needs. Obviously this means basically taking on a web development project by yourself, but if you eventually want to publish this for others to see, it might coincide with your plans pretty well. And it's probaly how the other, now better known websites started out as well.
I see. Well, an HTML interface for an ordinary database is not a big deal indeed, the real issue is the synchronization with the actual metadata and the possibility to send single tracks and playlists to a player. Rather than sites like discogs or musicbrainz I think I should understand how streaming services like spotify are structured on the server side, but I don't even know where to start searching.
As I mentioned earlier, there are DBMS with built-in media features, like oracle multimedia, well... actually oracle mulltimedia is the only one I know of. Could this be the way? Or is there some other kind of software I'm ignoring or overlooking?
HTML would mainly just be used to display static pages, the actual communication, queries and processing is generally handled by something like PHP. Your project touches on multiple subjects, but individually and in principle they are all relatively simple.
For example an SQL server like MySQL can hold your actual metadata, PHP can talk to it, parse the data and pass it on to your HTML site to be displayed.
If you want to link all this to actual files, you will have to create some kind of uploader or directory watcher that keeps track of your music files, assigns a unique ID to each track and syncs their location and ID with the database.
Then you need another thing that can play music files in a browser based on the location it gets from the DB with the right ID.
In practice this is a LOT of work but that is probably your best bet. I'm not familiar with the specific Oracle product you mention, but I personally wouldn't touch anything they have, with very few exceptions.
You could also hunt down similar projects on Github to get a headstart or to expand the functionality of an existing one.
(Also in reality you would probably want a login system so all the resources aren't just freely exposed to everyone on the internet, you would probably also want a playlist editor etc, so the list will keep growing on.)
I personally wouldn't touch anything they have, with very few exceptions.
Apologies if this is off-topic, but can I ask you why?
I've been working for years with Oracle db and I think it's one of the best RDBMS out there.
Just curious if there are specific technical reasons to advise against its use.
If you want to link all this to actual files, you will have to create some kind of uploader or directory watcher that keeps track of your music files, assigns a unique ID to each track and syncs their location and ID with the database.
That's the crucial part and the one that I really have no idea how to deal with. Please, if you have any link to online resources on this topic, even just some basic introduction, post it, because I can't find any. That's why I mentioned oracle multimedia, because as far as I read, it's the only database manager with such functionalities already implemented (even though they're rumored to be dismissed). But it was kind of a joke, given the price of their licences.
The reason I mentioned spotify and streaming services is only because of the size of their database, but I'm not planning to stream music on the web, nor do I need a multi-user environment. What I have in mind is more like a mixture of a standard music player and a software like encarta.
I've been working for years with Oracle db and I think it's one of the best RDBMS out there.
Do you have any experience with oracle multimedia?
Do you have any experience with oracle multimedia?
No, sorry. But looks like it's no longer supported (https://docs.oracle.com/en/database/oracle/oracle-database/18/upgrd/terminal-release-oracle-multimedia.html#GUID-DBA90E57-2BD8-4CEE-A74B-BFC889B78057), so I wouldn't recommend using it today.
That's the crucial part and the one that I really have no idea how to deal with. Please, if you have any link to online resources on this topic, even just some basic introduction, post it, because I can't find any. That's why I mentioned oracle multimedia, because as far as I read, it's the only database manager with such functionalities already implemented (even though they're rumored to be dismissed). But it was kind of a joke, given the price of their licences.
You mentioned Spotify, most successful companies like that will have an engineering blog where they discuss the infrastructure and tech stack they are using, for example:
https://labs.spotify.com/2013/03/15/backend-infrastructure-at-spotify/
https://labs.spotify.com/2019/03/25/building-spotifys-new-web-player/
But your problems will be very different from them as their biggest issue is scalability and yours is not. I don't have any guides top of my head that could help you.
I would personally try to give each file/album a unique ID that will remain unchanged no matter what (e.g. DAVID-0001 or something similar to MusicBrainz ID (https://wiki.musicbrainz.org/MusicBrainz_Identifier)). Then the actual music files can be stored in those folders, the metadata independently edited while the ID will link back to the files. The uploader part of your site would increment this ID with each upload, move them to the corresponding folder physically and either create an empty database row (with said ID as primary key) or ask you to fill out details like artist name and whatever else you want during the upload process. Your player part would know you want to play tracks from artist BOB, which it could look up had a unique ID of DAVID-0096, so the file should be located at 'yourserver.com/DAVID-0096/track 1.mp3'. Hope that helps.
If you break down this mechanism to smaller parts (How do I move files? How do I connect to the database? How do I write a new row to the database with an auto incrementing ID? How do I retrieve and show data from the database on the website? How do I play an mp3 file located at a specified URL in the browser?) and slowly stitch them together then it's definitely doable. Always start with a barebones proof of concept and slowly work your way up to something you would want to use daily, if the direction seems promising. I don't know how much development experience you have, so this might be really obvious or not at all to you.
You could also hunt down people from sites you like (perhaps not Spotify, but smaller hobby projects with similar goals to yours) and ask what they settled on. For example if someone made a 'Metal encyclopedia' and you want to do a 'Classical music encyclopedia' there's no conflict of interest, so I'd imagine those people would be helpful and maybe even allow you to reuse parts of their site.
Apologies if this is off-topic, but can I ask you why?
Oracle used to be very good without many alternatives. Now there are. Meanwhile they are busy using their entrenchment for: exploitative and expensive lincensing, terrible support, suing people, vendor locking, buying competitor products and burying them. Your mileage may vary. This is less so about any one specific product being good or bad (e.g. Oracle DB) and more so about everything going on around it. Obviously always worth considering all your options, so perhaps I should have said 'I would generally be very hesitant choosing them'.
I don't know how much development experience you have, so this might be really obvious or not at all to you.
I'm pretty confident with SQL and database architecture, much less in front-end programming (I only worked with application-level languages so far, one of which is jscript for foobar2000), but the basic issue I have is with neither of the two: I'm stuck on the synchronization of the database with the metadata in the actual audio files. How do you even do that? Is there something like the Oracle multimedia feature available as a module for other DBMS? Or do I have to implement it with my own code? If the latter, I might be better off waiting for
@Peter to switch to a proper database engine in 2059 ( :P ).
The static identifier would basically sidestep that issue (the metadata inside the actual tracks would never be used, instead, you would read that from the SQL DB and simply point the player part of your website to the statically named .mp3 or whatever). I would avoid trying to write metadata to tracks manually as every format has their own quirks which is probably a pain to maintain (although there might be code libraries that help with that somewhat). If you specifically want a multimedia DB that does it all, I'm not familiar with them, so can't help you.
I would avoid trying to write metadata to tracks manually as every format has their own quirks which is probably a pain to maintain (although there might be code libraries that help with that somewhat). If you specifically want a multimedia DB that does it all, I'm not familiar with them, so can't help you.
Yes, that is precisely what I'm looking for. I just found out about taglib and other such libraries, so at least the fog is clearing up a little.
This is less so about any one specific product being good or bad (e.g. Oracle DB) and more so about everything going on around it.
Ok, fair enough.
I'm not familiar with the support of other RDBMS, so can't comment on that.
License cost can be a problem, but perhaps it is worth mentioning that Oracle also offers a free version of their db: it has some limitations, but for small projects / personal use it's often more than enough.