Skip to main content
Topic: Crash report (Read 2010 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Crash report

I'm experiencing very frequent crashes since my library grew past 200K tracks, but I usually don't ask for support because of the four well known buggy components I use. Today I managed to crash the cleanest possible installation, though. I needed to delete two fields from all library tracks. My configuration is so memory greedy that I wouldn't even be able to open the properties dialog when the whole library is selected, therefore I set up a clean installation for this purpose only. I knew it would take several minutes, so after hitting OK in the properties dialog, I switched to other activities. When later I turned back to check if it was finished, foobar2000 had crashed. Attached is the report.
I'm late

Re: Crash report

Reply #1
I feel like we went through this in some other topic already, but:

CPU type, RAM, average RAM usage by foobar when it's not crashing, average free RAM, OS type. You could check if increasing virtual memory/pagefile size helps at all.

The log to me seems like it indeed ran out of RAM, but there are people more qualified than me to answer that.

Re: Crash report

Reply #2
I feel like we went through this in some other topic already, but:
We actually did, but it was before renewing my PC.  I was concerned about start up time then, not crashes. Now I have a more powerful computer and start up time is dramatically reduced, but foobar2000 is crashing a bit too often.


CPU type, RAM, average RAM usage by foobar when it's not crashing, average free RAM, OS type. You could check if increasing virtual memory/pagefile size helps at all.
CPU is an i5-8400 with 32GB of DDR4 RAM, much higher than foobar2000's threshold and I rarely exceed 30% usage. My custom theme requires about 2630 MB of RAM on start up. The clean installation I used to edit the tags, 780 MB. By the time the whole library was processed, in the first attempt to save the changes, it was using almost 3600 MB, but it was not able to write tags on a bunch of files, therefore I clicked on "cancel" and than "save changes", but eventually foobar2000 crashed before completing the second round.
At the end of the day only 32 out of 224000 tracks were not affected by the edits, so foobar2000 was almost finished when it crashed.


The log to me seems like it indeed ran out of RAM, but there are people more qualified than me to answer that.
I cannot understand the report, but I'm pretty sure it ran out of RAM. When this happens on my custom configuration I can blame the extended variables and the custom database plugins, the many autoplaylists based on custom tags, the EsPlaylist panels with multiple layers, my amateurish spider monkey scripts, etc., but this was an installation with no extra components and no autoplaylists, just the library and some core custom settings (I use over 100 standard fields, which I guess require a lot of memory).

What I need to know is if something's wrong with my settings and configuration or if I'm simply hitting foobar2000's memory threshold because of the library size. I have at least as many files to be added to the library as soon as they're tagged and organized properly, but I wonder if I will be able to still use foobar2000 with 500K tracks.
I'm late

Re: Crash report

Reply #3
What we have here is a case of foobar2000 running out of 32-bit address space.

No matter how much RAM you have, foobar2000 is a 32-bit application and the amount of address space that it can possibly operate on is physically limited to 4GB, some of which is reserved by the operating system so effectively you have even less.

I'm afraid that current foobar2000 design just doesn't cope with extreme amounts of media well, and it's not something that can be fixed without completely breaking compatibility with all currently available add-on components.

A 64-bit version of foobar2000 could be compiled but would not work with any of the available components, and all it would accomplish would be the mitigation of memory limits, I'd rather work on a total rewrite to deal with this problem completely (SQLite backed database/playlists, instant startup with millions of tracks) if I was to break compatibility, but it's not happening anytime soon.

Re: Crash report

Reply #4
Thanks for the explanation, Peter. Fact is that despite this limit, foobar2000 is still the fastest player I've tested, as far as library scanning and massive tagging is concerned. Furthermore, no other player comes even close in terms of tags customization, which is just as important for browsing and managing huge libraries. So, from great power... you know  :))
BTW, attached is one of the last crash reports I usually get from my custom configuration. Is it also a case of address space exhaustion or is there something else going on that I can take care of?
I'm late


Re: Crash report

Reply #6
Thanks for pointing me to that discussion. I tried editing the LargeFieldsConfig text file, but I didn't have the same luck as AndreaT. Still, I'd like to understand a little better how the LargeFieldsConfig settings affect memory usage. First of all, how does foobar2000 use the cached information? If I set the size limit to 0 for every field, foobar2000 seems to work as usual.
I'm late

Re: Crash report

Reply #7
First of all, how does foobar2000 use the cached information? If I set the size limit to 0 for every field, foobar2000 seems to work as usual.
All metadata is normally kept in memory and can be searched and used in titleformatting. The config file allows setting maximum length limits to fields and anything exceeding specified bytelength will get replaced by a single placeholder dot in the memory cache. That can drastically reduce memory consumption under normal operation.
But a component can request the core to return the full metadata. For example properties dialog will always load the full info. So the original scenario where you open properties dialog for all tracks in order to remove a tag will require just as much memory as always.

Re: Crash report

Reply #8
The config file allows setting maximum length limits to fields and anything exceeding specified bytelength will get replaced by a single placeholder dot in the memory cache. That can drastically reduce memory consumption under normal operation.

But what are the advantages of setting the length limits to other than 0? What reasons do I have not to reduce memory consumption as much as possible?
I'm late

Re: Crash report

Reply #9
Case already told you this? Things like being able to search as normal and values not showing as simple dots in lets say your playlist or library viewer? I don't know if a value of 0 is counted as unlimited or not, but try setting it to 1 and you might notice the differences.

My understanding is that said config file was mainly a way to mark spammy fields (logs, lyrics, biography, anything with way too long values etc) that usually have no real business kept in memory 24/7 for every single track in your library under normal usage. If you didn't have an abundance of these to begin with the setting will not help much (and Case already told you the Properties window will load all of these regardless).

I would also guess (admittedly based only on my personal experience) that for the majority of people memory is not an issue and yours is one of the few far edge cases. So setting this extremely low would be a hindrance to most people with no benefits.

For example I have lets say half of your tracks and foobar idles at around 180 MB. This is with a fair amount of plugins, custom layout etc. If I select every track in the library and open the properties window its a whopping 300 MB. Still very far from hitting the 3.6-4GB limit and I haven't even touched the largefieldsconfig.txt or optimized my layout. In that sense I'm somewhat puzzled what's going on in your layout that would already hammer at the limits so hard.

Re: Crash report

Reply #10
properties dialog will always load the full info
But it only loads full info for N tracks, which is set in File->Preferences->Advanced->Display->Properties dialog->Always preload complete info up to N tracks.

Re: Crash report

Reply #11
Case already told you this? Things like being able to search as normal and values not showing as simple dots in lets say your playlist or library viewer?
No, what Case said is that dots replace values in the cache memory. From my experience foobar2000 behaves the same whether the cached field-size limit is high or low. I'm sure there must be some scenario where a high cache limit makes a difference, but I haven't figured it out yet.
On the other hand neither did I experience memory usage reduction with low limits, so I guess my configuration simply bypasses those LargeFieldsConfig settings all along.
I'm late

Re: Crash report

Reply #12
Fresh install with the numerical values in LargeFieldsConfig.txt set to 1:

If you don't see this I would suspect that you didn't edit it properly or did not restart, and would only suspect the display elements bypassing/forcing the core to load all values directly last.

In any case the Properties window would show it. And Rollin is right, it actually only preloads up to 100 tracks by default. So as long as you are looking at more than that many tracks, the dots should definitely be visible there if you successfully edited them.

In case someone is curious, memory consumption went down to 60 MB idle and 150 MB with every track selected in the Properties window for me in this scenario.

Re: Crash report

Reply #13
Now that's weird, I never saw anything like that. Attached you can see the screenshot of a clean DUI installation, with no extra components, nor custom settings. Just to make sure I edited it properly, this is the edited LargeFieldsConfig text:
Quote
# Size limit for meta fields that appear in the basic fields list
basicMetaMax=1
# Size limit for meta fields that do not appear in either list
defaultMetaMax=1
# Size limit for tech info fields
infoMax=1
I'm late

Re: Crash report

Reply #14
I'm not really sure what to respond to this. My screenshot is definitely how it's supposed to work, so my gut reaction is that you are doing something wrong. But obviously I have no way to confirm or deny it.

That means random questions like, did you not accidentally save it as LargeFieldsConfig.txt.txt, or a complete step-by-step reproduction of what you are doing (e.g. when do you create/edit said file, given a fresh install at first launch will create it, possibly overwriting what you have created/edited).

Re: Crash report

Reply #15
Here's what I did, step by step (and I'm doing it once more as I'm typing):
- install a new portable application of foobar2000 v1.4.3 and launch it when finished
- apply the main layout of choice (album list + properties because it's listed as first) and close foobar2000
- trash the library folder and replace it with a copy of the library folder from my main set up (this is to skip a new library scan)
- open the LargeFieldsConfig file with notepad
- edit the numeric values as posted above
- close notepad and click on "save" when prompted
- start foobar2000 and watch it behave as usual, with no value replaced by dots
- check if the edits in the LargeFieldsConfig were actually saved (they were)



I'm late

Re: Crash report

Reply #16
- trash the library folder and replace it with a copy of the library folder from my main set up (this is to skip a new library scan)
This is why the file has no effect for you. You need to fully rescan the library for the filtering to take effect. In your situation you need to remove the monitored dirs and readd them to see the effect.

Re: Crash report

Reply #17
Ok, thanks. I just started a new library scan, and the first data already shows up as dots. Will experiment with my custom configuration then.
I'm late

Re: Crash report

Reply #18
So, after playing around a bit with the LargeFieldsConfig settings, I managed to spare no more than 200 MB, which is better than nothing, but not safe enough (total memory usage is still well over 2000 MB in idle state).



For example I have lets say half of your tracks and foobar idles at around 180 MB. This is with a fair amount of plugins, custom layout etc. If I select every track in the library and open the properties window its a whopping 300 MB.

@Daeron, are you sure of these numbers? 223742 tracks on a clean DUI installation, with no plugins, no playlists, default core settings and all LargeFieldsConfig values set to 0, require 540 MB on my PC (1440 MB with all music selected and properties dialog open). Your memory usage would be a lot less than proportional.

I'm late

 

Re: Crash report

Reply #19
I have seen it going as low as this in the maybe half an hour I kept my eyes on it (while playing in the background):


With tracks selected:


I think part of the reason is that you have a lot more RAM, so the operating system is less agressive at taking away unused/idle memory from apps. If I actually tinker around in the UI the normal usage would be around 250 MB on average.

A minute after startup I have seen it go up to 400-450 MB territory but that, I suspect, is due foo_dynfil or similar things initializing (and perhaps me opening the properties window).

I don't think you will see any notable improvement until you hammer down on whatever is eating so much memory in your layout (I'm guessing multiple things). Any small foothold you gain via the config tweaks will be quickly undone by what? 1000 tracks added at the current memory/track ratio? If you are predicting your library doubling in size, that is 100% unfeasable using your current setup.

I also have a handful of custom tags but definitely not anywhere close to a hundred like you mentioned.

Re: Crash report

Reply #20

Well, if the items numbered are tracks, they are closer to one third than half of my library, which at least makes the memory values a little more consistent.


I don't think you will see any notable improvement until you hammer down on whatever is eating so much memory in your layout (I'm guessing multiple things). Any small foothold you gain via the config tweaks will be quickly undone by what? 1000 tracks added at the current memory/track ratio? If you are predicting your library doubling in size, that is 100% unfeasable using your current setup.

500K tracks is more than a prediction, I already have the files on my computer, they are just not in the library. My prediction is 1 million tracks in the next one or two years. And that's the real issue. I know what components and settings require the most memory in my custom configuration, and I already have some ideas on how to replace them. Basically with javascript, which for a non-professional developer like I am means a lot of work and studying. That's ok, I actually enjoy it and I'm willing to invest my time in it if it's worth it. But if foobra200 can't handle such a big library even without customization, than I probably should look somewhere else, I just have no clue where.

I think I tried every free music player available for windows and nothing comes close to foobar2000 in terms of customization. f I have to stick to a standard player, I'm better off with MusicBee, which handled my 500K tracks easily and it's fully packed with a lot of nice features, but it's still just a music player. I can customize the theme or the layout in most other players, but what I'm trying to build is rather a sort of multimedia music encyclopedia, where the most important side is the metadatabase customization. Unfortunately I think there is no alternative to foobar2000 in this regard, at least no alternative music player. Perhaps I should turn to some other kind of software, but once more I have no clue.
If you have any suggestion, it's welcome.
I'm late

Re: Crash report

Reply #21
At the scale and purpose you mention I would probably wonder if you might be better off with just a database that is independent of the files themselves (think musicbrainz, discogs, freedb or other similar sites). But I'm guessing you would also like to listen to them, so that doesn't make it any easier.

You could try something like the m-tags component (foo_tags), but I'm guessing it'll load the values to memory the same or worse as usual and I doubt other players support it. I haven't used MusicBee in ages so don't really remember how customizable it is, but maybe you could pair that with something like MP3Tag for editing. Again, don't remember if it's any good, as foobar for me solves both problems and I don't have the memory issues, so haven't looked much elsewhere.

Otherwise I'm not sure how aggressively your caching has been set up in foobar to save that 200 MB, but maybe you can strip it to the bare minimum (just enough to find the artist, but all details would need to be revealed manually) to save more. If this is intended to be an encyclopedia and with the growth you are describing I'm assuming you don't exactly keep going back to update tags constantly. So maybe all you really need is 1) being able to find artists 2) leaving a simple marker whether it is considered 'finished' or not.

Re: Crash report

Reply #22
It's quite the opposite. I'm constantly checking and editing tags as I listen on. Not only track by track, but on large subsets of the library, checking for consistency of related data. Relating music through tags and being able to browse the library through these relations is my main purpose. The more articulate and refined these relations are, the better. Roon Labs is probably the closest to what I have in mind that I have come across, but besides being pretty expensive, it relies on their own database, whereas I have to be in total control of the data. I don't want to rely on external databases and their way to understand and classify music, because I know better. That's the whole point of my project, I want to be the author of said encyclopedia and I'm trying to build a tool to help me write it.

What could make sense, though, is separating the database from the player. A proper database manager would give me the freedom to articulate, query and display the data the way I want. I have been looking for a multimedia database manager, by which I mean a database manager with built-in features for reading and writing metadata as well as scanning and indexing audio files, but they are usually client/server based and pretty expensive. I might go down that route once I realize it's the only possible one, but for now I'm still searching for more friendly and possibly free solutions. Multimedia features can be implemented in some of the most common DBMS for local use, but it really feels like reinventing the wheel. The integration of music player and metadatabse you get with a software like foobar2000 is hard to trade off.
I'm late

Re: Crash report

Reply #23
Just to clarify I meant discogs and similar sites as working examples of the database-only approach being executed, not necessarily in terms of using their site for your needs. (Although the question of trying to unify efforts into a single database vs everyone is just doing their own thing is an interesting debate in itself, that I'm not planning to get into: https://xkcd.com/927/)

I personally would be pretty hesitant to buy any of the expensive custom made things unless you can try them beforehand and it's exactly what you want (which would be very rare). The sites I mentioned are probably just using an SQL-based database server and some basic frontend website that can query and show the data. In that sense you could probably find a lot of tutorials/working examples on how to build a simple site like that, which you could slowly expand to suit your needs. Obviously this means basically taking on a web development project by yourself, but if you eventually want to publish this for others to see, it might coincide with your plans pretty well. And it's probaly how the other, now better known websites started out as well.

Re: Crash report

Reply #24
The sites I mentioned are probably just using an SQL-based database server and some basic frontend website that can query and show the data. In that sense you could probably find a lot of tutorials/working examples on how to build a simple site like that, which you could slowly expand to suit your needs. Obviously this means basically taking on a web development project by yourself, but if you eventually want to publish this for others to see, it might coincide with your plans pretty well. And it's probaly how the other, now better known websites started out as well.

I see. Well, an HTML interface for an ordinary database is not a big deal indeed, the real issue is the synchronization with the actual metadata and the possibility to send single tracks and playlists to a player. Rather than sites like discogs or musicbrainz I think I should understand how streaming services like spotify are structured on the server side, but I don't even know where to start searching.
As I mentioned earlier, there are DBMS with built-in media features, like oracle multimedia, well... actually oracle mulltimedia is the only one I know of. Could this be the way? Or is there some other kind of software I'm ignoring or overlooking?
I'm late

 
SimplePortal 1.0.0 RC1 © 2008-2019