Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: memory leaks while scanning media library (Read 5379 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

memory leaks while scanning media library

How to reproduce:
1. install fresh foobar2000 1.5.5.
2. add folder "modarchive_2007_official_snapshot_120000_modules" (download torrents from this page) to media library.
3. wait for 15-20 minutes.
4. notice huge memory allocation in task manager. It reaches ~4G and after this random crashes/ANR's occured.

What in that torrent? Just zip archives with another zip archives inside with mod/xm/it, etc files inside which are not supported in default foobar2000 installation.
I noticed this leaks while developing foo_input_zxtune plugin.
I tested it with crt debug leaks - no leaks in my code.
I disabled my plugin - still leaks.
I reinstalled fresh foobar2000 - still leaks.
I tried older versions of foobar2000 - all leaks from 1.5 to 1.3, (unable to install 1.2 and lower on my system).
Also i noticed huge exception throwing for each unsupported file inside archives. May be this is a reason?
Also why don't you publish debug symbols for foobar2000? Open source?  :))

Re: memory leaks while scanning media library

Reply #1 –
Hi, of course it happens. And no, it's not a memory leak. It literally has to unpack all of these files, and it keeps them all open while it's scanning them, and they're all unpacked to memory, not disk.

Oh, and they'll be unpacked every time you go to play one of them.

Really, don't use the archive component for this sort of madness. Unpack the whole lot of files to a Linux server, and play them directly.

I say Linux server, because this archive set is known to contain at least a few files which differ from each other only in capitalization, and therefore, cannot be properly unpacked to a case-insensitive file system. And foo_unpack uses case-insensitive matching when asked to unpack something again, so while it may get indexed properly the first time, it'll never play properly again, as it will pick the first match in the whole archive.

Re: memory leaks while scanning media library

Reply #2 –
Quote from: kode54 on
It literally has to unpack all of these files, and it keeps them all open while it's scanning them, and they're all unpacked to memory, not disk.

Ok, the hugest archive is modarchive_2007_official_snapshot_120000_modules\T\TH.zip with 970 Mb size.
It's a .zip and it's not solid archive. So why foobar2000 unpack all in memory?
Even if so, 970 Mb unpacked to 992 Mb (because .zip's inside). It's < 2 Gb.

One more question about 64 bit foobar2000. I readed explanations from Peter why it's not 64 bit.
But now is 2020 year. 64 bit is native for OS (not 32). And i want all programs to be 64 bit on my 64 bit Windows.

Also about foobar2000 1.6.
>Default archive reader now supports 7-zip format.
I hope it will use C++ LZMA SDK, because C API was unable to unpack even 200 Mb solid 7z archives because of huge allocations.
I wrote foo_unpack_7z_ex because of that.

Re: memory leaks while scanning media library

Reply #3 –
64 bit will eliminate support for all existing plugins, requiring them all to be recompiled and possibly modified to support the 64 bit target. This eliminates all plugins whose authors have stopped updating them.

Re: memory leaks while scanning media library

Reply #4 –
Thanks for the bug reports.
Going 64bit would only allow for even more extreme memory usage, not fix the actual problem.
Archive reader needs rewriting to use temp files and not memory, which will eventually happen, I just don't have time for this right now.
Microsoft Windows: We can't script here, this is bat country.

Re: memory leaks while scanning media library

Reply #5 –
Quote from: kode54 on
64 bit will eliminate support for all existing plugins, requiring them all to be recompiled and possibly modified to support the 64 bit target. This eliminates all plugins whose authors have stopped updating them.
Yes. I remember days when Windows XP goes to 64 bit. No drivers, ton of bugs and so on. But then old unsupported code just died. 64 bit wins.

Re: memory leaks while scanning media library

Reply #6 –
Quote from: Peter on
Thanks for the bug reports.
Going 64bit would only allow for even more extreme memory usage, not fix the actual problem.
Archive reader needs rewriting to use temp files and not memory, which will eventually happen, I just don't have time for this right now.
Thanks for foobar2000!)
64 bit is native for modern CPUs/OSes. Apple killed 32 bit apps support in latest macOS. Ubuntu decided to kill, but reverted decision because of community (will kill soon i think).
For me when i open task manages and see 32 bit apps i think they're old and unsupported.
So i voting for 64 bit foobar2000 with basic plugins for beginning)

Re: memory leaks while scanning media library

Reply #7 –
I can assure you that if there's a proper announcement of Windows dropping support for 32bit apps, you will have a 64bit build of foobar2000 within my next workday as there's nothing stopping it from working, perhaps other than PPHS resampler using ancient Intel libraries.
Until then, I'm afraid I have to deny.
There are memory usage issues that would be partially mitigated by going 64bit. I am working on actual fixes for these problems (huge library/playlist memory usage, slow startup with large playlist/library, archive reader memory usage) that do not come with a tradeoff of breaking all component compatibility.
Microsoft Windows: We can't script here, this is bat country.

Re: memory leaks while scanning media library

Reply #8 –
BTW, some new investigations.
I've unpacked one level of zip's from that torrent.
So i have one zip per module (without nested zips).
After scanning ~130K zips foobar2000 memory usage is ~2Gb.

Re: memory leaks while scanning media library

Reply #9 –
What if you unzip all the zips and throw the mods at foobar2000 with no zip at all?
I added 20GB of zipped Bandcamp downloads to playlist and saw no evidence of a leak.
What module decoder are you using?

I'm getting the torrent you linked, will try the exact same content myself shortly.
Microsoft Windows: We can't script here, this is bat country.

Re: memory leaks while scanning media library

Reply #10 –
OK, problem reproduced, fixing. Thanks for reporting.
Microsoft Windows: We can't script here, this is bat country.

Re: memory leaks while scanning media library

Reply #11 –
Fixed in 1.6 beta 13. Fix will be included in v1.5.6 also.
Microsoft Windows: We can't script here, this is bat country.
 

Re: memory leaks while scanning media library

Reply #13 –
Quote from: Peter on
OK, problem reproduced, fixing. Thanks for reporting.
Huge thanks for fixing!)
Sorry for delayed answer.