7-zip reader issues (from "1.6.5 beta crashes")

Topic: 7-zip reader issues (from "1.6.5 beta crashes") (Read 1601 times) previous topic - next topic - Topic derived from 1.6.5 beta 2 crashes....

0 Members and 1 Guest are viewing this topic.

7-zip reader issues (from "1.6.5 beta crashes")

2021-03-26 15:15:06

I noticed different behavior of fb 1.6.5_b2 vs 1.6.5_b3 when working with large (348 MB) 7z archives.
For version 1.6.5_b3, the archive file is placed in the playlist as one item without unpacking.
For version 1.6.5_b2, the contents of the archive file are unpacked and placed in the playlist.
Tested on clean portable versions.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #1 – 2021-03-26 20:32:38

Topic split because this needs detailed explaining and may not be relevant to the crash issue from the original topic.

7-zip's solid mode requires extreme amounts of memory to unpack. This is problematic to satisfy in foobar2000's 32bit memory space. Even if required amounts of memory are available, it's impossible for 7-zip reader to allocate continuous memory blocks of the desired size.

I originally made foobar2000 v1.6.5 7-zip reader take as much memory as it asks for. This was OK during normal operation, but turned extremely ugly when trying to convert files from an archive, using multiple threads. Not only it ran out of memory space, but caused random stuff to fail until foobar2000 restart, due to heap fragmentation issues. Think of 32bit address space as of a flat 4GB file, where continuous chunks are allocated for various operations, but the free space cannot be defragmented so after a certain chain of events you end with a state where every multi megabyte allocation request fails.

For now I mitigated this by restricting the amount of memory that 7-zip reader is allowed to take, which unfortunately causes some files to fail reading.

Proper solution is to go 64-bit (and break all existing plug-ins) or read 7-zip out-of-process (slower, creates temporary files). I don't currently want to go either way (even though 64-bit is inevitable eventually), foobar2000 is a music player, not an archiver.

I might revise this logic, maybe block concurrent extraction of 7-zips, but there's basically no good solution to this.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #2 – 2021-03-26 20:52:51

You can do out of process like we already do things like ASIO and I do VST plugins: You can command an external executable to read a particular archive for access, not necessarily unpacking it to memory yet, and then command it two-way over pipes, requesting specific files or blocks of data, then commanding it to close when you're done with the archive. Bonus: You can also make it handle requests in a thread for buffering a whole file, and that way the pipes can be used quickly for requesting random access to the same file out of order. Of course, this way, you're still serializing the file. If you want to go extra complicated, you can make it a pipe server, and accept multiple requests simultaneously on different pipes. I wish you luck if you take any of this and roll with it.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #3 – 2021-03-27 07:25:48

Yes, but:
1) I don't really have time to do this.
2) Few people will ever know the difference. We're talking about fixes for issues that I found myself doing deliberate stress tests, not ones someone documented and reported...
3) I have already written TWO implementations of 7-zip reader (C library, C++ library). If the third one turns out to have issues also, I'm going to turn clinically insane.
4) External exe won't interop with fb2k file readers, whatever good that does, probably irrelevant.

I might still do this in some future version, but not right now.
64-bit seems inevitable and it fixes this problem also. I see that various third party libraries are having issues with 32bit already (Monkey's Audio....), just nothing essential to fb2k so far, but it's probably only a matter of time.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #4 – 2021-03-27 13:08:43

Thanks for the detailed explanation, I really appreciate it. Now I understand why this is happening and for me this is enough.
Also hope this information will be useful to other users.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #5 – 2021-03-27 13:18:47

could there ever be a 64 bit executable produced, say of a stable branch, that could be used on a portable basis just for edge cases?

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #6 – 2021-03-27 23:44:26

Quote from: 2tec on 2021-03-27 13:18:47

could there ever be a 64 bit executable produced, say of a stable branch, that could be used on a portable basis just for edge cases?

What kind of edge cases? I'm not aware of any 64-bit only Windows systems (aside from ARM-based, but those are different...) Genuinely curious.

My thoughts were more along the lines of:
The sooner a 64-bit version is released, the sooner plugin makers can work on supporting it, making the eventual transition much easier. Though that would obviously make things more complicated in the short term having yet another platform to support and work on at the same time.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #7 – 2021-03-28 03:52:10

Win10 64 bit, batch processing, long file names, large operations, those sort of one off kinda things. It would provide a way to test and allow for development of 64 bit components too.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #8 – 2021-03-28 15:49:53

Long file names or large conversion batches, I doubt you can exhaust 32bit address space with these.

The only real life case of failure due to 32bit address space that I have seen so far is the one I triggered myself-
Large solid 7-zip of albums in a lossless format, convert to another format on a machine with 32 hardware threads.
The operation not only runs out of address space failing the conversion, but leaves foobar2000 process in a state where other things fail at random due to heap fragmentation.
This is even worse than it sounds, Converter isn't solid archive aware and extracts the whole thing repeatedly per each file. I knew this was going to be bad but I wanted to see how bad exactly.

This will be fixed one way or another. Such stability issues are unacceptable; failure to read contents of 7-zip archives isn't acceptable either.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #9 – 2021-03-28 21:32:12

How about memory cache? Sync locked cache of unpacked files in memory, with maybe a 5-20 minute timeout before it releases it.

This was what I was also proposing for the executable thing, except that foobar would be keeping an executable around for a given archive for that long, and the executable would be doing the memory caching of individual files, or the entire solid chunk if the whole archive is one solid chunk.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #10 – 2021-03-29 02:34:15

Thanks I always wondered why so few people ever asked for or seemed to need a 64 bit executable.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #11 – 2021-03-29 03:48:51

Needs:

1) A whole new set of components
2) The components repository needs to support these new components

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #12 – 2021-03-29 15:12:50

Some more thoughts on this-

Original crash report that I suspected to be 7-zip related is still not actually confirmed to have anything to do with 7-zip, the user never responded with better details.
There's no more of anything resembling this on the crash report tracker.
It is not clear why user "Erratic" did not auto send the reports.

As for 7-zip issues, I'm refactoring the fixes to allow huge-dictionary archives to be read again, just block concurrent reading of many of them, which blew up spectacularly for me.

It appears that the same problem exists with RAR v5 but was never sighted or reported until now.

It is not clear to me why people would use extreme solid compression settings and expect the content to play gracefully in media players. I did this only out of professional curiosity - it is my job to try these things to make sure that my software does not explode in your face if you do anything like it.
The best I can possibly do is make this work at all, but with annoying lag.
The harder I try to mitigate the lag, the more temporary files will be made to help with consecutive operations; initial open will lag anyway. As far as I understand our userbase, creation of lots of temp files is not desired.

Finally, most audio file formats are already compressed, compressing them further achieves very little.
Support for reading of archive formats was introduced with modules and retro game music formats in mind - those do benefit from solid compression, but the archives do not reach sizes anywhere near enough to trigger the issues we're talking about.

If you just want to have an album (MP3, FLAC, whatever) in a single file, create a store-mode (uncompressed) zip of it - foobar2000 v1.6 introduces optimized reading music files packed into these, effectively as good as reading of plain files.
I noticed that Bandcamp does exactly this, and their files load instantly now.

Re: 7-zip reader issues (from "1.6.5 beta crashes")

Reply #13 – 2021-03-29 20:30:26

BTW. Another archiver that supports store mode archiving, and is supported by my Unix archive reader, is Unix Tape Archives, or .tar. As long as you use uncompressed .tar, it has no decompression overhead to reading them. In the future, I will also make it possible to read directly from the .tar file using a ranged reader, instead of reading to memory? I'm not sure if I read to memory, I'll have to look at my foo_unpack_unix archiver...

Notice