Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Problem with archive handling (Read 3861 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Problem with archive handling

I have a lot of musical releases packet into .7z files. Each 7z archive contains several *.mp3 and album art pics. I wanted mp3's from this releases to be indexed by foobar's ML, so i wrote unpacker plugin for 7z archives. I added path to the dir with *.7z files to "Music folders" list.
But I am getting strange behavior: foobar rescans (unpacks) all my *.7z files on every start. I added logging of that is going on to the console, so I see that foobar calls archive_list for every *.7z file in ML with p_want_readers always set to true. Because of want_readers == true plugin unpacks every file in archive. I have about 100Gb of 7z archives and unpacking of every foobar's launch such number of files is really annoying.
Can you fix this?

You can get source code of my plugin here and already compiled dll here.

Problem with archive handling

Reply #2 –
Whether he uses my component or his own (which he has told me uses the DLL modules installed by 7-zip, rather than static linking the relevant code) does not matter.

The issue is that Media Library scans will always parse over archives and also request the decompressed file contents in the process. I'm thinking that the process should be optimized into two passes, the first of which will run the archive_list function on each archive with p_want_readers set to false, then compare the list of files, their sizes, and their modification times, against the known Media Library database. It should only be performing the extraction pass if it can detect that the archive file has changed.

Detecting that the archive has changed could also be implemented by noting the archive itself in the Media Library, but that would require significant API changes. Checking the archive contents for new files or stats changes on existing files should be possible with minimal changes to the player core.

Of course, all of this is sort of silly, since I think the original purpose of this archive support was to allow for reasonably sized archives of many similar files to be compressed in a solid fashion. Like RAR files full of SNES soundtracks in SPC format. That it works at all for large archives of files which are relatively hard to compress further, is somewhat of a blessing.

Problem with archive handling

Reply #3 –
Quote from: marc2003 on
i guess you weren't aware of this?

I developed my plugin several years ago, it was never released. It worked perfectly on foobar version < 1.0 and on some 1.* versions. I don't know on what exactly version it stopped to work, but i know, that it works as expected few versions ago.

Problem with archive handling

Reply #4 –
Please follow tech support guidelines; in particular, state which foobar2000 version you are running.
I have tested the current foobar2000 version for what you are describing and no archive re-indexing happens unless your archive's file modification timestamp changes or you don't let the previous instance finish indexing specific archive.
Microsoft Windows: We can't script here, this is bat country.

Problem with archive handling

Reply #5 –
I am sorry for not following the rules. I am using foobar version 1.1.8 b3, here is a list of my components:
Code: [Select]
Core (2011-07-29 18:31:28 UTC)
    foobar2000 core 1.1.8 beta 3
foo_albumlist.dll (2011-07-29 10:50:28 UTC)
    Album List 4.5
foo_bpm.dll (2010-10-16 14:30:50 UTC)
    BPM Analyser 0.2.4.1
foo_burninate.dll (2011-02-24 04:40:34 UTC)
    Audio CD Writer 3.0.3
foo_cdda.dll (2011-07-29 10:50:26 UTC)
    CD Audio Decoder 3.0
foo_converter.dll (2011-07-29 10:50:06 UTC)
    Converter 1.5
foo_dsp_std.dll (2011-07-29 10:50:30 UTC)
    Standard DSP Array 1.0
foo_fileops.dll (2011-07-29 10:49:14 UTC)
    File Operations 2.1.3
foo_httpcontrol.dll (2011-07-04 14:27:56 UTC)
    HTTP Control 0.97.9
foo_input_alac.dll (2011-02-26 06:53:14 UTC)
    ALAC Decoder 1.0.7
foo_input_monkey.dll (2011-02-24 06:13:54 UTC)
    Monkey's Audio Decoder 2.1.5
foo_input_std.dll (2011-07-29 18:30:52 UTC)
    Standard Input Array 1.0
foo_input_tak.dll (2010-01-09 13:11:58 UTC)
    TAK Decoder 0.4.4
foo_jesus.dll (2010-09-22 19:34:06 UTC)
    Autosave & Autobackup 10
foo_keep_queue.dll (2010-11-11 22:28:46 UTC)
    Keep Queue 0.3.5
foo_masstag.dll (2009-09-18 04:01:36 UTC)
    Masstagger 1.8.4
foo_out_asio.dll (2009-03-22 09:15:46 UTC)
    ASIO support 1.2.7
foo_playcount.dll (2010-10-12 08:37:48 UTC)
    Playback Statistics 3.0.1
foo_plorg.dll (2011-05-05 12:03:00 UTC)
    Playlist Organizer 2.4
foo_queuecontents.dll (2011-05-20 01:52:42 UTC)
    Queue Contents Editor 0.4.2
foo_ui_columns.dll (2011-02-27 20:22:58 UTC)
    Columns UI 0.3.8.8
foo_ui_std.dll (2011-07-29 10:50:28 UTC)
    Default User Interface 0.9.5
foo_uie_albumart.dll (2010-10-11 13:44:20 UTC)
    Album Art Panel 0.2.7
foo_uie_vis_channel_spectrum.dll (2008-05-17 20:02:12 UTC)
    Channel Spectrum panel 0.17.2
foo_unpack.dll (2011-07-29 10:49:38 UTC)
    ZIP/GZIP/RAR Reader 1.6
foo_unpack_7z.dll (2011-08-05 11:57:25 UTC)
    7z Unpacker 2.0
foo_w7shell.dll (2010-01-16 22:09:20 UTC)
    Windows 7 integration 0.2.9.1
foo_wave_seekbar.dll (2011-04-06 23:01:36 UTC)
    Waveform seekbar 0.2.13

Let's say i have "c:\ML\" path in the Music folders list. That path contains archive "023 - Commie64 - Boolean Values.7z" with following structure:
023 - Commie64 - Boolean Values.7z\02 - hero.mp3
023 - Commie64 - Boolean Values.7z\03 - warm.mp3
023 - Commie64 - Boolean Values.7z\04 - cold.mp3
023 - Commie64 - Boolean Values.7z\05 - love.mp3
023 - Commie64 - Boolean Values.7z\06 - hate.mp3
023 - Commie64 - Boolean Values.7z\cover.gif

Every time a start fb2k, i get this in foobar's console:
Code: [Select]
Watching: C:\ML
archive_list file://C:\ML\023 - Commie64 - Boolean Values.7z; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|01 - start.mp3; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|02 - hero.mp3; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|03 - warm.mp3; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|04 - cold.mp3; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|05 - love.mp3; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|06 - hate.mp3; want_readers=1
archive_list unpack://7z|47|file://C:\ML\023 - Commie64 - Boolean Values.7z|cover.gif; want_readers=1

Problem with archive handling

Reply #6 –
Did you intentionally miss the rest of his post indicating that your problem should not be occurring if you allow indexing to complete first, and if your archive reader is reporting all the correct information? Perhaps you could PM me the source code to at least the archive class, and I can tell you exactly why it's not working correctly. If, after looking at that, I find nothing wrong with it, then we can talk about core bugs. (You could also test this against the latest beta, in the mean time.)

EDIT: Oh, I see you linked the source in your first post. Now who's failing to read?

Further edit: Try replacing the filesystem::g_open_tempmem calls with new service_impl_t<file_buffer>([insert file modification time here]). You can lift the file_buffer class from the source code to either my JMA or LHA archive components. It's a basic reimplementation of a tempmem style file, only with a file modification timestamp which you can set on object creation, presumably from the m_file_list[].m_stats.m_timestamp. If that doesn't fix it, then I don't know what the problem is.

Further edit again: Try logging the modification timestamps of the archives themselves when they're opened. Maybe something is causing them to change.

Problem with archive handling

Reply #7 –
I tried file_buffer class, it didn't help. Also, timestamp of the archive doesn't change.
After indexing is completed, i can see in foobar's database.dat file strings with the names of indexed files from the archive, but on next foobar's launch it starts indexing again.
Don't know what to do with it

Problem with archive handling

Reply #8 –
kode64, can you test your 7z plugin for bad things, that i described ?

Problem with archive handling

Reply #9 –
The issue is that the core expects the individual files' modification timestamps to be the same as the archive itself. My latest archive reader components reflect this requirement.

file_buffer or similar is the correct way to implement the extracted files for callers, especially my newest version that allows you to create the buffer pre-sized to fit the extracted file.

The functions get_stats_in_archive and archive_list need to return the archive's own modification timestamp in their stats objects, and archive_list and open_archive should also pass the archive's timestamp in the file_buffer created to contain any decompressed data to return to the caller.

Problem with archive handling

Reply #10 –
This is a product of a bug in my archive readers: last-modified timestamps for the archived files are not processed; archive's own last-modified time is used instead. Unfortunately it's not so easy to workaround without more severe changes in the folder watcher code than I don't currently have time to make. foobar2000 is not an archiver, not reporting accurate timestamps for in-archive files is not a problem that our users should be concerned about. I recommend changing your archive readers to duplicate this bug and moving over.
Microsoft Windows: We can't script here, this is bat country.
 

Problem with archive handling

Reply #11 –
Finally i fixed it! Thanks you!