Skip to main content
Topic: Can I verify output when compressing to lossy + correction file? (Read 427 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Can I verify output when compressing to lossy + correction file?

Just as the title says, is -v supported when using hybrid lossless mode to encode a file? My testing would suggest that's not the case (I get a message that "MD5 signatures should match, but do not!" and WavPack deletes the temporary files), though I'd expect (or like) it to be.

Re: Can I verify output when compressing to lossy + correction file?

Reply #1
"[...]wavpack -f -b320 -m -v -c "$i" -o "$o"-wv-fb320.wv

 WAVPACK  Hybrid Lossless Audio Compressor  Linux Version 5.0.0
 Copyright (c) 1998 - 2016 David Bryant.  All Rights Reserved.

original md5 signature: dca8ba633e47baccb775e34b16b2512b                               
created (and verified) MF-wv-fb320.wv (+.wvc) in 21.47 secs (lossless, 32.00%)"

check your memory modules by memtest

Re: Can I verify output when compressing to lossy + correction file?

Reply #2
Just as the title says, is -v supported when using hybrid lossless mode to encode a file? My testing would suggest that's not the case (I get a message that "MD5 signatures should match, but do not!" and WavPack deletes the temporary files), though I'd expect (or like) it to be.
The -v should allow you to verify operation when using any mode, including when making a correction file, and that message is indicating that something is wrong (and is exactly why the -v option is provided).

There are several possibilities. One is that there is something wrong or weird about your source file. For example I think one situation that comes up is a WAV file indicating an incorrect bit depth (like a 24-bit file that says it's just 20-bit).

Another thing could be a hardware problem with your computer, like failing memory or storage, or over-clocking.

Finally, it could be a bug in the WavPack program (like an arithmetic overflow).

In any event, if you can provide some more information (like what is the source file format and does this happen on all files) I can probably figure out what's going on.

Re: Can I verify output when compressing to lossy + correction file?

Reply #3
These are all FLAC files, and the only thing I can think of that they have in common is that they're downloaded from BandCamp. Though it's certainly not the first time I transcoded FLAC files from them to WavPack.

From a bit of testing it triggers after I specify a bitrate lower than a certain value somehow applicable to that file:

D:\Utils\Wavpack>..\FLAC\flac.exe -d -c --totally-silent "D:\New folder\Carpenter Brut\Carpenter Brut - Leather Teeth (2018)\Carpenter Brut - LEATHER TEETH - 01 Leather Teeth.flac" | wavpack.exe -b1362 -c -m -v -i - temp.wv

 WAVPACK  Hybrid Lossless Audio Compressor  Win64 Version 5.1.0
 Copyright (c) 1998 - 2017 David Bryant.  All Rights Reserved.

original md5: 85437dd8ed2fecc8c0264d8c5c0372e0
verified md5: f85f965575ed6723089abb70857ad6d2
MD5 signatures should match, but do not!                                                                                                                                                       

D:\Utils\Wavpack>..\FLAC\flac.exe -d -c --totally-silent "D:\New folder\Carpenter Brut\Carpenter Brut - Leather Teeth (2018)\Carpenter Brut - LEATHER TEETH - 01 Leather Teeth.flac" | wavpack.exe -b1363 -c -m -v -i - temp.wv

 WAVPACK  Hybrid Lossless Audio Compressor  Win64 Version 5.1.0
 Copyright (c) 1998 - 2017 David Bryant.  All Rights Reserved.

original md5 signature: 85437dd8ed2fecc8c0264d8c5c0372e0
created (and verified) temp.wv (+.wvc) in 8.62 secs (lossless, -262878.10%)

The only reason I can think of for the failure depending on the bitrate is if the verification doesn't consider the correction file, so raw audio will different up to a threshold above which compression is actually lossless (which certainly seems to be the case with the example above, as the .wv file ends up having 31MB but the .wvc file only 63K).

I thought this might be a bug with WavPack 5.1, so I tried previous versions as well, namely 5.0 and 4.80, both 32 and 64bit variants, but it didn't make a difference. I then re-downloaded an album that I had already converted from FLAC to WavPack a while ago without issues, and got the same error (i.e. "Kaleida - Think EP - 02 Tropea.flac" downloaded from them throws the MD5 mismatch for -b1339 and below, but is OK for -b1340 and above).

While doing this comparison with my usual .bat file, I noticed the new file had the encoder and settings tags, while the old one didn't. So my conclusion was that some time ago, when I added the "-w Encoder -w Settings" parameters, I also added the -v one and didn't bother testing nor actually used WavPack since. But I also thought of asking for a confirmation here.

Now I'm actually thinking it might be a HW issue with this system, which I can confirm as soon as I get to test the same binaries with the same files on another PC (which will probably be tomorrow).

Re: Can I verify output when compressing to lossy + correction file?

Reply #4
wavpack --help
"-v                      verify output file integrity after write (no pipes)"
and yes, -b1363... "Why, mister President! Why?"(c).

Re: Can I verify output when compressing to lossy + correction file?

Reply #5
I can see why -v doesn't work when output is a pipe, but it should work with piped input.

Re: Can I verify output when compressing to lossy + correction file?

Reply #6
Not if it does a byte by byte comparison with the input, rather than doing a hash check.

Re: Can I verify output when compressing to lossy + correction file?

Reply #7
wavpack --help
"-v                      verify output file integrity after write (no pipes)"
add
web documentation say: "Because the output file is actually rewound and reread, this option cannot be used >>>when writing to pipes<<<."
check hardware... & cast David. :)

Re: Can I verify output when compressing to lossy + correction file?

Reply #8
Now I'm actually thinking it might be a HW issue with this system, which I can confirm as soon as I get to test the same binaries with the same files on another PC (which will probably be tomorrow).
There's no reason to try on another system because I have reproduced the issue, and it has nothing to do with hardware.

You were right that the problem was that the correction file was not being used in the verify mode. In the cases where the specified bitrate was so high that even the lossy file by itself was lossless, the verification went fine. When the correction file was needed for lossless, then the verification would fail.

What's weird is that this only happens when the source is a pipe (stdin), and it only happens on Windows (not Linux)! Neither of those factors should make a difference, but there it is. I also verified that this has been the behavior since the verify mode was introduced in 4.70.

The good news is that it's not your hardware and your files should all be fine...just don't use verify mode for hybrid output with piped input on Windows. You can always do the verify in a separate pass after the encoding (wvunpack -vm).

I also found another Windows-only issue that is new for the more recent releases. The -i flag causes weird formatting of the progress messages and produces that weird message you show above about -262878.10% lossless compression. That also doesn't happen on Linux.

I am not set up to build and debug on Windows tonight, so this will have to wait a few days to figure out what's going on. But I appreciate you taking the time to point this out, and I'll post again when I figure out why this is happening.  :)

Re: Can I verify output when compressing to lossy + correction file?

Reply #9
So I guess I was right assuming I added the -v parameter and didn't use WavPack since (I use foobar2000 to convert files, but even that has always been with a custom encoder that calls a .bat file - I started like that since fb2k came with an older version of WavPack when I first switched to it, and stuck with my .bat file since).

Thank you for your help! And for making and keeping WavPack awesome!

Re: Can I verify output when compressing to lossy + correction file?

Reply #10
I tried a custom Foobar converter configuration adding the -v, and that also fails with hybrid files. I am a little surprised that nobody has seen this there, but I guess most people just use the non-custom setup which doesn't include a checkbox for verify.


Re: Can I verify output when compressing to lossy + correction file?

Reply #11
Thanks again for reporting this.

I have fixed it in the repository here so it will be in the next release.

 
SimplePortal 1.0.0 RC1 © 2008-2018