HydrogenAudio

Lossless Audio Compression => WavPack => Topic started by: x0tester0x on 2021-02-09 15:52:25

Title: Which WavPack version does FFMPEG use?
Post by: x0tester0x on 2021-02-09 15:52:25
Which WavPack version does FFMPEG use?
Title: Re: Which WavPack version does FFMPEG use?
Post by: Rollin on 2021-02-09 17:33:14
FFmpeg uses its own implementation of WavPack.
In the past it also could use libwavpack. But it wasn't used properly and its support was dropped - https://hydrogenaud.io/index.php?topic=120038.0
Title: Re: Which WavPack version does FFMPEG use?
Post by: bryant on 2021-02-09 22:55:29
As Rollin says, FFmpeg has its own implementation of the decoder and encoder, so there's no real version there.

However if you're asking about the stream version, or what WavPack would report as the version of file generated by FFmpeg, then the answer is 4.

FFmpeg currently does not have the additional WavPack 5 features implemented, except DSD decoding. So, for example, the quick verify feature will not work on FFmpeg-generated files.
Title: Re: Which WavPack version does FFMPEG use?
Post by: mycroft on 2021-02-09 23:00:48
Isn't quick verify feature just way to do hashes on compressed data without ever decompressing it?
Title: Re: Which WavPack version does FFMPEG use?
Post by: bryant on 2021-02-10 00:03:27
Isn't quick verify feature just way to do hashes on compressed data without ever decompressing it?
Yes, but that hash has to be present.

For version 5, a new metadata chunk was added to the end of each block containing a checksum for the encoded bytes. If that chunk is absent then the only way to verify the block is to actually decode the audio (the decoded audio checksum is still present in the header).
Title: Re: Which WavPack version does FFMPEG use?
Post by: mycroft on 2021-02-10 15:38:22
what kind of hash it uses, i only see 2 variants, 16bit and 32bit, but nowhere mentioned standard.
Title: Re: Which WavPack version does FFMPEG use?
Post by: mycroft on 2021-02-10 17:49:17
Also this is badly designed, there is no way to update block checksum for first block and at same time update its final number of samples, which is by the way uint32_t
Title: Re: Which WavPack version does FFMPEG use?
Post by: bryant on 2021-02-10 21:27:19
The hashes are byte checksums, with an initial value of all ones and a multiply of the kernel by 3 before the add (which I find adds robustness without a lot of complexity or cost). Either the entire 32-bit value is used, or the upper and lower halves are xor’d for the 16-bit version.

The checksum always comes last and includes everything in the block except itself. It’s also optional.

I’m not sure what you mean by not being able to update the block checksum and the number of samples. This is done by WavpackUpdateNumSamples() in libwavpack using these functions:

block_add_checksum() and block_update_checksum() (https://github.com/dbry/WavPack/blob/master/src/pack_utils.c#L1315)

And by the way, since you mention uint32_t, one of the other changes for WavPack 5 was the ability to handle WavPack files over 2 GB and with up to 2^40 samples, but again these changes were done in a backward-compatible way so as to not break anything that currently works with WavPack files or libwavpack. They are all described here:

WavPack 5 Porting Guide (https://github.com/dbry/WavPack/blob/master/doc/WavPack5PortingGuide.pdf)

And thanks again for the past and ongoing support from the FFmpeg team!  :)
Title: Re: Which WavPack version does FFMPEG use?
Post by: mycroft on 2021-02-11 00:06:45
This is all API documentation and not format specification, making wavpack just slightly better than other closed codec.

AFAIK first block header still only contain uint32_t number of samples.
And in streaming mode, when you seek to first block you need to reread whole first block to update checksum.

Also how good is checksum quality versus bruteforce attempts?
Title: Re: Which WavPack version does FFMPEG use?
Post by: arkhh on 2021-02-11 12:57:12
Also how good is checksum quality versus bruteforce attempts?

Could it be that you are confusing a CRC / checksum with a cryptographic hash?
Title: Re: Which WavPack version does FFMPEG use?
Post by: Porcus on 2021-02-11 13:32:43
Also how good is checksum quality versus bruteforce attempts?
Irrelevant. These are not intended to safeguard against malicious manipulation. only to scream at errors. Like if you paginate all pages in your 42 page document by n/42 and you see them in order "18, 666, 20" you can know that there was something wrong.

If someone replaced segments of your Queens of the Stone Age with Kanye West and just re-encoded and replaced the file, how would a WavPack decoder know the difference? I'm quite sure it was never intended to object to that kind of stuff (https://totom.bandcamp.com/album/kanye-of-the-stone-age).
Title: Re: Which WavPack version does FFMPEG use?
Post by: bryant on 2021-02-11 23:04:24
This is all API documentation and not format specification, making wavpack just slightly better than other closed codec.
You are correct. However, the format specification (https://github.com/dbry/WavPack/blob/master/doc/WavPack5FileFormat.pdf) is sitting in that same folder.

I admit that this format specification is not nearly complete enough to independently create a WavPack decoder. However, this has not prevented the creation of several alternative decoders, including the FFmpeg one and others in other languages.

I also believe that while some formats have more complete formal specifications, even one as complete as FLAC leaves out too much to be used exclusively, as indicated in this description. (https://www.nayuki.io/page/simple-flac-implementation) And unfortunately WavPack is significantly more complex than FLAC.

Quote
AFAIK first block header still only contain uint32_t number of samples.
If you look in the format document I link to above you will find the extra 8 bits extending the sample count to 40-bits. Note that there are macros in wavpack.h (https://github.com/dbry/WavPack/blob/master/include/wavpack.h) to get and set those fields because the extension was complicated by the fact that having the lower 32 bits all ones has a special meaning.

Quote
And in streaming mode, when you seek to first block you need to reread whole first block to update checksum.
Yes, this is correct. But why is that such a big deal? The blocks aren't that big and the size is stored at the beginning so you know how much to allocate and read.

Quote
Also how good is checksum quality versus bruteforce attempts?
As was said, it's a simple checksum to allow a user to quickly verify that WavPack files have not been unintentionally corrupted. It is not intended for files intentionally corrupted.
Title: Re: Which WavPack version does FFMPEG use?
Post by: DARcode on 2021-02-17 01:12:58
Thanks David, it's much clearer to me too now :) , a great guy as always replying to uselessly aggressive/borderline rude questions and remarks like a true gentleman.

No thanks to Procus ;) for exposing me to this (https://totom.bandcamp.com/album/kanye-of-the-stone-age).
Title: Re: Which WavPack version does FFMPEG use?
Post by: mycroft on 2021-02-17 10:35:21
Yes, this is correct. But why is that such a big deal? The blocks aren't that big and the size is stored at the beginning so you know how much to allocate and read.

Thanks for confirming it is correct, but note that this "blocks aren't that big" is very limited view. For streaming oriented apps like ffmpeg, it needs to parse whole packet yet again to write updated header is sign of bad design. I could do several version of hacks to mitigate this, but why?

Quote
As was said, it's a simple checksum to allow a user to quickly verify that WavPack files have not been unintentionally corrupted. It is not intended for files intentionally corrupted.
Than its usage extremely very limited. Thanks for confirming yet another my point.
Title: Re: Which WavPack version does FFMPEG use?
Post by: Porcus on 2021-02-17 11:00:18
Than its usage extremely very limited.
It is precisely what you need for an integrity check.

I wonder, what else do you even imagine you need it for - and what other audio format provides it? There was a discussion here when the author of a certain codec thought that encryption was the be-all-end-all feature to be touted in the wiki, but that is about it.