Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: WavPack 5.5.0 Release Candidate (Read 26971 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

WavPack 5.5.0 Release Candidate

Considering that CVE-2021-44269 was assigned for a bug in the wavpack command-line program, it was suggested on GitHub (and I agreed) that it was time for a new WavPack release. While I was at it I decided to add a handful of new minor features and fixed some issues that had been nagging me.

Here’s what’s new:

  • fixed: CVE-2021-44269 (encoding crafted DSD file causes out-of-bounds read crash)
  • fixed: very long filenames cause stack-overflow crash in command-line programs
  • fixed: attempting to encode raw DSD audio from stdin sometimes causes crash
  • fixed: DSD to PCM decimation: small clicks between tracks and tiny DC offset
  • fixed: the length stored in the WAV header not always corrected when using -i
  • added: accepting brace-delimited options in the wavpack executable filename
  • added: "--drop" option to Windows executables for multi-file "drag-and-drop"
  • added: --raw-pcm option to wvunpack executable (does DSD --> 24-bit PCM)
  • added: --no-overwrite option to wavpack executable (to resume sessions)
  • improved: build system clean-up including switch to non-recursive "make"

The new --no-overwrite option for wavpack is handy when resuming a long conversion that was interrupted. I also had fun using it to allow multiple instances of the wavpack program to operate in the same folder and files without overwriting each other (poor man’s multi-threading)! Thanks to @Porcus for the idea!

Another idea from our porcine friend was allowing brace-delimited options in the wavpack command-line program filename. You could already add a "debug" to the filename for extra logging that would also get written to a file on Windows. This new feature is useful for using with Windows “drag and drop” to change the configuration of the conversion, but it can also be used anywhere to create a custom configuration. For instance, one might rename the executable:

wavpack{--pause}{-hxvm}{--import-id3}.exe

I have prepared a release candidate and greatly appreciate any testing or comments that anyone has.

Thanks as always to the entire HA community for the continued support!

Re: WavPack 5.5.0 Release Candidate

Reply #1 –
Thnx bryant. :)

Re: WavPack 5.5.0 Release Candidate

Reply #2 –
A question not specific to this release: I can't really find a statement on the WavPack website or man page that WavPack not only losslessly encodes audio, but the structure and metadata in the source file as well.

The thing is: in the past I've pointed people to WavPack when complaining about FLAC missing some functionality (DSD, floats, more than 8 channels etc.) and now I've gotten a bug report that FLAC is not able to restore AIFF/sowt (AIFF with little endian byte order) properly: it instead creates a 'normal', big-endian AIFF file. So, the audio is carried over losslessly, but is not restored to exactly the same format. This could be added to FLAC of course, but I'd rather also point this person to WavPack. In my view, FLAC is really an audio compressor with the ability to store en restore foreign metadata (e.g. RIFF chunks), but not a format that is meant to be able to replicate the exact WAV/AIFF/etc. structure bit-for-bit.

Could you add a statement to the website and/or man-page like: "WavPack compresses the whole input file and can restore it bitperfect. It does this not only for the audio, but for the whole file." You can probably think of something better/catchier. That way I can point people there. Or am I mistaken and does WavPack not do this?
Music: sounds arranged such that they construct feelings.

Re: WavPack 5.5.0 Release Candidate

Reply #3 –
"In the default lossless mode WavPack acts just like a WinZip compressor for audio files" from the "About" is clear when you know what it means - but not until then (it took me quite some time too). A bit/byte more foolproofing could do yes.

But, WavPack does not support AIFF at all. (It handles CAF.) Looks like the options for AIFF are then MPEG-4 ALS and Monkey's (I haven't tested either - actually I just found out that Monkey's implemented that last year; it also handles AU/SND, if anyone cares).

Re: WavPack 5.5.0 Release Candidate

Reply #4 –
Quote from: Porcus on
But, WavPack does not support AIFF at all. (It handles CAF.)
Ha, silly me. Didn't notice that little detail :-[
Music: sounds arranged such that they construct feelings.

Re: WavPack 5.5.0 Release Candidate

Reply #6 –
Well that was some discussion that did not consistently bring out the best of HA ...

Meanwhile, I tried - and failed - to get Monkey's and ALS to accept the sowt files. So there are no such options then?
Also, it seems to me that none of the codecs can force an output choice between WAVE_FORMAT_EXTENSIBLE and old WAVE. (Well ffmpeg can, but ffmpeg does not default to lossless operation in the first place, it is a different tool.)
 

Re: WavPack 5.5.0 Release Candidate

Reply #7 –
Quote from: ktf on
Quote from: Porcus on
But, WavPack does not support AIFF at all. (It handles CAF.)
Ha, silly me. Didn't notice that little detail :-[

Quote from: Porcus on
Meanwhile, I tried - and failed - to get Monkey's and ALS to accept the sowt files. So there are no such options then?
Also, it seems to me that none of the codecs can force an output choice between WAVE_FORMAT_EXTENSIBLE and old WAVE. (Well ffmpeg can, but ffmpeg does not default to lossless operation in the first place, it is a different tool.)

Oh, looks like bryant will once again be overloaded with requests from ... people who don't use WavPack that much :-o

Do I kinda feel guilty here? I kinda should. I have like a measly (for my hoarding habits) 20 GB lossless that I need to distinguish in case I should stupidly destroy tags, and having started to read myself up on the codecs, I kinda liked WavPack and chose that for my 2nd codec back in the day. And the more of these weird tests I do, the more I like WavPack, it behaves utterly sane. A compressor is more than just an encoding/decoding algorithm. (And for once I shall refrain from knocking the competition.)
 

Re: WavPack 5.5.0 Release Candidate

Reply #8 –
Gee, I was hoping to be able to edit the first post for an RC2, but I guess there’s a time limit. I hate to start a new thread, so I’ll just post here and maybe a kind mod can delete the downloadable files from the first post.

So the reason for the update was to allow the drag-and-drop feature to handle multiple files. This is not straightforward because the syntax on Windows is to have the optional second specified file be the output file (even though wildcards are supported in the first file). Well, the solution was to add a new option “--drop” which changes the syntax to simply a list of input files. You still can’t drop a folder (which is a whole new set of complications) but this is still a lot better than one at a time. Here’s a new example filename:

wavpack{--pause}{--drop}{-hxvm}{--import-id3}.exe

Here’s the new change list:

  • fixed: CVE-2021-44269 (encoding crafted DSD file causes out-of-bounds read crash)
  • fixed: very long filenames cause stack-overflow crash in command-line programs
  • fixed: attempting to encode raw DSD audio from stdin sometimes causes crash
  • fixed: DSD to PCM decimation: small clicks between tracks and tiny DC offset
  • fixed: the length stored in the WAV header not always corrected when using -i
  • added: accepting brace-delimited options in the wavpack executable filename
  • added: "--drop" option to Windows executables for multi-file "drag-and-drop"
  • added: --raw-pcm option to wvunpack executable (does DSD --> 24-bit PCM)
  • added: --no-overwrite option to wavpack executable (to resume sessions)
  • improved: build system clean-up including switch to non-recursive "make"



 :-[  MOD blunder: deleted files from wrong post.

Re: WavPack 5.5.0 Release Candidate

Reply #9 –
Quote from: ktf on
Quote from: Porcus on
But, WavPack does not support AIFF at all. (It handles CAF.)
Ha, silly me. Didn't notice that little detail :-[
Yeah, AIFF is on my wish-list, and in fact there's a half-done version in an old GitHub branch. I got bogged down on ID3v2.2 support which I wanted to do at the same time (don't ask). Maybe next time.

But your understanding is correct that WavPack by default creates and restores a byte-exact representation of the entire source file, regardless of the format. And in the case of CAF that includes both endian versions. Of course it doesn't really understand any of the metadata beyond the basics, so if you export to a different format then that stuff is lost.

But I will update the webpage to make that feature clearer. When I originally wrote about the WinZip relationship I wasn't aware that saving the headers would be one of the main draws ... I was just trying to explain what lossless audio was!  :)

And, BTW, thanks for the referrals! I'm actually working with some people who want to compress thousands of channels and they were referred (perhaps by you).

Re: WavPack 5.5.0 Release Candidate

Reply #11 –
Quote from: bryant on
And, BTW, thanks for the referrals! I'm actually working with some people who want to compress thousands of channels and they were referred (perhaps by you).
If that is with electrophysiology data, then yes, that was probably me. :D It is rather interesting that audio compressors like FLAC and WavPack are being used for storing other kinds of signals. In this case brainwaves, FLAC has also seen use to compress laserdisc rips. There are probably tons of other electrical signals with lots of spatial redundancy that can be compressed well, so I suspect that won't be the last of it.

edit: I see now the discussion on WavPack github, very interesting to read indeed.
Music: sounds arranged such that they construct feelings.

Re: WavPack 5.5.0 Release Candidate

Reply #12 –
There was such a discussion here some time too. @polemon, did you end up with WavPack or something else?   :)

MPEG-4 ALS supposedly supports 65536 channels. I have not tried to max it out.  (Also, I did at one stage manage to create .als files which would make the decoder consistently crash, so it's not production ready ... and will likely never be.)

Re: WavPack 5.5.0 Release Candidate

Reply #13 –
Quote from: ktf on
FLAC has also seen use to compress laserdisc rips. There are probably tons of other electrical signals with lots of spatial redundancy that can be compressed well, so I suspect that won't be the last of it.

It's first time I've heard of this, it's very interesting! I'd like to try it out, but I don't know if there is some sort of demo/free laserdisk image I can use for that.
Error 404; signature server not available.

Re: WavPack 5.5.0 Release Candidate

Reply #14 –
Quote from: Porcus on
Meanwhile, I tried - and failed - to get Monkey's and ALS to accept the sowt files. So there are no such options then?
Also, it seems to me that none of the codecs can force an output choice between WAVE_FORMAT_EXTENSIBLE and old WAVE. (Well ffmpeg can, but ffmpeg does not default to lossless operation in the first place, it is a different tool.)
For SoX supported formats (so, not APE), find keyword "wavpcm" in the link below:
http://sox.sourceforge.net/soxformat.html

Re: WavPack 5.5.0 Release Candidate

Reply #15 –
Quote from: bennetng on
For SoX supported formats (so, not APE), find keyword "wavpcm" in the link below:
http://sox.sourceforge.net/soxformat.html
The comment wasn't about being able to compress audio data in AIFF-C with sowt data, but about being able to compress and restore the file bitperfect including headers and metadata.

Quote from: ktf on
In this case brainwaves
Use of the name 'BrainWavPack' is now warranted :D
Music: sounds arranged such that they construct feelings.

Re: WavPack 5.5.0 Release Candidate

Reply #16 –
Quote from: ktf on
Quote from: bryant on
And, BTW, thanks for the referrals! I'm actually working with some people who want to compress thousands of channels and they were referred (perhaps by you).
If that is with electrophysiology data, then yes, that was probably me. :D It is rather interesting that audio compressors like FLAC and WavPack are being used for storing other kinds of signals. In this case brainwaves, FLAC has also seen use to compress laserdisc rips. There are probably tons of other electrical signals with lots of spatial redundancy that can be compressed well, so I suspect that won't be the last of it.
Yes, I’ve been working with them on the electrophysiology data. I hadn’t heard of those laserdisc rips either, which is interesting. I still have a bunch of those (LDs) and people sometimes forget that they were purely analog RF (even the sound was FM early on).

Any analog sampling capture is going to benefit from lossless compression (unless it’s fullscale white noise) so these applications make sense, especially when they generate so much data that they want archived in the cloud. I've worked with researchers interested in storing seismological data and even astronomical radiophysics data.

And in some cases the lossy mode of WavPack is additionally useful because the application might not necessarily require all the resolution provided by their sampling, but they don’t want to deal with the dynamic range reduction that comes with truncation.
 

Re: WavPack 5.5.0 Release Candidate

Reply #17 –
So it seems that the status on file restore and foreign metadata is as follows, assuming compliant < 4GB files, excluding raw PCM - and excluding the TTA and WMAL formats:

WAVE: WavPack/Monkey's/OptimFROG/TAK/ALS do full file restore by default. Monkey's and ALS "enforce" this: they have no provision to switch off metadata storage (but by now support piping, so can be fed just a PCM stream, and metadata can be destroyed by outputting to different format). FLAC can optionally store the metadata but has no ambition of recreating the file structure. (ref)ALAC has no such provision.
All accept both old WAVE and WAVE_FORMAT_EXTENSIBLE input, all can force WAVE output regardless of input (well TAK/OFR cannot do anything else but .wav and raw) - but none have an option to select output between the two.

RF64/BWF/w64: Essentially as .wav support as goes FLAC/WavPack/Monkey's/ALS. refalac supports RF64(/BWF?) but not w64. OptimFROG/TAK have no support for these.

AIFF: Monkey's supports big-endian with full file restore, reject the little-endian - apparenty ALS as well. FLAC reads both endiannesses as infile but upon asked to output AIFF will write only big-endian. Again FLAC has no ambition to preserve file structure.

CAF: WavPack. User can force either endianness for output. Monkey's seems to have gotten it, but I don't know what endianness(es) (haven't tested the fixes after initial .caf-supported version bugged) - presumably with full restore, as that is how the ape works.

AU/SND: Monkey's only. Presumably with full restore.


Quote from: bryant on
Yeah, AIFF is on my wish-list, and in fact there's a half-done version in an old GitHub branch. I got bogged down on ID3v2.2 support

2.two ... and then you got 2.4. And who knows about Lyrics (is that extension even possible in WAVE/AIFF?)

But if you don't need to import them to WavPack's tagset, is this in principle any worse than what you do about ID3v2.nonthree-tagged .dsf?

Quote from: bryant on
(don't ask).
Oops I did it again.

Re: WavPack 5.5.0 Release Candidate

Reply #18 –
Quote from: Porcus on
Quote from: bryant on
Yeah, AIFF is on my wish-list, and in fact there's a half-done version in an old GitHub branch. I got bogged down on ID3v2.2 support

2.two ... and then you got 2.4. And who knows about Lyrics (is that extension even possible in WAVE/AIFF?)

But if you don't need to import them to WavPack's tagset, is this in principle any worse than what you do about ID3v2.nonthree-tagged .dsf?

Quote from: bryant on
(don't ask).
Oops I did it again.
Yes, Britney, you did...  :)

The original impetus for starting on AIFF file support was that I happened onto a cache of them from my then employer (a well-known high-fidelity firm). I had previously never encountered any AIFF files (not being much of an Apple guy) and had sort of naively assumed that CAF would take over AIFF because it’s so much better in pretty much every way (it turns out that’s not necessarily a recipe for success).

Well, it turns out that cache of AIFF files was tagged with ID3v2.2 and so, based on a sample size of one, decided that implementing AIFF without ID3v2.2 would be pointless (i.e., my only test files wouldn’t even work right).

So, I will eventually put in AIFF for completeness (but probably not AU/SND  :) ) and will probably opt for ID3v2.4 support and not ID3v2.2 (as I don’t have those files any more). Not exactly ideal for a "universal audio archiver", but with all the fuzz testing going on (check out today's bug) it's silly to spend time working on formats that aren't going to be used.

Re: WavPack 5.5.0 Release Candidate

Reply #19 –
Quote from: bryant on
and had sort of naively assumed that CAF would take over AIFF because it’s so much better in pretty much every way (it turns out that’s not necessarily a recipe for success).
Yeah, the CAF itself is not only 64 bit based but also flexible enough to carry virtually any audio codec with sample accurate seekability and gapless playback support. Technically it's superior to any of WAV/AIFF/AU/wave64/rf64, Still, it's a lot simpler than quicktime MOV or ISOBMFF container.
However, it's not actually possible except for a few codecs, due to lack of codec mapping definition (it's not open and has to be defined by Apple, unlike matroska or something).
And sad truth is that none of these has nothing to do with success of the format (as always).

Quote from: bryant on
Well, it turns out that cache of AIFF files was tagged with ID3v2.2 and so, based on a sample size of one, decided that implementing AIFF without ID3v2.2 would be pointless (i.e., my only test files wouldn’t even work right).
Do you actually need it? As far as I know wavpack has to just read/store/restore them as an opaque data chunk.
Of course it would be better If wavpack understand them and automatically convert them to wavpack container-level tags, but wavpack doesn't do it anyway for RIFF INFO chunks, I supporse?

Re: WavPack 5.5.0 Release Candidate

Reply #20 –
I agree that AIFF support can come without transferring ID3v2.nonthree tags. Or even without transferring tags at all - if that is what it takes, then ... why not? (Except it still takes time and hassle, of course).

As for .wav, I don't know all the format's tagging whereabouts, but
wavpack --import-id3 wav_file_tagged_by_mp3tag.wav
wavpack --import-id3 wav_file_tagged_by_foobar2000.wav
will indeed give me a tagged .wv file.

Provided the tags are not too big, in which case WavPack should likely make a clearer warning to the user that NO OUTPUT FILE WRITTEN. The console output does not give any information that the .wv file is deleted.
Same is the case (well or, fails to be the case) when encountering the "ID3v2 import: bad frame identity".

Not sure what is best behaviour here, really - IMHO, writing a file makes the claim that "I was able to do this!", which is a case for deleting it (and of course, telling the user clearly!). On the other hand, you can imagine an annoyed user who has waited for -hhx6'ing a full multichannel DVD just to see the file deleted.

Possible solution 1: Check metadata size first.
Possible solution 2: Rather than deleting, rename to filename.wv.NOTAGSIMPORTED.tmp and tell the user that they can change the name and use a different tagger to copy the tags, or re-run with --allow-huge-tags


(And: I don't anymore remember what message WavPack returns when I try to import ID3v2.otherthanthree. Likely it should tell me to RTFM.)

Re: WavPack 5.5.0 Release Candidate

Reply #21 –
Yeah, playing around with it now and seeing that the ID3 import stuff is a little rough around the edges. Will have to improve that when I add ID3v2.4 support.

The first thing is that we look specifically for ID3v2.3, so even though the parsing code has a nice message about only supporting v2.3, we never get there because ID3v2.4 is treated like “no tag found”, and I didn’t want to error out on that and delete the output file because it’s nice to specify the --import-id3 by default (like in my drag and drop example).

And yeah, deleting the output file is a little severe for exceeding 1 MB, but a warning that nobody sees isn’t great either (drag and drop). People will notice a deleted output!  :)

There are a couple of tricks that I use to reduce some of the potential frustration of waiting for a long compression and then having it abort. The first is when I’m ripping CDs (or whatever I’m doing to obtain the initial WavPack files) I usually just use the default lossless mode because it’s really safe and I don’t want to slow what I’m doing (this is one reason I hesitate changing the default). Afterward I use the re-encode feature to get the modes I want (“high” lossless, or hybrid lossless, or even lossy) and other tweaks.

Another thing I do is use wvtag afterward to clean up the tags and add the artwork if it isn’t there already, and you can give the --import-id3 option there also and it will rescan the end of the file for potential tags. That way if you get the dreaded “over 1MB” error you can just do it again (or use a more reasonable image).

In any event, thanks for all the comments, suggestions, and testing! I've done some clean-up of the web site, including a link to the cool WebAssembly player that Peter McQuillan created. There's still some more clean-up to do there (especially in the ancient "hardware" section), but for now the release is done:

WavPack 5.5.0 Release on Github

Cheers!

Re: WavPack 5.5.0 Release Candidate

Reply #22 –
Thanks, but I can't get the WebAssembly player to work on Firefox, the "Load WavPack File" does nothing. Is is a known issue or somehow related to Firefox's default security settings?

Also, the "XHTML 1.0" link indicates the page is no longer valid.

Re: WavPack 5.5.0 Release Candidate

Reply #23 –
Yeah, I have only got it work on Chrome and Edge myself and figured Firefox was an outlier. I just looked a little deeper and found the problem to be the file picker:

Quote
Uncaught (in promise) TypeError: window.showOpenFilePicker is not a function
    <anonymous> https://www.wavpack.com/WebAssembly/mywavpack_worklet.js:31

Googling that got me to this page which shows Firefox not supporting the file picker. So I'm guessing that the Firefox people consider that too dangerous?

Of course, a real world application of this (if there was one) would probably not involve trying to play existing files on the user's computer but would involve files stored on a server somewhere.

Perhaps Peter will chime in with more context.

Re: WavPack 5.5.0 Release Candidate

Reply #24 –
Quote from: bryant on
Yeah, I have only got it work on Chrome and Edge myself and figured Firefox was an outlier. I just looked a little deeper and found the problem to be the file picker:

Quote
Uncaught (in promise) TypeError: window.showOpenFilePicker is not a function
    <anonymous> https://www.wavpack.com/WebAssembly/mywavpack_worklet.js:31

Googling that got me to this page which shows Firefox not supporting the file picker. So I'm guessing that the Firefox people consider that too dangerous?

The File System Access API (of which showOpenFilePicker is a part) is currently an unofficial draft, i.e. not standardised. Browsers should not be expected to implement and support it, and indeed CanIUse indicates that the support for the API is spotty and inconsistent.

Here's a guide on how to do this with standardised APIs.