Considering that CVE-2021-44269 (https://nvd.nist.gov/vuln/detail/CVE-2021-44269) was assigned for a bug in the wavpack command-line program, it was suggested on GitHub (and I agreed) that it was time for a new WavPack release. While I was at it I decided to add a handful of new minor features and fixed some issues that had been nagging me.
Here’s what’s new:
- fixed: CVE-2021-44269 (encoding crafted DSD file causes out-of-bounds read crash)
- fixed: very long filenames cause stack-overflow crash in command-line programs
- fixed: attempting to encode raw DSD audio from stdin sometimes causes crash
- fixed: DSD to PCM decimation: small clicks between tracks and tiny DC offset
- fixed: the length stored in the WAV header not always corrected when using -i
- added: accepting brace-delimited options in the wavpack executable filename
- added: "--drop" option to Windows executables for multi-file "drag-and-drop"
- added: --raw-pcm option to wvunpack executable (does DSD --> 24-bit PCM)
- added: --no-overwrite option to wavpack executable (to resume sessions)
- improved: build system clean-up including switch to non-recursive "make"
The new
--no-overwrite option for wavpack is handy when resuming a long conversion that was interrupted. I also had fun using it to allow multiple instances of the wavpack program to operate in the same folder and files without overwriting each other (poor man’s multi-threading)! Thanks to
@Porcus for the idea!
Another idea from our porcine friend was allowing brace-delimited options in the wavpack command-line program filename. You could already add a "
debug" to the filename for extra logging that would also get written to a file on Windows. This new feature is useful for using with Windows “drag and drop” to change the configuration of the conversion, but it can also be used anywhere to create a custom configuration. For instance, one might rename the executable:
wavpack{--pause}{-hxvm}{--import-id3}.exeI have prepared a release candidate and greatly appreciate any testing or comments that anyone has.
Thanks as always to the entire HA community for the continued support!
Thnx bryant. :)
A question not specific to this release: I can't really find a statement on the WavPack website or man page that WavPack not only losslessly encodes audio, but the structure and metadata in the source file as well.
The thing is: in the past I've pointed people to WavPack when complaining about FLAC missing some functionality (DSD, floats, more than 8 channels etc.) and now I've gotten a bug report that FLAC is not able to restore AIFF/sowt (AIFF with little endian byte order) properly: it instead creates a 'normal', big-endian AIFF file. So, the audio is carried over losslessly, but is not restored to exactly the same format. This could be added to FLAC of course, but I'd rather also point this person to WavPack. In my view, FLAC is really an audio compressor with the ability to store en restore foreign metadata (e.g. RIFF chunks), but not a format that is meant to be able to replicate the exact WAV/AIFF/etc. structure bit-for-bit.
Could you add a statement to the website and/or man-page like: "WavPack compresses the whole input file and can restore it bitperfect. It does this not only for the audio, but for the whole file." You can probably think of something better/catchier. That way I can point people there. Or am I mistaken and does WavPack not do this?
"In the default lossless mode WavPack acts just like a WinZip compressor for audio files" from the "About" is clear when you know what it means - but not until then (it took me quite some time too). A bit/byte more foolproofing could do yes.
But, WavPack does not support AIFF at all. (It handles CAF.) Looks like the options for AIFF are then MPEG-4 ALS and Monkey's (I haven't tested either - actually I just found out that Monkey's implemented that last year; it also handles AU/SND, if anyone cares).
But, WavPack does not support AIFF at all. (It handles CAF.)
Ha, silly me. Didn't notice that little detail :-[
Changing WinZip to 7-Zip may not make the sentence better/catchier, but at least less awkward.
The main issue, IMO, is some (if not many) users are unwilling to use more than one format, like this:
https://hydrogenaud.io/index.php/topic,83520.msg920111.html#msg920111
Well that was some discussion that did not consistently bring out the best of HA ...
Meanwhile, I tried - and failed - to get Monkey's and ALS to accept the sowt files. So there are no such options then?
Also, it seems to me that none of the codecs can force an output choice between WAVE_FORMAT_EXTENSIBLE and old WAVE. (Well ffmpeg can, but ffmpeg does not default to lossless operation in the first place, it is a different tool.)
But, WavPack does not support AIFF at all. (It handles CAF.)
Ha, silly me. Didn't notice that little detail :-[
Meanwhile, I tried - and failed - to get Monkey's and ALS to accept the sowt files. So there are no such options then?
Also, it seems to me that none of the codecs can force an output choice between WAVE_FORMAT_EXTENSIBLE and old WAVE. (Well ffmpeg can, but ffmpeg does not default to lossless operation in the first place, it is a different tool.)
Oh, looks like bryant will once again be overloaded with requests from ... people who don't use WavPack that much :-o
Do I kinda feel guilty here? I kinda should. I have like a measly (for my hoarding habits) 20 GB lossless that I need to distinguish in case I should stupidly destroy tags, and having started to read myself up on the codecs, I
kinda liked WavPack and chose that for my 2nd codec back in the day. And the more of these weird tests I do, the more I like WavPack, it behaves utterly
sane. A compressor is more than just an encoding/decoding algorithm. (And for once I shall refrain from knocking the competition.)
Gee, I was hoping to be able to edit the first post for an RC2, but I guess there’s a time limit. I hate to start a new thread, so I’ll just post here and maybe a kind mod can delete the downloadable files from the first post.
So the reason for the update was to allow the drag-and-drop feature to handle multiple files. This is not straightforward because the syntax on Windows is to have the optional second specified file be the
output file (even though wildcards are supported in the first file). Well, the solution was to add a new option “
--drop” which changes the syntax to simply a list of input files. You still can’t drop a folder (which is a whole new set of complications) but this is still a lot better than one at a time. Here’s a new example filename:
wavpack{--pause}{--drop}{-hxvm}{--import-id3}.exeHere’s the new change list:
- fixed: CVE-2021-44269 (encoding crafted DSD file causes out-of-bounds read crash)
- fixed: very long filenames cause stack-overflow crash in command-line programs
- fixed: attempting to encode raw DSD audio from stdin sometimes causes crash
- fixed: DSD to PCM decimation: small clicks between tracks and tiny DC offset
- fixed: the length stored in the WAV header not always corrected when using -i
- added: accepting brace-delimited options in the wavpack executable filename
- added: "--drop" option to Windows executables for multi-file "drag-and-drop"
- added: --raw-pcm option to wvunpack executable (does DSD --> 24-bit PCM)
- added: --no-overwrite option to wavpack executable (to resume sessions)
- improved: build system clean-up including switch to non-recursive "make"
:-[ MOD blunder: deleted files from wrong post.
But, WavPack does not support AIFF at all. (It handles CAF.)
Ha, silly me. Didn't notice that little detail :-[
Yeah, AIFF is on my wish-list, and in fact there's a half-done version in an old GitHub branch (https://github.com/dbry/WavPack/tree/aiff). I got bogged down on ID3v2.2 support which I wanted to do at the same time (don't ask). Maybe next time.
But your understanding is correct that WavPack by default creates and restores a byte-exact representation of the entire source file, regardless of the format. And in the case of CAF that includes both endian versions. Of course it doesn't really understand any of the metadata beyond the basics, so if you export to a
different format then that stuff is lost.
But I will update the webpage to make that feature clearer. When I originally wrote about the WinZip relationship I wasn't aware that saving the headers would be one of the main draws ... I was just trying to explain what lossless audio was! :)
And, BTW, thanks for the referrals! I'm actually working with some people who want to compress thousands of channels and they were referred (perhaps by you).
Here are the RC2 files... :)
And, BTW, thanks for the referrals! I'm actually working with some people who want to compress thousands of channels and they were referred (perhaps by you).
If that is with electrophysiology data, then yes, that was probably me (https://github.com/xiph/flac/issues/319). :D It is rather interesting that audio compressors like FLAC and WavPack are being used for storing other kinds of signals. In this case brainwaves, FLAC has also seen use to compress laserdisc rips (https://github.com/happycube/ld-decode/wiki/ld-compress). There are probably tons of other electrical signals with lots of spatial redundancy that can be compressed well, so I suspect that won't be the last of it.
edit: I see now the discussion on WavPack github, very interesting to read indeed.
There was such a discussion here some time too.
@polemon, did you end up with WavPack or something else? :)
MPEG-4 ALS supposedly supports 65536 channels. I have not tried to max it out. (Also, I did at one stage manage to create .als files which would make the decoder consistently crash, so it's not production ready ... and will likely never be.)
FLAC has also seen use to compress laserdisc rips (https://github.com/happycube/ld-decode/wiki/ld-compress). There are probably tons of other electrical signals with lots of spatial redundancy that can be compressed well, so I suspect that won't be the last of it.
It's first time I've heard of this, it's very interesting! I'd like to try it out, but I don't know if there is some sort of demo/free laserdisk image I can use for that.
Meanwhile, I tried - and failed - to get Monkey's and ALS to accept the sowt files. So there are no such options then?
Also, it seems to me that none of the codecs can force an output choice between WAVE_FORMAT_EXTENSIBLE and old WAVE. (Well ffmpeg can, but ffmpeg does not default to lossless operation in the first place, it is a different tool.)
For SoX supported formats (so, not APE), find keyword "wavpcm" in the link below:
http://sox.sourceforge.net/soxformat.html
For SoX supported formats (so, not APE), find keyword "wavpcm" in the link below:
http://sox.sourceforge.net/soxformat.html
The comment wasn't about being able to compress audio data in AIFF-C with sowt data, but about being able to compress and restore the file bitperfect including headers and metadata.
In this case brainwaves
Use of the name 'BrainWavPack' is now warranted :D
And, BTW, thanks for the referrals! I'm actually working with some people who want to compress thousands of channels and they were referred (perhaps by you).
If that is with electrophysiology data, then yes, that was probably me (https://github.com/xiph/flac/issues/319). :D It is rather interesting that audio compressors like FLAC and WavPack are being used for storing other kinds of signals. In this case brainwaves, FLAC has also seen use to compress laserdisc rips (https://github.com/happycube/ld-decode/wiki/ld-compress). There are probably tons of other electrical signals with lots of spatial redundancy that can be compressed well, so I suspect that won't be the last of it.
Yes, I’ve been working with them on the electrophysiology data. I hadn’t heard of those laserdisc rips either, which is interesting. I still have a bunch of those (LDs) and people sometimes forget that they were purely analog RF (even the sound was FM early on).
Any analog sampling capture is going to benefit from lossless compression (unless it’s fullscale white noise) so these applications make sense, especially when they generate so much data that they want archived in the cloud. I've worked with researchers interested in storing seismological data and even astronomical radiophysics data.
And in some cases the lossy mode of WavPack is additionally useful because the application might not necessarily require all the resolution provided by their sampling, but they don’t want to deal with the dynamic range reduction that comes with truncation.
So it seems that the status on file restore and foreign metadata is as follows, assuming compliant < 4GB (https://hydrogenaud.io/index.php/topic,122222.0.html) files, excluding raw PCM - and excluding the TTA and WMAL formats:
WAVE: WavPack/Monkey's/OptimFROG/TAK/ALS do full file restore by default. Monkey's and ALS "enforce" this: they have no provision to switch off metadata storage (but by now support piping, so can be fed just a PCM stream, and metadata can be destroyed by outputting to different format). FLAC can optionally store the metadata but has no ambition of recreating the file structure. (ref)ALAC has no such provision.
All accept both old WAVE and WAVE_FORMAT_EXTENSIBLE input, all can force WAVE output regardless of input (well TAK/OFR cannot do anything else but .wav and raw) - but none have an option to select output between the two.
RF64/BWF/w64: Essentially as .wav support as goes FLAC/WavPack/Monkey's/ALS. refalac supports RF64(/BWF?) but not w64. OptimFROG/TAK have no support for these.
AIFF: Monkey's supports big-endian with full file restore, reject the little-endian - apparenty ALS as well. FLAC reads both endiannesses as infile but upon asked to output AIFF will write only big-endian. Again FLAC has no ambition to preserve file structure.
CAF: WavPack. User can force either endianness for output. Monkey's seems to have gotten it, but I don't know what endianness(es) (haven't tested the fixes after initial .caf-supported version bugged (https://hydrogenaud.io/index.php/topic,122222.0.html)) - presumably with full restore, as that is how the ape works.
AU/SND: Monkey's only. Presumably with full restore.
Yeah, AIFF is on my wish-list, and in fact there's a half-done version in an old GitHub branch (https://github.com/dbry/WavPack/tree/aiff). I got bogged down on ID3v2.2 support
2.
two ... and then you got 2.4. And who knows about Lyrics (is that extension even possible in WAVE/AIFF?)
But if you don't need to import them to WavPack's tagset, is this in principle any worse than what you do about ID3v2.nonthree-tagged .dsf?
(don't ask).
Oops I did it again.
Yeah, AIFF is on my wish-list, and in fact there's a half-done version in an old GitHub branch (https://github.com/dbry/WavPack/tree/aiff). I got bogged down on ID3v2.2 support
2.two ... and then you got 2.4. And who knows about Lyrics (is that extension even possible in WAVE/AIFF?)
But if you don't need to import them to WavPack's tagset, is this in principle any worse than what you do about ID3v2.nonthree-tagged .dsf?
(don't ask).
Oops I did it again.
Yes, Britney, you did... :)
The original impetus for starting on AIFF file support was that I happened onto a cache of them from my then employer (a well-known high-fidelity firm). I had previously never encountered any AIFF files (not being much of an Apple guy) and had sort of naively assumed that CAF would take over AIFF because it’s
so much better in pretty much every way (it turns out that’s not necessarily a recipe for success).
Well, it turns out that cache of AIFF files was tagged with ID3v2.2 and so, based on a sample size of one, decided that implementing AIFF without ID3v2.2 would be pointless (i.e., my only test files wouldn’t even work right).
So, I will eventually put in AIFF for completeness (but probably
not AU/SND :) ) and will probably opt for ID3v2.4 support and
not ID3v2.2 (as I don’t have those files any more). Not exactly ideal for a "universal audio archiver", but with all the fuzz testing going on (check out today's bug) (https://github.com/dbry/WavPack/issues/121) it's silly to spend time working on formats that aren't going to be used.
and had sort of naively assumed that CAF would take over AIFF because it’s so much better in pretty much every way (it turns out that’s not necessarily a recipe for success).
Yeah, the CAF itself is not only 64 bit based but also flexible enough to carry virtually any audio codec with sample accurate seekability and gapless playback support. Technically it's superior to any of WAV/AIFF/AU/wave64/rf64, Still, it's a lot simpler than quicktime MOV or ISOBMFF container.
However, it's not actually possible except for a few codecs, due to lack of codec mapping definition (it's not open and has to be defined by Apple, unlike matroska or something).
And sad truth is that none of these has nothing to do with success of the format (as always).
Well, it turns out that cache of AIFF files was tagged with ID3v2.2 and so, based on a sample size of one, decided that implementing AIFF without ID3v2.2 would be pointless (i.e., my only test files wouldn’t even work right).
Do you actually need it? As far as I know wavpack has to just read/store/restore them as an opaque data chunk.
Of course it would be better If wavpack understand them and automatically convert them to wavpack container-level tags, but wavpack doesn't do it anyway for RIFF INFO chunks, I supporse?
I agree that AIFF support can come without transferring ID3v2.nonthree tags. Or even without transferring tags at all - if that is what it takes, then ... why not? (Except it still takes time and hassle, of course).
As for .wav, I don't know all the format's tagging whereabouts, but
wavpack --import-id3 wav_file_tagged_by_mp3tag.wav
wavpack --import-id3 wav_file_tagged_by_foobar2000.wav
will indeed give me a tagged .wv file.
Provided the tags are not too big, in which case WavPack should likely make a clearer warning to the user that NO OUTPUT FILE WRITTEN. The console output does not give any information that the .wv file is deleted.
Same is the case (well or, fails to be the case) when encountering the "ID3v2 import: bad frame identity".
Not sure what is best behaviour here, really - IMHO, writing a file makes the claim that "I was able to do this!", which is a case for deleting it (and of course, telling the user clearly!). On the other hand, you can imagine an annoyed user who has waited for -hhx6'ing a full multichannel DVD just to see the file deleted.
Possible solution 1: Check metadata size first.
Possible solution 2: Rather than deleting, rename to filename.wv.NOTAGSIMPORTED.tmp and tell the user that they can change the name and use a different tagger to copy the tags, or re-run with --allow-huge-tags
(And: I don't anymore remember what message WavPack returns when I try to import ID3v2.otherthanthree. Likely it should tell me to RTFM.)
Yeah, playing around with it now and seeing that the ID3 import stuff is a little rough around the edges. Will have to improve that when I add ID3v2.4 support.
The first thing is that we look specifically for ID3v2.3, so even though the parsing code has a nice message about only supporting v2.3, we never get there because ID3v2.4 is treated like “no tag found”, and I didn’t want to error out on that and delete the output file because it’s nice to specify the --import-id3 by default (like in my drag and drop example).
And yeah, deleting the output file is a little severe for exceeding 1 MB, but a warning that nobody sees isn’t great either (drag and drop). People will notice a deleted output! :)
There are a couple of tricks that I use to reduce some of the potential frustration of waiting for a long compression and then having it abort. The first is when I’m ripping CDs (or whatever I’m doing to obtain the initial WavPack files) I usually just use the default lossless mode because it’s really safe and I don’t want to slow what I’m doing (this is one reason I hesitate changing the default). Afterward I use the re-encode feature to get the modes I want (“high” lossless, or hybrid lossless, or even lossy) and other tweaks.
Another thing I do is use wvtag afterward to clean up the tags and add the artwork if it isn’t there already, and you can give the --import-id3 option there also and it will rescan the end of the file for potential tags. That way if you get the dreaded “over 1MB” error you can just do it again (or use a more reasonable image).
In any event, thanks for all the comments, suggestions, and testing! I've done some clean-up of the web site, including a link to the cool WebAssembly player that Peter McQuillan created (https://www.wavpack.com/WebAssembly/index.htm). There's still some more clean-up to do there (especially in the ancient "hardware" section), but for now the release is done:
WavPack 5.5.0 Release on Github (https://github.com/dbry/WavPack/releases/tag/5.5.0)
Cheers!
Thanks, but I can't get the WebAssembly player to work on Firefox, the "Load WavPack File" does nothing. Is is a known issue or somehow related to Firefox's default security settings?
Also, the "XHTML 1.0" link indicates the page is no longer valid.
Yeah, I have only got it work on Chrome and Edge myself and figured Firefox was an outlier. I just looked a little deeper and found the problem to be the file picker:
Uncaught (in promise) TypeError: window.showOpenFilePicker is not a function
<anonymous> https://www.wavpack.com/WebAssembly/mywavpack_worklet.js:31
Googling that got me to this page (https://developer.mozilla.org/en-US/docs/Web/API/Window/showOpenFilePicker) which shows Firefox
not supporting the file picker. So I'm guessing that the Firefox people consider that too dangerous?
Of course, a real world application of this (if there was one) would probably
not involve trying to play existing files on the user's computer but would involve files stored on a server somewhere.
Perhaps Peter will chime in with more context.
Yeah, I have only got it work on Chrome and Edge myself and figured Firefox was an outlier. I just looked a little deeper and found the problem to be the file picker:
Uncaught (in promise) TypeError: window.showOpenFilePicker is not a function
<anonymous> https://www.wavpack.com/WebAssembly/mywavpack_worklet.js:31
Googling that got me to this page (https://developer.mozilla.org/en-US/docs/Web/API/Window/showOpenFilePicker) which shows Firefox not supporting the file picker. So I'm guessing that the Firefox people consider that too dangerous?
The File System Access API (of which
showOpenFilePicker is a part) is currently an unofficial draft (https://wicg.github.io/file-system-access/), i.e. not standardised. Browsers should not be expected to implement and support it, and indeed CanIUse indicates that the support for the API is spotty and inconsistent (https://caniuse.com/native-filesystem-api).
Here's a guide on how to do this with standardised APIs. (https://developer.mozilla.org/en-US/docs/Web/API/File_API/Using_files_from_web_applications)
For wiki etc, and since trying to create a 257 channel .wav is a hassle and I'd anyway have to ask to get the "officially supported":
(1) I see the WavPack front page has changed the description of channel count to "up to at least 256 channels".
* Does that mean you have removed the 256 channel cap to accommodate that BrainWavPack application
and if so:
* Would a "reasonably accurate" description be that WavPack supports 256 channels but will accept higher channel count provided that other limitations are satisfied (not all tested) - up to a hard definite maximum of 4096 (https://github.com/dbry/WavPack/issues/117#issuecomment-1129148365)?
(2) "integer sampling rates up to 1 GHz" it says. I have gotten it to work to 4 Gi - 1.
* Is it kinda the same as I suggested above, officially 1 GHz but won't object until 4 GiHz?
Just because "supports any .wav sampling rate (i.e. integers between 0 4 GiHz)" is easier to explain than "you see there is this thing called 'long integer' which is why you can cover signals you won't even call audio ...".
For wiki etc, and since trying to create a 257 channel .wav is a hassle and I'd anyway have to ask to get the "officially supported":
(1) I see the WavPack front page has changed the description of channel count to "up to at least 256 channels".
* Does that mean you have removed the 256 channel cap to accommodate that BrainWavPack application
and if so:
* Would a "reasonably accurate" description be that WavPack supports 256 channels but will accept higher channel count provided that other limitations are satisfied (not all tested) - up to a hard definite maximum of 4096 (https://github.com/dbry/WavPack/issues/117#issuecomment-1129148365)?
The
official support remains 256 channels, which seems high enough to me for all but the most esoteric brain-wave type stuff.
The format and library have supported 4096 channels for a while, but has not been well-tested, and there's not been a way to even make them. With this release I have added an
undocumented option
--raw-pcm-ex that lifts the limitation for experimentation purposes (and the brain-wave guys).
But just one example of how this can manifest in weirdness, if you unpack one of these files to another uncompressed format (like WAV) then that resulting WAV file will
not not be compatible with WavPack. So maybe in the future, after some more testing, I will lift the 256 channel limit everywhere. Or maybe have some sort of
--chill-out option for that (the alternative always reminds me of the airport).
(2) "integer sampling rates up to 1 GHz" it says. I have gotten it to work to 4 Gi - 1.
* Is it kinda the same as I suggested above, officially 1 GHz but won't object until 4 GiHz?
Just because "supports any .wav sampling rate (i.e. integers between 0 4 GiHz)" is easier to explain than "you see there is this thing called 'long integer' which is why you can cover signals you won't even call audio ...".
Here again the format and library limit is 2 Gi - 1, so I suspect that's what you got to work. And of course since the WAV header has another unsigned 32-bit field "bytes per second", its
real limit varies with channel count and sample depth. Oof.
I've had at least one CVE around wacky sample rates and so have gotten a little shy in this area, which may explain why I set the
--raw-pcm limit to 1 GHz. It certainly could have been 2 GHz, and you
can get that with WAV (disregarding the other issue above), but 1 GHz is easy to remember and seems like a reasonable
upper limit for most sampled physical phenomena.
The cool formats (AIFF and CAF) support floating-point values and I really should have done that, and maybe will in the future. Handy when specifying 0.01 Hz for barometric data, for example, but then I need to add months and years to the "duration" display. You can see how this can be complicating in unpredictable ways.
Sorry I couldn't just answer "yes" or "no" for these and thanks again for your valuable updates to the Wiki! :)
Edit: So with the last few wiki updates, I think I have removed some errors and not introduced new ones? https://wiki.hydrogenaud.io/index.php?title=WavPack&diff=35791&oldid=35777
Here again the format and library limit is 2 Gi - 1, so I suspect that's what you got to work.
Looks like my memory had gotten too much pro-WavPack bias after everything else working just as good with WavPack as with anything else. So here are the corner cases where some animals are outpacking WavPack, I think:
* 2Gi fails for WavPack. (Also ffmpeg cannot handle it.)
* 4Gi - 1 is accepted by Monkey's and OptimFROG and also by MPEG-4 ALS which with -v returns this fun output:
Audio format : int / 16 bit / -1 Hz / 1 ch
Bit rate : -0.0 kbit/s
Playing time : -100000000.0 sec
PCM file size: 200000104 bytes
ALS file size: 11466700 bytes
Compr. ratio : 17.442 (5.73 %)
Average bps : 0.917
Average rate : -0.0 kbit/s
Processing took 7.59 sec (-13168290.8 x real-time)
The cool formats (AIFF and CAF) support floating-point values
Quick and pigsty-dirty idea for those who use have allocated 4 bytes and declared it a long signed integer variable for something inherently positive: use the sign to indicate that it is float :))
Perhaps Peter will chime in with more context.
I'll have a look at this again to see if I can get it working with more supported/standard APIs
Thanks. If it is not too complicated, it would be interesting to have a browser-based decoding speed benchmark with user-supplied WavPack files too.
And of course since the WAV header has another unsigned 32-bit field "bytes per second", its real limit varies with channel count and sample depth.
My pathetic SATA SSD is not fast enough to handle this data rate, but then the file can only be 1 second long and therefore not so useful either. So WAV is basically useless for this purpose. Luckily some of the "best" audio formats like DSD2048 is still below 100MHz :))
The cool formats (AIFF and CAF) support floating-point values
Quick and pigsty-dirty idea for those who use have allocated 4 bytes and declared it a long signed integer variable for something inherently positive: use the sign to indicate that it is float :))
Haha, yeah, that would work! At first I thought you would only get 31 bits for the float value, but it would actually be the negation of the sample rate. Clever! Unfortunately I can't add it to WavPack that way because it wouldn't be backward compatible. I would need to have the integer version remain (and be as close as possible to the actual value) and then have another field for the floating point that old decoders would never see.
And what's crazy is that in AIFF it's actually an 80-bit extended precision float! I wonder what language back then (1988) supported that (it's poorly supported now). They tried to cover all the possibilities but never considered anyone would need over 4 GB. Anyway, for CAF they went back to a more reasonable 64-bit double.
And of course since the WAV header has another unsigned 32-bit field "bytes per second", its real limit varies with channel count and sample depth.
My pathetic SATA SSD is not fast enough to handle this data rate, but then the file can only be 1 second long and therefore not so useful either. So WAV is basically useless for this purpose. Luckily some of the "best" audio formats like DSD2048 is still below 100MHz :))
4 GB per second is crazy fast. I'm not sure PC RAM can go that fast.
My Tektronix oscilloscope is up to 8 GB per second (2 GS/s * 4-ch * 8-bit) but only goes for 2.5K samples (i.e., just over 1 uS).
RAM can go that fast - most storage, however, would find it rather challenging.
And what's crazy is that in AIFF it's actually an 80-bit extended precision float! I wonder what language back then (1988) supported that (it's poorly supported now).
Probably Pascal. It was a standard data type on the 68k FPU and in Apple's mathematics libraries.
Concerning why put AIFF off until WavPack can
import AIFF metadata:
WavPack does not universally have that capability with WAVE either. (E.g. here, (https://soundcloud.com/hypnoskull/hypnoskull-1992-l-rm-2012) Mp3tag sees a COMMENT tag in non-ID3 RIFF.)
So ...
porcine ignorant question time again, but as far as WavPack cannot universally import WAVE RIFF metadata (that it stores perfectly) and WavPack cannot universally import metadata from DSD files (that it stores perfectly) - is there anything in particular about AIFF except getting the --import-id3 to work [which, anyway, it doesn't do for all ID3]?
... well there might be, of course, not related to metadata, but ...
And what's crazy is that in AIFF it's actually an 80-bit extended precision float!
... the choir demands 64-bit float "just in case" they would ever encounter such a file ;D
Concerning why put AIFF off until WavPack can import AIFF metadata:
WavPack does not universally have that capability with WAVE either. (E.g. here, (https://soundcloud.com/hypnoskull/hypnoskull-1992-l-rm-2012) Mp3tag sees a COMMENT tag in non-ID3 RIFF.)
So ... porcine ignorant question time again, but as far as WavPack cannot universally import WAVE RIFF metadata (that it stores perfectly) and WavPack cannot universally import metadata from DSD files (that it stores perfectly) - is there anything in particular about AIFF except getting the --import-id3 to work [which, anyway, it doesn't do for all ID3]?
No, the two projects (AIFF and ID3v2.4) are completely orthogonal. For silly historical reasons I was doing them together, but I could certainly do AIFF first and it would just work with ID3v2.3 (like everything else). That may even be the most common ID3 variant...
And what's crazy is that in AIFF it's actually an 80-bit extended precision float!
... the choir demands 64-bit float "just in case" they would ever encounter such a file ;D
Don't know if I've ever mentioned this story, but I got a 64-bit float WAV file from a guy who was asking about support for it. The very first thing I tried was to verify that the 64-bit float values were in fact not representable in 32-bit floats, because otherwise this would just be 100% bloat. Well, every sample survived the roundtrip to 32-bit losslessly! I got him to convert it to 32-bit float (with FFmpeg or Foobar2000, I don't remember) and he agreed it sounded the same. I've been doubly dubious of this ever since.
At one point I was thinking that it would be fairly east to implement this in a backward-compatible way by dividing each 64-bit float value into the sum of two 32-bit floats. One could be stored in the regular stream and the "correction" stream would be stored such that old decoders would ignore it (or omitted in lossy mode, obviously). Unfortunately when I looked a little deeper I realized that since the mantissa is more than twice as large on 64-bit floats (52 vs. 23 bits) this won't work. :(
Thanks. If it is not too complicated, it would be interesting to have a browser-based decoding speed benchmark with user-supplied WavPack files too.
I've updated the code so it should now work with Firefox too.
https://www.wavpack.com/WebAssembly/index.htm
If you're interested in the source code etc, that can be found on my GitHub page:
https://github.com/soiaf/WebAssembly-WavPack
I also wrote a very quick (read 'hack') system to show how quick it can decode a file. David was kind enough to also host this on his site at
https://www.wavpack.com/WebAssembly/timer.htm
It's certainly interesting to see how fast it can decode WavPack files, from my very brief tests, Firefox seemed to be the fastest browser, but I don't know if this is better JavaScript or WebAssembly processing (or both!). However maybe others will get different results!
Thank you very much! Here is what I got from a CDDA image I encoded many years ago using WavPack 4 with high setting, I am using i3-12100, 16GB DDR4 on Windows 10 and Firefox:
It took 27094 milliseconds to decode the file
We decoded 187598460 samples
Another CDDA image encoded with WavPack 5 fast settings:
It took 10299 milliseconds to decode the file
We decoded 133040880 samples
I repeated the test several times and got similar results.
Thanks
@soiaf ... very nice!
And thanks
@bennetng for posting your timing results.
It's amazing to me that WavPack can decode in a browser at over 100x realtime! Just for fun I opened up 5 different tabs and had a WavPack file playing in each one and was still under 20% CPU load.
Just for the braintrust ...
There is actually some scientific literature on lossless compression of EEG data (https://scholar.google.com/scholar?q="lossless+compression"+"eeg") - that Google Scholar search returened 1620 hits although not all are relevant. In particular, none of the hits on https://scholar.google.com/scholar?q="lossless+compression"+"eeg"+"wavpack" (https://scholar.google.com/scholar?q="lossless+compression"+"eeg"+"wavpack") are ...
(Shorten was used here, as they found a 1999 paper that tested it: https://dr.ntu.edu.sg/bitstream/10356/101141/1/A%20two-dimensional%20approach%20for%20lossless%20EEG%20compression.pdf )
- fixed: DSD to PCM decimation: small clicks between tracks and tiny DC offset
I just read the other thread about the audible clicks, but out of curiosity, what was the DC offset bug in the previous versions?
- fixed: DSD to PCM decimation: small clicks between tracks and tiny DC offset
I just read the other thread about the audible clicks, but out of curiosity, what was the DC offset bug in the previous versions?
It was a rounding omission in the DSD to PCM code that introduced a negative DC offset, but only 1/2 of a LSB (at 24 bits), so nothing that could be audible (or even measurable except on silent test files).
I happen to see it when I was in there attempting to reduce the transition clicks and fixed it more for “correctness” than anything else.
P.S. This was the offending source line (https://github.com/dbry/WavPack/blob/095f5e98ba7cccef468bdeb32f87e2079b9ebbb8/src/unpack_dsd.c#L616).
It was a rounding omission in the DSD to PCM code that introduced a negative DC offset, but only 1/2 of a LSB (at 24 bits), so nothing that could be audible (or even measurable except on silent test files).
I happen to see it when I was in there attempting to reduce the transition clicks and fixed it more for “correctness” than anything else.
P.S. This was the offending source line (https://github.com/dbry/WavPack/blob/095f5e98ba7cccef468bdeb32f87e2079b9ebbb8/src/unpack_dsd.c#L616).
Thank you, bryant, for the explanation!
If I want to use WvUnpack to convert DSD to PCM and avoid sample extrapolation, I was considering the following process:
- append 2 seconds of digital silence to the beginning of the DSF file
- compress it with WavPack
- modify the line "*samples++ = sum >> 4;" to "*samples++ = (sum + 8 ) >> 4;" in the source code (5.4.0)
- rebuild WvUnpack from modified version 5.4.0
- unpack the compressed DSF file using the "--wav" option
- and finally use something like SoX to remove the first 2 seconds (then resample or apply a low-pass filter)
This way, any clicks that may occur will be contained within the added 2 seconds and can be discarded after conversion. Do you see any potential issues with this process?
I’m not sure exactly what you’re trying to do, or avoid, but let me see if I can clarify things.
The issue with WavPack’s DSD to PCM conversion is that because it attempts to preserve the exact length of the DSD file, and the decimation filter requires some history, the first 6 PCM samples do not have the actual correct history to work with. In version 5.4.0 (and before) those samples were created by simply pre-filling the filter with DSD “silence”.
The problem with that was that, unlike PCM where silence is represented with zeros, DSD silence is represented with various repeating patterns of +1 and -1, and when two different “flavors” of such silence are joined (or even the same flavor is joined with itself, but offset) a glitch results. And, of course, if the file wasn’t silent at the beginning then another kind of glitch would be created, although this was less likely to be audible because it would be masked by signal.
The solution for version 5.5.0 was to replace just those 6 samples with a linear extrapolation of the beginning of the PCM audio. In all the audio clips I tried, including those generously supplied by the OP, with both silent and non-silent transitions, this eliminated the audible glitch as far as I could tell. There are still probably samples where it might still be audible, but I suspect that this is better in the vast majority of cases. Unfortunately, that thread’s OP never responded as to whether it fixed his problem, nor did anyone else ever indicate that it did or didn’t make an improvement.
So obviously my first question is are you still hearing glitches in some circumstances? Is 5.5.0 better or worse than 5.4.0? Or is this all somewhat academic?
As for an answer to your question, if you want to use WvUnpack to do the DSD to PCM conversion without the extrapolation, the easiest way to do that is to add --skip=6 to the command line to discard those extrapolated samples. That will work perfectly with silent transitions, but of course might introduce a small glitch in non-silent transitions. It also will work perfectly in a stand-alone case where the file is never played gaplessly with the previous file.
Unfortunately, your suggested fix is basically the same thing that I did originally and a glitch may occur where your selected flavor of silence meets the silence of the source file. It is in fact exactly the same except you don’t have to use 2 seconds of silence; 56 bits of silence will do it.
As I type I realize that a more complete fix would have been to first look at the first few bytes of DSD audio and search for a repeating pattern. If one is found then pre-filling the decimation filter with an extrapolation of that would be ideal, and only reverting to the extrapolated PCM when that doesn’t work. Maybe I’ll look at that for the next release.
Does this help and/or make sense?
It does help (and makes sense)!
Regarding your questions, version 5.5.0+ is better than 5.4.0, but I did hear a few glitches in specific circumstances. For instance, when transitioning from KV 16 (the files shared by the OP) first movement to the second movement, I could hear a tiny click on the right channel. However, such clicks are not very noticeable.
I used a 2-second buffer because that is typically the pre-gap duration before the first track of a SACD. My intention was to convert some of my DSD/SACD collection to PCM so I could play them on my iPhone. To completely avoid clicks in my use scenario, I was thinking to concatenate all tracks from an album to a single file, convert it to PCM, discard the first pre-gap, and then split the tracks. Since the first pre-gap is discarded, extrapolation is unnecessary in this case. I didn't realize until now that only the first 6 PCM samples would be affected. I did some tests in the past few days, using modified versions 5.4.0 (sum → sum + 8 ), 5.5.0, and 5.6.0 to convert a DSD album as a whole to PCM. The results were identical after the first pre-gap was trimmed out, so I'm happy with it :-)
Okay, great. Thanks for the feedback!
Yeah, you'll definitely get the best results by leaving the image in the full album version as long as possible and not splitting until after all downsampling and processing (if at all). When I use sacd_extract I always use the "full image" option and embed the cuesheet which Foobar is happy to honor.