The major enhancement of WavPack 5.0 will be breaking the 2/4 GB .wav limit and additional source file formats. At this point I have added BWF/RF64 (.wav extension), Sony's Wave64 (.w64) and Apple's Core Audio Format (.caf). Of course, all these formats still require PCM audio data to be usable in WavPack. I have also added a big-endian option to the raw PCM mode.
Part of this upgrade includes the capability of decoding and seeking in WavPack files larger than 2/4 GB. To demonstrate that I have updated the winamp plugin and will work on updating the DirectShow filter and Audition plugins in the near future.
These enhancements have required changes to the libwavpack API (which is part of the reason to bump the major revision). However, all of the existing APIs remain so that applications can move to the new library without any immediate need for changes.
Most people who simply use WavPack for their personal music will probably not be interested in the 5.0 changes, and the 4.80 release will be kept current. However, for more advanced use cases these improvements (along with 32-bit audio and unlimited channels) will hopefully keep WavPack relevant.
Any testing results or comments are, as always, greatly appreciated.
edit 1: replaced alpha attachments with alpha2
edit 2: replaced alpha2 attachments with alpha3
edit 3: deleted attachments
edit 4: deleted alphas link, latest release is here (https://hydrogenaud.io/index.php/topic,113213.0.html)
Hi David, Many thanks for adding this feature, this will save a huge amount of space in my multitrack audio projects archive!
Just done 1 quick test
firstly its very fast, compressed a 5.5GB stereo wav file in about 80 seconds, and looking at the cpu usage it was only utilizing 14% of my cpu
compressed 5.5gb into 2.7gb
unpacking was even quicker, about 45 seconds, but once it unpacked, it added the wrong file extension (it added .w64)
However the CRC of the original WAV and the unpacked .W64 file matched, so this is no major issue at all.
I have a load more stuff to compress this weekend so I will no doubt do some more testing, many many thanks for this excellent addition!!
Garry
(http://s32.postimg.org/tjrsin35x/wavpack_test.jpg)
Hi Garry, thanks for trying this out and posting your results...very cool!
It's interesting about your .w64 extension though. That really could only happen if your source file actually was a Sony Wave64 file, which are supposed to have a .w64 extension (at least according to the spec and everything I've read online). WavPack detects the file type by looking at the file data rather than using the extension, and stores the "recommended" extension in the file to be used when unpacking, which explains what you're seeing. I guess maybe the best solution would be for wavpack to store the actual extension rather than the recommended one...I'll fix that in the next version.
About processing a lot more files, obviously more testing is better, but please keep in mind that this is still an alpha version and I reserve the right to change something that might make your files unusable on future decoders (for example, any files you do now will always decode to .w64, even after I fix that issue). I don't think it's real likely I'll change something fundamental, but it's possible.
Thanks again!
I created 6 GB wav file (not w64) with foobar2000. It has riffsize=datasize=0xFFFFFFFF.
Then I encoded it with -i option: wavpack.exe -i -m 6GB.wav And wavpack writes to the console during encoding: "creating 6GB.wv, 152% done..." and then "created 6GB.wv in 332.14 secs (lossless, -12.40%)" (the size of the resulting .wv file is ~4.5GB)
After the decoding (wvunpack.exe 6GB.wv) the decoded file has .wav extension, riff size = 0x900647b4 and data size = 0x90064790. Probably wvunpack should write 0 or -1 into these fields in this case?
Or maybe it should default to .w64 output format in this case?
P.S. foobar2000 can decode but cannot seek if the WV size is bigger than 4 GB. No problems with the bundled Winamp plugin.
After the decoding (wvunpack.exe 6GB.wv) the decoded file has .wav extension, riff size = 0x900647b4 and data size = 0x90064790. Probably wvunpack should write 0 or -1 into these fields in this case?
Or maybe it should default to .w64 output format in this case?
Haven't yet tried it myself, but automatic switching to RF64 on large file looks like already implemented in the source (21 days ago).
https://github.com/dbry/WavPack/blob/five/cli/riff.c#L308 (https://github.com/dbry/WavPack/blob/five/cli/riff.c#L308)
If it's not working as intended, maybe there's a bug or something?
I downloaded sources form github, and this version doesn't write progress message with >100% value.
But WriteRiffHeader() is not called here. Looks like wvunpack calls DoWriteFile(outfile, WavpackGetWrapperData (wpc), WavpackGetWrapperBytes (wpc), &bcount) that writes audio header.
But WriteRiffHeader() is not called here. Looks like wvunpack calls DoWriteFile(outfile, WavpackGetWrapperData (wpc), WavpackGetWrapperBytes (wpc), &bcount) that writes audio header.
Oh, I see. wvunpack normally just restores the stored wrapper (to behave lossless including the container), so it cannot create new header or switch to RF64 or something unless explicitly told to.
Better placeholder length should be fine.
I downloaded sources form github, and this version doesn't write progress message with >100% value.
This is a little weird because I have not made any changes in the repository since the alpha release. Could this be a compiler difference (mingw?), or perhaps you didn't really use the same source file or command-line arguments?
As for the other stuff, I have duplicated the issues, and also discovered that the funny progress percentage and incorrect compression ratio report have been around for a while. I suspect that I wanted the -i option to not do weird things if the wav headers actually were correct (or almost correct) and so I assume they are correct and behave accordingly. I don't really like the behavior, and I'll look into it some (and it does really bad stuff if, for example, the header indicates a 1 sample file).
As for the writing bad values into the riff size and data size, that problem has also been around a while. As nu774 says, I don't really want to write a completely different (and possibly incompatible) type of header unless the user specifically asks for it (via -r to wavpack or -w to wvunpack). However, I did decide at some point that I would attempt to fix a bad header if -i was specified, but of course in this case the header can't really be fixed and I've actually made it worse. I'll definitely change this to simply leave the header alone (lossless) unless I can actually fix it correctly.
Thanks for the testing and sharing your results!
This is a little weird because I have not made any changes in the repository since the alpha release. Could this be a compiler difference (mingw?), or perhaps you didn't really use the same source file or command-line arguments?
I recompiled wavpack.exe from the sources that I downloaded previously, and I can confirm that they also show > 100% progress. But this also means that I don't have my prevoius compiles, so I can't re-test them...
Thanks so much for all the testing and suggestions; it's greatly appreciated!
I have fixed all the issues reported here and on GitHub and am releasing alpha2 (downloads replaced at top of thread).
- store (and restore) the actual file extension instead of the file type default
- explicitly store the source file type instead of relying on the header, because we might not store the header (e.g., wavpack -r)
- actually ignore length specified in header when we're told to, but try to determine length from file size or, if we can't (pipes), don't display progress or compression percentage
- never try to fix a header that we didn't make, even if ignoring length (we are supposed to be lossless)
- handle CAF channel layouts that are not in MS order and make the channel layout and reordering information available to applications (but applications that don't know about this will get MS order)
- handle CAF channel description arrays and map them to MS order as far as they'll go (no re-ordering and a decoding application will have to manually parse the description array to access it)
- when unpacking, always use the correct extension and data format (endian) for the output format
- add display of the source file format and extension to the wvunpack -ss command
- make sure transcoding copies all the new features across
- several minor fixes
Files created with the previous alpha release should continue to be usable, unless they are subject to one of the above issues. I changed the way I stored some information in the WavPack file, but will continue to honor the original method.
Cheers!
Hi David, Many thanks for the update, yep I can confirm all is perfect now, 5gb .wav before = .wav after wavpack & decompress... & the hash's match, amazing stuff! thanks so much
(http://s33.postimg.org/8lbr3ykvz/Capture.jpg)
Hi David, Many thanks for the update, yep I can confirm all is perfect now, 5gb .wav before = .wav after wavpack & decompress... & the hash's match, amazing stuff! thanks so much
Cool...thanks for testing and letting me know!
BTW, I am considering dropping support for WavPack files created prior to version 4.0, which was introduced in 2004. Files from versions before that (i.e., 3.97) are not compatible with Rockbox or DirectShow or Gstreamer, are not quickly seekable, and can be completely destroyed with a single bit error. I have therefore long recommended that those files be transcoded to the newer version, and doing this is especially easy now that the wavpack command-line program can do it automatically.
However, I originally said I would never drop support for older files (never say never!) and so I consider this with some trepidation even though the sources and binaries for 4.80 will be available as long as WavPack is available. It seems to me that Matt did a similar thing with Monkey's Audio and got grief. Does anyone reading this have a collection of pre-4.0 files, or have a strong opinion?
Of course there's an appropriate XKCD comic...be sure to read the mouseover! (http://www.xkcd.com/1683/)
BTW, the definition of CUR_STREAM_VERS differs between include\wavpack.h and src\wavpack_local.h
BTW, the definition of CUR_STREAM_VERS differs between include\wavpack.h and src\wavpack_local.h
Thanks...I'll fix that! :)
I may just delete it from wavpack.h because only the MIN and MAX should be used by anyone outside the library.
I am considering dropping support for WavPack files created prior to version 4.0
I don't have any such files but in principle I'm against reducing compatibility. Dropping support for old files forces any self-respecting player to use two versions of the decoder library.
Hello, I use Wavpack to reduce the size of files recorded from a software called SDRSharp.
It is a radio receiving software capable of saving wav files containing the baseband of several megahertz.
Wavpack works very well with files smaller than 16 GB, but if i try to compress files bigger than that, Wavpack says "xxx.wav has too many samples for Wavpack!"
Is this to be expected or is it a bug?
The files are of the RF64 format.
Is this to be expected or is it a bug?
To be expected.
Max. number of samples is 4 294 967 294, so for 16-bit stereo or 32-bit mono signal max. filesize is 16GB.
For samplerate=44100, max. duration is 27 hours, for 192kHz it's more than 6 hours. But if you have several megaherz...
I thought the raw option would ignore the WAV or W64 restrictions. If I use the raw option it crops my file. This is the output i get:
wavpack.exe --raw-pcm=100000,16,2 -hh -x1 --pre-quantiz
e=12 *.wav
WAVPACK Hybrid Lossless Audio Compressor Win64 Version 5.0.0-alpha2
Copyright (c) 1998 - 2016 David Bryant. All Rights Reserved.
*** This is an experimental alpha version and should be used for ***
*** testing only. Please do not use it for important archiving!! ***
creating 18-12-35_91700kHz.wv, 1111111111111111111111111111111111111036% done...
11111111111111111111111111111111111111111111111111111111111111111111104% done...
11111111111111111111111111111111111111111111111111111111111111111111172% done...
11111111111111111111111111111111111111111111111111111111111111111111004% done...
11111111111111111111111111111111111111111111111111111111111111111111072% done...
11111111111111111111111111111111111111111111111111111111111111111111140% done...
11111111111111111111111111111111111111111111111111111111111111111111208% done...
created 18-12-35_91700kHz.wv in 9450.87 secs (lossless, -1049.57%)
And the resulting file is useless.
I can pack the original wav file with OptimFROG and unpack with raw option without problems. But OptimFROG is VERY slow.
I thought the raw option would ignore the WAV or W64 restrictions. If I use the raw option it crops my file. This is the output i get:
wavpack.exe --raw-pcm=100000,16,2 -hh -x1 --pre-quantiz
e=12 *.wav
I'm not sure what you mean by "the resulting file is useless". If you use --pre-quantize with --raw-pcm then your header is going to pre-quantized, which it's not going to like. Probably your unpacked file will not be recognized after that, even though the only problem is some corrupt bytes in the header.
Anyway, --raw-pcm is not going to get around the 2^32 sample problem, but using "-i" to ignore length in header might (I have to try it), but that would be for archiving only (you could only restore the entire original WAV/RF64 file). Would that work for you? What do you do with these files after you compress them?
I considered at one point extending the 2^32 samples limit at one point (there's spare room in the header to do 1 Tera-sample) but I couldn't think of a reasonable use case that would require it – obviously I was not considering megahertz sampling rates! I'll take a look again (although I think it's a little bit of a mess).
BTW, another limit I just noticed is that I limit the sampling frequency to around 16.7 MHz. Unlike the other issue, that one's easy to fix...
I am considering dropping support for WavPack files created prior to version 4.0
I don't have any such files but in principle I'm against reducing compatibility. Dropping support for old files forces any self-respecting player to use two versions of the decoder library.
Ah, thanks Case, it never occurred to me that a player developer would do that. I just assumed, perhaps naively or paternally, that if I dropped support for older files in the decode library (and all my tools and plugins), then developers would follow my lead (and perhaps be relieved) rather than go to the trouble of using dual libraries. I definitely would not want them to have to do that.
Another option would be for me to drop support from my tools and plugins, but leave the support in the decoder library and have it disabled by default. That way I would create almost no extra work for developers who wanted to retain the support, but would give me a chance to make my point, and maybe find out how many of these file are actually in the wild.
I thought the raw option would ignore the WAV or W64 restrictions. If I use the raw option it crops my file. This is the output i get:
wavpack.exe --raw-pcm=100000,16,2 -hh -x1 --pre-quantiz
e=12 *.wav
I'm not sure what you mean by "the resulting file is useless". If you use --pre-quantize with --raw-pcm then your header is going to pre-quantized, which it's not going to like. Probably your unpacked file will not be recognized after that, even though the only problem is some corrupt bytes in the header.
Anyway, --raw-pcm is not going to get around the 2^32 sample problem, but using "-i" to ignore length in header might (I have to try it), but that would be for archiving only (you could only restore the entire original WAV/RF64 file). Would that work for you? What do you do with these files after you compress them?
I considered at one point extending the 2^32 samples limit at one point (there's spare room in the header to do 1 Tera-sample) but I couldn't think of a reasonable use case that would require it – obviously I was not considering megahertz sampling rates! I'll take a look again (although I think it's a little bit of a mess).
BTW, another limit I just noticed is that I limit the sampling frequency to around 16.7 MHz. Unlike the other issue, that one's easy to fix...
Sorry for the late reply.
The resulting file is useless because it is cropped to about 16 GB but the original file was 34 GB.
Anyway this is not a problem anymore as I have set SDRSharp to record max 16 GB files and then split to a new file if I record that long.
And to explain what I use those files for:
It is a hobby of mine to listen for distant radio stations that are usually impossible to hear. Described here: https://en.wikipedia.org/wiki/TV_and_FM_DX
I use a software called SDRSharp with the software defined radio hardware "Airspy" that can sample 10 MHz of radio spectrum in 12 bits resolution and save it in 16 bit WAV files (or RF64 with a plugin I use).
Then afterwards I have like a "time machine" so I can tune around in that piece of radio spectrum I have recorded.
The recordings take up a huge amount of space but I want to have an archive of those unusual receptions where there is much going on at different frequencies at the same time. WavPack saves about 55 to 80 percent of disk space with these files.
Thank you for your answers and your excellent software!
Thanks for the description! So, I imagine that you must unpack the entire archive back to RF64 before your "player" can use it, right? It seems like it would be possible to record directly to and play directly from WavPack (and save a lot of disk trashing), but that would obviously require a new plugin.
In any event, I'm working on eliminating the 4 Giga-sample limitation (I'm sure it will come up again), so hopefully that will make this even a little easier.
Cheers!
Yes, I have to unpack the files to play them in SDRSharp.
When I record at 10 MHz bitrate I use a dedicated SSD because my mechanical drives are too slow to avoid dropped data.
I think the slow hard drive problem would be gone if it was possible to record directly to WavPack, but then maybe the CPU would be the bottleneck fore some computers. Not for mine I think.
I have sent a comment to the developer of the recorder and player plugins.
Link to my comment (page is in Russian, there is a Google Translate tool on the page):
http://rtl-sdr.ru/page/obnovlen-file-player-1#comment-2148
I have again fixed all the issues reported on GitHub and here (including "...too many samples...") and am releasing alpha3 (downloads replaced at top of thread).
I also updated the DirectShow filters to handle all the new large files, and am including those installers in the downloads in addition to a new winamp plugin.
- increased maximum sample count from 4 giga-samples to almost 1 tera-sample (2^40)
- improved handling of CAF channel description arrays so that all MS channels are reordered to the front (and bitmasked)
- add --raw-pcm-skip option (wavpack) to ignore specified byte count (headers?)
- allow minus --skip values (wvunpack) to specify start point relative to EOF
- add optional parameter to wvunpack -f to display specific item only
- support sampling rates over 16777215 Hz (probably non-audio)
- several minor fixes
Thanks so much for all the testing and suggestions; it's greatly appreciated!
Cheers!
Alpha 3 is working great for my needs. Many thanks for making it accept larger files!
Now I'm waiting for a release... :)
I'm glad the latest alpha is working for you!
I'm currently working on adding another capability for the 5.0 release that is taking longer than I expected, so I'm not sure the final release will be real soon. In fact, I'm sure they'll be another round of alphas for that, and I don't think I want to break up the 5.0 improvements into multiple batches.
However, the feedback on the alpha3 has been good, and so you can feel comfortable using it because I'm pretty confident nothing about RF64 file handling will change, and even if it does your files will remain usable. There might be some changes with multichannel CAF files, but that won't affect you.
Thanks again for the feedback!
Feature request:
When dealing with hybrid files, decode as lossy if the correction file appears unreadable.
In foobar2000, I search for matching correction file (possibly stored in a different location) once WavPack library asks for one, not each time before opening a WavPack file. I used to run a slightly modified version of WavPack library to accomplish this, with an "open" file callback added.
I propose adding this change in open_utils.c:
if (wpc->wvc_in && wps->wphdr.block_samples && (wps->wphdr.flags & HYBRID_FLAG)) {
wpc->file2len = wpc->reader->get_length (wpc->wvc_in);
if (wpc->file2len > 0) wpc->wvc_flag = TRUE;
}
Assume no correction file if the correction file appears empty; no changes to callbacks necessary so the library stays compatible with existing frontends.
Thanks for your attention.
Further feature requests.
Allow building of libwavpack static library only via configure && make.
Remove forced iconv dependency (needed for the program itself, not libwavpack).
If it's somehow possible already without editing the scripts, please correct me.
I propose adding this change in open_utils.c:
if (wpc->wvc_in && wps->wphdr.block_samples && (wps->wphdr.flags & HYBRID_FLAG)) {
wpc->file2len = wpc->reader->get_length (wpc->wvc_in);
if (wpc->file2len > 0) wpc->wvc_flag = TRUE;
}
Unfortunately that might break an app that reads the correction stream from a pipe, and
in fact does break my library tester which does just that. This is a little ugly, but might
be a little more robust. Would this work for you?
if (wpc->wvc_in && wps->wphdr.block_samples && (wps->wphdr.flags & HYBRID_FLAG)) {
unsigned char ch;
if (wpc->reader->read_bytes (wpc->wvc_in, &ch, 1) == 1) {
wpc->reader->push_back_byte (wpc->wvc_in, ch);
wpc->file2len = wpc->reader->get_length (wpc->wvc_in);
wpc->wvc_flag = TRUE;
}
}
And thanks for your continued WavPack support in foobar2000.
Tested, works for me, thanks.
Allow building of libwavpack static library only via configure && make.
Remove forced iconv dependency (needed for the program itself, not libwavpack).
If it's somehow possible already without editing the scripts, please correct me.
It doesn't seem to be possible with the existing scripts.
I added a "--disable-apps" option to configure that skips the iconv dependency and builds only libwavpack. You can include "--disable-shared" to build only the static library.
Unfortunately I do not have a system that lacks iconv, so I can't really be sure this fixes the issue, but it should.
Thanks for testing the wvc_flag stuff...I'll commit it.
Currently I'm updating my music collection from Flac to Wavpack (highest x3).
Are there any advantages to use 5.0 instead of the stable 4.8 version like smaller filesize or optimized speed? According to the startpost I think not.
Any comments on that? It will be a few days of recoding so I want to make the right decision.
Well, first of all, 5.0 is still alpha, so it's really not a great idea to use that yet for a huge conversion (especially if you will be deleting your originals in the process).
Also, 5.0 files will actually be slightly larger than 4.80 files because it creates smaller blocks, there are the new block checksums, and there's another 2-byte metadata item just to flag version 5.0 (although I just realized I could use the block checksum for that now). In any event, there's no advantage performance-wise in 5.0.
I think it makes sense to upgrade to 5.0 eventually, but I certainly would not recommend it before the final release unless you want to part of the development process and don't mind a potential problem to come up and waste some of your time. But if you do, be sure to use the most recent alpha (5) from the other thread (https://hydrogenaud.io/index.php/topic,112529.0.html).
Thanks! :)
My question is a bit offtopic here, but I think it can be answered in one single answer so i thought opening a seperate post for it is useless.
Looking at MPEG-DASH/HLS adaptive streaming transports i was wondering if Wavpack Hybrid could be streamed on top of them? Can Wavpack's correction layer be seamlessly turned on/off while playing an existing file? Can i extract segments of a Wavpack file and decode them separately maintaining gapless playback? (I know mp3 can't be split at arbitrary frame boundaries because of the bit reservoir) Also by both formats i have to either package Wavpack segments into MPEG-TS or ISO MP4 which might be a bigger issue than actually implementing the segmenting and reconstructing.
I am not familiar with the MPEG transports, but I am a little familiar with Matroska, and in that case the WavPack correction data is sent as “block additions” as described here (https://www.matroska.org/technical/specs/codecid/wavpack.html).
There is a one-to-one correspondence between WavPack lossy blocks and their matching correction blocks. The lossy blocks can be independently decoded, with or without the correction block (there are no inter-block dependencies).
One issue is that WavPack blocks may have any number of samples, and in the hybrid case these vary block to block (for the dynamic noise shaping). Of course this can also be an advantage in some situations because you could (in theory) add or subtract arbitrary samples in a stream without re-encoding the entire file.
Hope this addresses your questions...
Yes it does. Thank you bryant!
What tool can i use to produce such Matroska files for Hybrid multitrack mode? Can something play it directly? Or is it just a recommendation yet?
There is actually some support for this, but it's spotty and probably not well tested at all. Here's a bug report (https://github.com/mbunkus/mkvtoolnix/issues/713) for mkvmerge showing that combining the wv and wvc files was broken there until quite recently and nobody noticed. And further down it mentioned that the Haali splitter is the only one on Windows that supports this (although Foobar might play them also).
My guess is that this is just not something that very many people care about, and even I have not been keeping up with it.
Just tried it and indeed mkvmerge packs wv+wvc into mka nicely. However not even the latest version of Foobar2000 is able to decode it in lossless mode according to my FFT which shows the added noise of lossy coding clearly.
However this is still an interesting thing because MPEG-DASH can use Matroska/WebM files as segments: http://wiki.webmproject.org/adaptive-streaming/webm-dash-specification
Thanks bryant once again! :)
Just to let you know. I'm currently updating my entire album collection.
Right now I'm converting my 50.000+ files to Wavpack (4.8 stable offcourse). Converting goes quite fast in highx3 mode on my NAS (i5 2400s quadcore). I have tried the highest setting but that resulted in a estimated drop of just a few MB's in total, but conversion would take incredibly long.
Soon we will have some new 128 core multi-cpu VM SSD backed servers running at my work. Maybe I could let them do some converting. Shouldn't take long I guess :)
Hello , I have a quesetion here.
Can we covert Wavpack-DSD directly to other pcm format?
For example I'd like to use qaac to covert Wavpack-DSD to aac .
Is it possible?
Now I'm waiting for a release... :)
Huh? WavPack 5.0.0 Final Release (https://hydrogenaud.io/index.php/topic,113213.0.html)
Can we covert Wavpack-DSD directly to other pcm format?
For example I'd like to use qaac to covert Wavpack-DSD to aac .
Is it possible?
It easily can be done using converter in foobar2000 1.3.13 (or later).
Can we covert Wavpack-DSD directly to other pcm format?
For example I'd like to use qaac to covert Wavpack-DSD to aac .
Is it possible?
It easily can be done using converter in foobar2000 1.3.13 (or later).
Well, I want to use in command-line.
Assuming that qaac will accept 352.8 kHz PCM, then you can probably output directly from wvunpack with the --wav option (to convert to PCM) and -o - (to send it to stdout).
Admins: it would probably make sense to lock this thread too (just like the other "alpha" thread (https://hydrogenaud.io/index.php/topic,112529) that was locked).
I also tried to go over there and delete the attachments at the top and link to the release thread (https://hydrogenaud.io/index.php/topic,113213.0.html), but couldn't. Thanks!