Skip to main content

Topic: WavPack 5.0.0 alpha release available (Read 6296 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 5.0.0 alpha release available
The major enhancement of WavPack 5.0 will be breaking the 2/4 GB .wav limit and additional source file formats. At this point I have added BWF/RF64 (.wav extension), Sony's Wave64 (.w64) and Apple's Core Audio Format (.caf). Of course, all these formats still require PCM audio data to be usable in WavPack. I have also added a big-endian option to the raw PCM mode.

Part of this upgrade includes the capability of decoding and seeking in WavPack files larger than 2/4 GB. To demonstrate that I have updated the winamp plugin and will work on updating the DirectShow filter and Audition plugins in the near future.

These enhancements have required changes to the libwavpack API (which is part of the reason to bump the major revision). However, all of the existing APIs remain so that applications can move to the new library without any immediate need for changes.

Most people who simply use WavPack for their personal music will probably not be interested in the 5.0 changes, and the 4.80 release will be kept current. However, for more advanced use cases these improvements (along with 32-bit audio and unlimited channels) will hopefully keep WavPack relevant.

Any testing results or comments are, as always, greatly appreciated.

edit 1: replaced alpha attachments with alpha2
edit 2: replaced alpha2 attachments with alpha3
edit 3: deleted attachments
edit 4: deleted alphas link, latest release is here
  • Last Edit: 10 December, 2016, 02:03:14 PM by kode54

  • kC_
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #1
Hi David, Many thanks for adding this feature, this will save a huge amount of space in my multitrack audio projects archive!

Just done 1 quick test
firstly its very fast, compressed a 5.5GB stereo wav file in about 80 seconds, and looking at the cpu usage it was only utilizing 14% of my cpu
compressed 5.5gb into 2.7gb
unpacking was even quicker, about 45 seconds, but once it unpacked, it added the wrong file extension (it added .w64)

However the CRC of the original WAV and the unpacked .W64 file matched, so this is no major issue at all.

I have a load more stuff to compress this weekend so I will no doubt do some more testing, many many thanks for this excellent addition!!

Garry

  • Last Edit: 05 May, 2016, 07:13:54 PM by kC_

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #2
Hi Garry, thanks for trying this out and posting your results...very cool!

It's interesting about your .w64 extension though. That really could only happen if your source file actually was a Sony Wave64 file, which are supposed to have a .w64 extension (at least according to the spec and everything I've read online). WavPack detects the file type by looking at the file data rather than using the extension, and stores the "recommended" extension in the file to be used when unpacking, which explains what you're seeing. I guess maybe the best solution would be for wavpack to store the actual extension rather than the recommended one...I'll fix that in the next version.

About processing a lot more files, obviously more testing is better, but please keep in mind that this is still an alpha version and I reserve the right to change something that might make your files unusable on future decoders (for example, any files you do now will always decode to .w64, even after I fix that issue). I don't think it's real likely I'll change something fundamental, but it's possible.

Thanks again!

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #3
I created 6 GB wav file (not w64) with foobar2000. It has riffsize=datasize=0xFFFFFFFF.

Then I encoded it with -i option: wavpack.exe -i -m 6GB.wav And wavpack writes to the console during encoding: "creating 6GB.wv, 152% done..." and then "created 6GB.wv in 332.14 secs (lossless, -12.40%)" (the size of the resulting .wv file is ~4.5GB)

After the decoding (wvunpack.exe 6GB.wv) the decoded file has .wav extension, riff size = 0x900647b4 and data size = 0x90064790. Probably wvunpack should write 0 or -1 into these fields in this case?
Or maybe it should default to .w64 output format in this case?

P.S. foobar2000 can decode but cannot seek if the WV size is bigger than 4 GB. No problems with the bundled Winamp plugin.
  • Last Edit: 07 May, 2016, 06:56:22 AM by lvqcl

  • nu774
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #4
After the decoding (wvunpack.exe 6GB.wv) the decoded file has .wav extension, riff size = 0x900647b4 and data size = 0x90064790. Probably wvunpack should write 0 or -1 into these fields in this case?
Or maybe it should default to .w64 output format in this case?

Haven't yet tried it myself, but automatic switching to RF64 on large file looks like already implemented in the source (21 days ago).
https://github.com/dbry/WavPack/blob/five/cli/riff.c#L308
If it's not working as intended, maybe there's a bug or something?

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #5
I downloaded sources form github, and this version doesn't write progress message with >100% value.

But WriteRiffHeader() is not called here. Looks like wvunpack calls DoWriteFile(outfile, WavpackGetWrapperData (wpc), WavpackGetWrapperBytes (wpc), &bcount) that writes audio header.

  • nu774
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #6
Quote
But WriteRiffHeader() is not called here. Looks like wvunpack calls DoWriteFile(outfile, WavpackGetWrapperData (wpc), WavpackGetWrapperBytes (wpc), &bcount) that writes audio header.
Oh, I see. wvunpack normally just restores the stored wrapper (to behave lossless including the container), so it cannot create new header or switch to RF64 or something unless explicitly told to.
Better placeholder length should be fine.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #7
I downloaded sources form github, and this version doesn't write progress message with >100% value.
This is a little weird because I have not made any changes in the repository since the alpha release. Could this be a compiler difference (mingw?), or perhaps you didn't really use the same source file or command-line arguments?

As for the other stuff, I have duplicated the issues, and also discovered that the funny progress percentage and incorrect compression ratio report have been around for a while. I suspect that I wanted the -i option to not do weird things if the wav headers actually were correct (or almost correct) and so I assume they are correct and behave accordingly. I don't really like the behavior, and I'll look into it some (and it does really bad stuff if, for example, the header indicates a 1 sample file).

As for the writing bad values into the riff size and data size, that problem has also been around a while. As nu774 says, I don't really want to write a completely different (and possibly incompatible) type of header unless the user specifically asks for it (via -r to wavpack or -w to wvunpack). However, I did decide at some point that I would attempt to fix a bad header if -i was specified, but of course in this case the header can't really be fixed and I've actually made it worse. I'll definitely change this to simply leave the header alone (lossless) unless I can actually fix it correctly.

Thanks for the testing and sharing your results!

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #8
This is a little weird because I have not made any changes in the repository since the alpha release. Could this be a compiler difference (mingw?), or perhaps you didn't really use the same source file or command-line arguments?
I recompiled wavpack.exe from the sources that I downloaded previously, and I can confirm that they also show > 100% progress. But this also means that I don't have my prevoius compiles, so I can't re-test them...

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #9
Thanks so much for all the testing and suggestions; it's greatly appreciated!

I have fixed all the issues reported here and on GitHub and am releasing alpha2 (downloads replaced at top of thread).

  • store (and restore) the actual file extension instead of the file type default
  • explicitly store the source file type instead of relying on the header, because we might not store the header (e.g., wavpack -r)
  • actually ignore length specified in header when we're told to, but try to determine length from file size or, if we can't (pipes), don't display progress or compression percentage
  • never try to fix a header that we didn't make, even if ignoring length (we are supposed to be lossless)
  • handle CAF channel layouts that are not in MS order and make the channel layout and reordering information available to applications (but applications that don't know about this will get MS order)
  • handle CAF channel description arrays and map them to MS order as far as they'll go (no re-ordering and a decoding application will have to manually parse the description array to access it)
  • when unpacking, always use the correct extension and data format (endian) for the output format
  • add display of the source file format and extension to the wvunpack -ss command
  • make sure transcoding copies all the new features across
  • several minor fixes

Files created with the previous alpha release should continue to be usable, unless they are subject to one of the above issues. I changed the way I stored some information in the WavPack file, but will continue to honor the original method.

Cheers!

  • kC_
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #10
Hi David, Many thanks for the update, yep I can confirm all is perfect now, 5gb .wav before = .wav after wavpack & decompress... & the hash's match, amazing stuff! thanks so much

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #11
Hi David, Many thanks for the update, yep I can confirm all is perfect now, 5gb .wav before = .wav after wavpack & decompress... & the hash's match, amazing stuff! thanks so much
Cool...thanks for testing and letting me know!

BTW, I am considering dropping support for WavPack files created prior to version 4.0, which was introduced in 2004. Files from versions before that (i.e., 3.97) are not compatible with Rockbox or DirectShow or Gstreamer, are not quickly seekable, and can be completely destroyed with a single bit error. I have therefore long recommended that those files be transcoded to the newer version, and doing this is especially easy now that the wavpack command-line program can do it automatically.

However, I originally said I would never drop support for older files (never say never!) and so I consider this with some trepidation even though the sources and binaries for 4.80 will be available as long as WavPack is available. It seems to me that Matt did a similar thing with Monkey's Audio and got grief. Does anyone reading this have a collection of pre-4.0 files, or have a strong opinion?

Of course there's an appropriate XKCD comic...be sure to read the mouseover!

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #12
BTW, the definition of CUR_STREAM_VERS differs between include\wavpack.h and src\wavpack_local.h

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #13
BTW, the definition of CUR_STREAM_VERS differs between include\wavpack.h and src\wavpack_local.h
Thanks...I'll fix that!  :)

I may just delete it from wavpack.h because only the MIN and MAX should be used by anyone outside the library.

  • Case
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #14
I am considering dropping support for WavPack files created prior to version 4.0
I don't have any such files but in principle I'm against reducing compatibility. Dropping support for old files forces any self-respecting player to use two versions of the decoder library.

  • Funk
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #15
Hello, I use Wavpack to reduce the size of files recorded from a software called SDRSharp.
It is a radio receiving software capable of saving wav files containing the baseband of several megahertz.
Wavpack works very well with files smaller than 16 GB, but if i try to compress files bigger than that, Wavpack says "xxx.wav has too many samples for Wavpack!"
Is this to be expected or is it a bug?
The files are of the RF64 format.
  • Last Edit: 27 May, 2016, 05:29:45 PM by Funk

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: WavPack 5.0.0 alpha release available
Reply #16
Is this to be expected or is it a bug?
To be expected.

Max. number of samples is 4 294 967 294, so for 16-bit stereo or 32-bit mono signal  max. filesize is 16GB.
For samplerate=44100, max. duration is 27 hours, for 192kHz it's more than 6 hours. But if you have several megaherz...
  • Last Edit: 27 May, 2016, 06:35:36 PM by lvqcl

  • Funk
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #17
I thought the raw option would ignore the WAV or W64 restrictions. If I use the raw option it crops my file. This is the output i get:
wavpack.exe --raw-pcm=100000,16,2 -hh -x1 --pre-quantiz
e=12 *.wav

 WAVPACK  Hybrid Lossless Audio Compressor  Win64 Version 5.0.0-alpha2
 Copyright (c) 1998 - 2016 David Bryant.  All Rights Reserved.

 *** This is an experimental alpha version and should be used for ***
 *** testing only. Please do not use it for important archiving!! ***

creating 18-12-35_91700kHz.wv, 1111111111111111111111111111111111111036% done...
11111111111111111111111111111111111111111111111111111111111111111111104% done...

11111111111111111111111111111111111111111111111111111111111111111111172% done...
11111111111111111111111111111111111111111111111111111111111111111111004% done...
11111111111111111111111111111111111111111111111111111111111111111111072% done...
11111111111111111111111111111111111111111111111111111111111111111111140% done...
11111111111111111111111111111111111111111111111111111111111111111111208% done...
created 18-12-35_91700kHz.wv in 9450.87 secs (lossless, -1049.57%)

And the resulting file is useless.
I can pack the original wav file with OptimFROG and unpack with raw option without problems. But OptimFROG is VERY slow.
  • Last Edit: 28 May, 2016, 04:24:15 PM by Funk

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #18
I thought the raw option would ignore the WAV or W64 restrictions. If I use the raw option it crops my file. This is the output i get:
wavpack.exe --raw-pcm=100000,16,2 -hh -x1 --pre-quantiz
e=12 *.wav
I'm not sure what you mean by "the resulting file is useless". If you use --pre-quantize with --raw-pcm then your header is going to pre-quantized, which it's not going to like. Probably your unpacked file will not be recognized after that, even though the only problem is some corrupt bytes in the header.

Anyway, --raw-pcm is not going to get around the 2^32 sample problem, but using "-i" to ignore length in header might (I have to try it), but that would be for archiving only (you could only restore the entire original WAV/RF64 file). Would that work for you? What do you do with these files after you compress them?

I considered at one point extending the 2^32 samples limit at one point (there's spare room in the header to do 1 Tera-sample) but I couldn't think of a reasonable use case that would require it – obviously I was not considering megahertz sampling rates! I'll take a look again (although I think it's a little bit of a mess).

BTW, another limit I just noticed is that I limit the sampling frequency to around 16.7 MHz. Unlike the other issue, that one's easy to fix...



  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #19
I am considering dropping support for WavPack files created prior to version 4.0
I don't have any such files but in principle I'm against reducing compatibility. Dropping support for old files forces any self-respecting player to use two versions of the decoder library.
Ah, thanks Case, it never occurred to me that a player developer would do that. I just assumed, perhaps naively or paternally, that if I dropped support for older files in the decode library (and all my tools and plugins), then developers would follow my lead (and perhaps be relieved) rather than go to the trouble of using dual libraries. I definitely would not want them to have to do that.

Another option would be for me to drop support from my tools and plugins, but leave the support in the decoder library and have it disabled by default. That way I would create almost no extra work for developers who wanted to retain the support, but would give me a chance to make my point, and maybe find out how many of these file are actually in the wild.

  • Funk
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #20
I thought the raw option would ignore the WAV or W64 restrictions. If I use the raw option it crops my file. This is the output i get:
wavpack.exe --raw-pcm=100000,16,2 -hh -x1 --pre-quantiz
e=12 *.wav
I'm not sure what you mean by "the resulting file is useless". If you use --pre-quantize with --raw-pcm then your header is going to pre-quantized, which it's not going to like. Probably your unpacked file will not be recognized after that, even though the only problem is some corrupt bytes in the header.

Anyway, --raw-pcm is not going to get around the 2^32 sample problem, but using "-i" to ignore length in header might (I have to try it), but that would be for archiving only (you could only restore the entire original WAV/RF64 file). Would that work for you? What do you do with these files after you compress them?

I considered at one point extending the 2^32 samples limit at one point (there's spare room in the header to do 1 Tera-sample) but I couldn't think of a reasonable use case that would require it – obviously I was not considering megahertz sampling rates! I'll take a look again (although I think it's a little bit of a mess).

BTW, another limit I just noticed is that I limit the sampling frequency to around 16.7 MHz. Unlike the other issue, that one's easy to fix...




Sorry for the late reply.

The resulting file is useless because it is cropped to about 16 GB but the original file was 34 GB.
Anyway this is not a problem anymore as I have set SDRSharp to record max 16 GB files and then split to a new file if I record that long.

And to explain what I use those files for:
It is a hobby of mine to listen for distant radio stations that are usually impossible to hear. Described here: https://en.wikipedia.org/wiki/TV_and_FM_DX
I use a software called SDRSharp with the software defined radio hardware "Airspy" that can sample 10 MHz of radio spectrum in 12 bits resolution and save it in 16 bit WAV files (or RF64 with a plugin I use).
Then afterwards I have like a "time machine" so I can tune around in that piece of radio spectrum I have recorded.
The recordings take up a huge amount of space but I want to have an archive of those unusual receptions where there is much going on at different frequencies at the same time. WavPack saves about 55 to 80 percent of disk space with these files.

Thank you for your answers and your excellent software!
  • Last Edit: 02 June, 2016, 05:16:56 PM by Funk

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #21
Thanks for the description! So, I imagine that you must unpack the entire archive back to RF64 before your "player" can use it, right? It seems like it would be possible to record directly to and play directly from WavPack (and save a lot of disk trashing), but that would obviously require a new plugin.

In any event, I'm working on eliminating the 4 Giga-sample limitation (I'm sure it will come up again), so hopefully that will make this even a little easier.

Cheers!

  • Funk
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #22
Yes, I have to unpack the files to play them in SDRSharp.
When I record at 10 MHz bitrate I use a dedicated SSD because my mechanical drives are too slow to avoid dropped data.
I think the slow hard drive problem would be gone if it was possible to record directly to WavPack, but then maybe the CPU would be the bottleneck fore some computers. Not for mine I think.

I have sent a comment to the developer of the recorder and player plugins.
Link to my comment (page is in Russian, there is a Google Translate tool on the page):
http://rtl-sdr.ru/page/obnovlen-file-player-1#comment-2148

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha release available
Reply #23
I have again fixed all the issues reported on GitHub and here (including "...too many samples...") and am releasing alpha3 (downloads replaced at top of thread).

I also updated the DirectShow filters to handle all the new large files, and am including those installers in the downloads in addition to a new winamp plugin.

  • increased maximum sample count from 4 giga-samples to almost 1 tera-sample (2^40)
  • improved handling of CAF channel description arrays so that all MS channels are reordered to the front (and bitmasked)
  • add --raw-pcm-skip option (wavpack) to ignore specified byte count (headers?)
  • allow minus --skip values (wvunpack) to specify start point relative to EOF
  • add optional parameter to wvunpack -f to display specific item only
  • support sampling rates over 16777215 Hz (probably non-audio)
  • several minor fixes

Thanks so much for all the testing and suggestions; it's greatly appreciated!

Cheers!

  • Funk
  • [*]
Re: WavPack 5.0.0 alpha release available
Reply #24
Alpha 3 is working great for my needs. Many thanks for making it accept larger files!
Now I'm waiting for a release...  :)