Skip to main content

Topic: WavPack 5.0.0 alpha4 release: DSD audio support added (Read 7690 times) previous topic - next topic

0 Members and 2 Guests are viewing this topic.
  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 5.0.0 alpha4 release: DSD audio support added
  • DSD support with two new formats: Philips DSDIFF (.dff) and Sony's DSF (.dsf)
  • two DSD modes: lossless compression and “high” lossless compression
  • export WavPack DSD to either DSDIFF or DSF (format conversion)
  • export WavPack DSD to 24-bit PCM in any format (8X decimation)
  • Cool Edit / Audition (1.0–3.0) filter that reads DSD as 24-bit PCM (8X decimation)
  • winamp plugin that plays standard rate DSD files at 88.2 kHz (32X decimation)
  • added --disable-apps to configure script (removes iconv dependency)
  • add optional trailer specification to --raw-pcm-skip
  • increase --raw-pcm sampling rate max to 1 GHz
  • stability enhancements

That last major feature for WavPack 5.0 is DSD support, and I have added both Philips' .dff and Sony's .dsf to the list of formats. I realize that most HA users will not be too interested in this, but it seems to be popular in the audiophile community and I thought that it would nicely compliment the other features in WavPack. Also, a few years back I did some spec work on DSD compression algorithms, but it didn't work out and I wanted to do something useful with that work. For users with both PCM and DSD music in their collection, WavPack now offers a “one format” solution, which might appeal to some.

The compression is only lossless (no lossy or hybrid) and there are just two modes, the default and a “high” mode. The default mode uses negligible CPU and achieves about 35% to 50% compression for most files. The “high” mode uses about five times the CPU as the default mode and gets about 10% to 15% better compression. The “high” mode is still easily playable on any modern PC, but might present a challenge for some dedicated devices to decode in real time – it uses about 50% of one core on a Raspberry Pi 2.

I think that the “high” mode compares favorably to the Philips DST compressor. Its ratio is about the same as the reference encoder (the one with source floating around), although I have seen that the commercial DST compressor performs somewhat better than the reference. However, the CPU usage for the DST format seems to be much higher, both for encode and decode, on all the implementations I've measured.

The WavPack library can decode to the original DSD, or can optionally decimate 8X to 24-bit PCM as a convenience for apps not set up for DSD. I am including a copy of the Cool Edit / Audition filter that reads DSD files as PCM in this way. I also created a version of the winamp plugin that further downsamples another 4X to play standard rate DSD at 88.2 kHz.

Thanks again for all the bug reports and suggestions. With continued good luck, I hope to have a final release of WavPack 5.0 by the end of September.

edit: replaced alpha4 with alpha4 attachments (see post below for latest changes)
edit2: note that the Cooledit/Audition filter provided produces corrupt files...do not use!
edit3: deleted attachments, latest release is here
  • Last Edit: 10 December, 2016, 02:04:42 PM by kode54

  • Wombat
  • [*][*][*][*][*]
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #1
Very interesting to see how wavpack extends its features. These compression ratios should make storing these dsd64 files roughly as big as 24/88.2 PCM. I guess since software like the Logitech Mediaserver already supports dsd and PCM wavpack this can be a very welcome extension!
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #2

Very interesting to see how wavpack extends its features.
IMO, one can infer such versatility (hybrid encoding, apparently-complete DSD handling, etc.) has placed Wavpack into a unique position of having not only compression ratio as the only card up its sleeve, as far as lossless compression usually goes; but of becoming a thoroughgoing lossless solution by and by.

A feat that is even more commendable when you remind yourself it is (stiil?) a one man's job!

Listen to the music, not the media.

Best,
Nilson

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #3
DSD encoding is very interesting (for me) and really unexpected. Excellent work David!
Comparison between DST (encoding speed, decoding speed, ratio) could be really interesting.
I just tried with two albums:

https://channelclassics.nativedsd.com/albums/22105-bolivian-baroque (length=74'06")

— DFF stereo: 3 120 257 Kb
— WavPack default: 1 723 555 (59 sec encoding time)
— WavPack high: 1 412 167 (377 sec encoding time)
— DST (SACD): 1 218 081

Second disc, more compressible (solo piano, DSD recorded, 49'30'')
— DFF stereo: 2 053 667 Kb
— WavPack default: 1 147 428 Kb (enc. time: 38")
— WavPack high: 919 839 Kb (enc. time: 225")
— DST (SACD): 740 487 Kb

I can't do more for now. At first sight WavPack is less efficient as DST (but it's only two discs, results may be different with others).
I also note one issue with CoolEdit playback: DSD *.wv files are really, really noisy when played. It's surely a consequence of the onboard chip (basic Realtek) which doesn't handle 352/32: once dithered to 44,1/16 it sounds fine.

I hope that foobar2000 library will quickly add the new features (in order to use the utilities: speed bench, bit-to-bit comparison…).


EDIT: at the moment I can't use foobar2000's converter to get DSD/wv files. When I use it, I get very large *.wv file, and they're apparently PCM. I tried to play with foo_input_SACD options (DSD output mode, Edited Master Playback…) without success.
  • Last Edit: 28 August, 2016, 04:44:59 PM by guruboolez

  • kode54
  • [*][*][*][*][*]
  • Administrator
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #4
A thing to note about DST compression is that when you're using anything like foo_input_sacd that does DST decompression using the reference decoder, you're using 4-8 threads at near full utilization for real time playback. Whereas the WavPack decoding is likely going to use 10-50% of a single core.

Will there be any tools for doing things like merging DSD audio into single files? I already experience issues with gaplessness that can only be caused by the decimator not being able to start with the history of the previous track on each track transition. This could also be corrected on the player side by handling raw DSD directly.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #5
A thing to note about DST compression is that when you're using anything like foo_input_sacd that does DST decompression using the reference decoder, you're using 4-8 threads at near full utilization for real time playback. Whereas the WavPack decoding is likely going to use 10-50% of a single core.
I tried. DST playback with foobar2000 => ~100% CPU/8 cores (4 physical) during few seconds, and then ~6% on all cores. DSD-WV doesn't work at the moment with foobar2000 1.3.11. so I can't compare directly. But using wvunpack.exe x64, decoding takes a while and cores (all 8 apparently) don't go beyond 25%. Decoding is slow using the CLI (×44 with default mode, ×12 with high mode). DST speed is also slow and reaches ×18 but using 100% of eight cores (which should be no more than ×4 if it uses the same amount of CPU as wvunpack).

I did the test with: Core i7-3610QM @2.30GHz
Harpsichord recording:
— DSD = 3 086 227 Kb
— DST = 1 161 184 Kb (=37.63% of original source)
— WVn = 1 680 841 Kb (=53,82% of original source)
— WVh = 1 425 140 Kb (=45,54% of original source)

First impressions:
— DST is a significantly most efficient tool for compressing DSD than Wavpack
— WavPack should be much more hardware friendly (decoding speed): current portable players or even standalone one which supports DSD and even ISO files can not decode DST, which makes DSD format a really huge one for µSD cards)
— WavPack is maybe easier to handle (tagging DST/DFF files seems to be possible with ID3v2 but is widely accepted?)
— It's a lot of trouble for a format (DSD) which doesn't bring significant advantage (to say the least) over PCM. But for those who like, it might become popular over DSF, DFF, or ISO (or even DOP-FLAC which I heard from but never see it)
  • Last Edit: 28 August, 2016, 06:00:29 PM by guruboolez

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #6
I also note one issue with CoolEdit playback: DSD *.wv files are really, really noisy when played. It's surely a consequence of the onboard chip (basic Realtek) which doesn't handle 352/32: once dithered to 44,1/16 it sounds fine.
Yes, I should have mentioned that the 8x decimation still leaves a lot of noise between about 50 kHz and 110 kHz, and if your driver/sound-card does not properly downsample that, you'll hear it. I also get noise here and I think in my case some process is simply throwing away samples instead of filtering, which of course just moves that noise into the audible range instead of eliminating it!

EDIT: at the moment I can't use foobar2000's converter to get DSD/wv files. When I use it, I get very large *.wv file, and they're apparently PCM. I tried to play with foo_input_SACD options (DSD output mode, Edited Master Playback…) without success.
I haven't tried that yet. Assuming an DSD source file, I would think that the converter should be able to output something like a .dff or .dsf file, but since nothing else up to now would accept such a file as input, probably not.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #7
First impressions:
— DST is a significantly most efficient tool for compressing DSD than Wavpack
— WavPack should be much more hardware friendly (decoding speed): current portable players or even standalone one which supports DSD and even ISO files can not decode DST, which makes DSD format a really huge one for µSD cards)
— WavPack is maybe easier to handle (tagging DST/DFF files seems to be possible with ID3v2 but is widely accepted?)
— It's a lot of trouble for a format (DSD) which doesn't bring significant advantage (to say the least) over PCM. But for those who like, it might become popular over DSF, DFF, or ISO (or even DOP-FLAC which I heard from but never see it)
Thanks for your exhaustive testing and sharing your impressions, guru, it's greatly appreciated!

The goal was to come up with an alternative to DST compression, because there are a couple of problems with DST, as far as I see it. The most obvious problem is that it is not readily available, probably because Philips intended it to be just for SACDs – that people might want to have DSD files in their music library and play on portables does not interest them. There's a reference encoder/decoder with source, but it's extremely slow (1/5X realtime on my PC) and the encoder doesn't compress nearly as well as the format is capable of. You can buy the Philips ProTECH encoder for many thousands of dollars, but that's obviously not a solution for most people.

The other problem is that the DST format, while offering very good compression (perhaps approaching entropy), is very CPU intensive, which again is not a problem for Philips because they probably sell/license hardware decoders for SACD. People have optimized the reference decoder in attempts to make it useful in real applications, but it's still not fast enough for many portable players. The best performance I have seen is the decoder in FFmpeg (see below).

My thought was that a freely-available DSD compressor with reduced CPU requirements (compared to DST) would be useful, even if it provided more modest compression. Also, as you point out, there are two major DSD transport formats, and one offers compression but no tagging (Philips) and the other offers tagging but no compression (Sony), so that's the justification for adding DSD compression to WavPack.

Using the Philips ProTECH Audio Converter, I created 10 difference DSD encodings of a single PCM sample using 10 difference SDM settings, and then compressed all 10 using WavPack's two modes and the reference and pro versions of the DST compressors. I measured the decode speed (to 8x decimated PCM) using x86 versions of FFmpeg and WavPack, both running under Wine and using a single thread:

                       total bytes  compr%   compr range  decode
                     -------------------------------------------
Original DFF data:   1,591,929,920    0.00 
WavPack default:       843,395,464   47.02   41.97–52.38    34x
WavPack high:          708,570,418   55.49   50.22-60.04    10x
DST reference:         694,851,546   56.35   50.01-61.69   5.5x
DST ProTECH:           606,176,506   61.92   54.84-68.20   5.6x

On these files the difference between WavPack "high" and DST pro encoder on the same sample varied from 3.48% and 8.85%, so that pretty much matches your results.

And, based on this quick test, WavPack “high” is almost twice as easy to decode as DST, and the default mode is six times easier, which would certainly make them both more likely candidates for portable decoding than DST.

  • Archimago
  • [*]
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #8
Just wanted to say congrats Bryant for coming up with a freely available solution for DSD compression and tagging!

I've been talking about this since 2013 and glad to see someone has taken up the "cause" :-).

I think the folks who sell these DSD downloads should seriously support this if they're serious about DSD.




  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #9
Thanks David for your reply.
I'm not familiar with DST tools. I know they're slow but I always thought the main cause was a true lack of optimization. Apparently not… DST compression seems to be really and very strong.
WavPack should be much more friendly. I'm really interested in building my own DSD library but for the moment it's a lot of troubles.

ISO format: DST compression already done but includes usually both stereo and MCH. Moreover, I can fully tag them with foobar2000: it creates a conveniant side car. But the latter is not compatible with Jriver Media Center which only uses an internal database for metadata and it's a lot of troubles if you simply move your files in another directory.
DSD files can easily be tagged and I can only keep the stereo files. But size is huge (almost the same than *.iso which contains the MCH mastering…)
DST files which seems to be painful for the CPU (and makes transcoding really less comfortable).
— WavPack should be much faster for batch encoding, is better for transcoding, tagging… It's easier to keep the stereo files. Any user can therefore significantly reduce the total size of its DSD library. And the format has more compatibility.

But at the moment it's a bit theoretical. What is needed is a good and friendly batch tool (foobar2000 should be perfect but doesn't seem to keep DSD bitstream while converting). But I'm sure it will come :)

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #10
I want to give a heads-up that I will definitely be changing the DSD compression format before the next release, so any compressed DSD files created now will become obsolete and unplayable.

I know the program displays a warning about the alpha status every time it's run, but I just wanted to make sure everyone understands that this is just for testing / experimentation, although regular PCM files will be fine. Thanks!

Just wanted to say congrats Bryant for coming up with a freely available solution for DSD compression and tagging!
I've been talking about this since 2013 and glad to see someone has taken up the "cause" :-).
I think the folks who sell these DSD downloads should seriously support this if they're serious about DSD.
I actually had read that article in the past...very nice!

I hope you're right about the distributors supporting this, although I'm sure their biggest worry will be playback support. It's a little bit of a chicken and egg problem.

I'm not familiar with DST tools. I know they're slow but I always thought the main cause was a true lack of optimization. Apparently not… DST compression seems to be really and very strong.
The reference encoder/decoder is definitely slow because it's not optimized – it's written to be understandable. However, it's soooo bad that I think it might even be written to be intentionally slow because, again, Philips has no interest in people using this. You're supposed to buy SACD discs and play them on your Philips SACD player.

The ProTECH tools probably are well optimized, but how close they (or FFmpeg) are to “fully” optimized is anyone's guess. I'm also not sure how more optimized my format can become, but I am reorganizing it so that it's more SIMD-friendly in case I want to tackle that in the future.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #11
Thanks for warning us about the future change. Do you plan to work further on the DSD compression engine?

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #12
Thanks for warning us about the future change. Do you plan to work further on the DSD compression engine?
Yes, I am refactoring the "high" mode so that it will possible to encode and decode stereo streams using SIMD instructions like I do in the PCM assembly optimizations.

I am not actually going to do that optimization so there will not be any significant change is performance now, but it will be possible in the future.

The default mode will probably not change.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #13
I have released the fifth and [hopefully] final alpha of WavPack 5.0.0 (see top of thread for attachments). This release includes the last of the major changes I wanted to incorporate and I am pretty confident that the file format and API will be stable going forward and that any files created now will be usable in the future (notwithstanding bugs, of course).

As I mentioned, "high" mode DSD files created with alpha4 will no longer decode. If you have some and want to keep them, the easiest thing would be to use alpha4 to transcode them to the default mode (which is still usable) and then use alpha5 to transcode them back to "high". All tags will be retained.

  • new "high" mode DSD compression (about 15% faster and SIMD-compatible)
  • store identities of non-Microsoft channels (completes CAF support)
  • added block checksums for more robust handling of corrupt files
  • improved seeking performance (through the use of shorter blocks)
  • fixed problems with formats that require data chunk alignment
  • removed default support for legacy Wavpack files (< 4.0)
  • allow --raw-pcm option to handle DSD (set bps=1)
  • add ability to scan DSD files to wvgain
  • other minor fixes and enhancements

Thanks again to everyone in the HA community for your continued support!!

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #14
In my testing I have discovered that the Cooledit/Audition filter provided above generates corrupt output files, so it's not recommended that it be used. The resulting files are not really corrupt (old versions of WavPack fille be fine with them), but they will have a bad block checksum at the beginning which will cause problems for the newest decoders.

If you have any of these files the easiest thing to do would be to transcode them using the released version of the command-line tool (4.80); you won't lose anything. This will all be fixed in the upcoming Release Candidate.

Thanks for your patience!

  • conta69
  • [*]
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #15
Hi there,
i tested the latest WavPack Linux Package with MPD --> it seems there are some problems. I started a thread:
MPD thread
The developer there tells, the code does not work proper, but it's also possible that it is necessary to readjust the MPD-interface.

Maybe you are interested in looking into this?

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #16
This actually makes sense.

FFmpeg will not play DSD files until all of the DSD stuff has been incorporated into FFmpeg. It should have been able to detect the error earlier, but it's good that it didn't try to decode anything or crash.

As for using libwavpack directly (it looks like MDP tries both) it's required for the application to set the OPEN_DSD_AS_PCM in the call to open the file to allow libwavpack to open a DSD file as PCM (which I assume MDP is going to require).

Fortunately, older versions of the library (<= 4.80.0) ignore this flag, so it can just be set always, assuming the program is prepared to get very high PCM sampling rates (352.8 kHz for DSD64).

You can also patch the library to make this the default. Just insert this at line 42 in src/open_utils.c:

Code: [Select]
if (!(flags & OPEN_DSD_NATIVE)) flags |= OPEN_DSD_AS_PCM;

BTW, the official 5.0.0 was released yesterday, so you can start with that. (I'll start another thread today).

Thanks for bringing this up and I would be curious if this patch fixes the problem (at least with MDP).

  • conta69
  • [*]
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #17
Thanks for the quick answer!

Do i understand you correct, that FFmpeg is needed to playback DSD-WavPack in Linux?
DSD and even DST is listed on the FFmpeg-site as supported audio codecs. Do you mean, that FFmpeg is not able to decode DSD-WavPack?

Thanks for the library patch, but i think thats the wrong way. MPD is able to playback DSF-files in DSD or DoP directly. No need for transfering DSD to PCM. To the contrary - it is desired to leave DSD as it is!  :)

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: WavPack 5.0.0 alpha4 release: DSD audio support added
Reply #18
Oh, I'm sorry! I did not realize that MDP now has native support for DSD, but I just took a longer look at their changelog and see that it does.

In that case, you're right that having libwavpack decode to PCM is not the ideal way to go, but it would certainly be usable as a start (assuming, again, that MDP is going to do something reasonable with 352.8 kHz PCM).

As for natively handling DSD in WavPack there are 2 options, just like with PCM in WavPack.

The first is for MDP to use libwavpack 5.0.0. All they need to do is set the OPEN_DSD_NATIVE flag in the call to open and WavPack DSD files will open. They'll have to do a little work to detect and handle the DSD data through the libwavpack API, but the data will basically be there.

I created a specific document for application developers moving to Wavpack 5 here that they could use.

The other option would be for FFmpeg to incorporate the new DSD decoder from WavPack. They generally write their own decoders from scratch, although in this case they could borrow heavily from the code in libwavpack, or even use libwavpack directly like they do for encoding. That would probably be a fair amount of work, so unless they've already started I would not expect that really soon.