Skip to main content

Topic: How to compress DSD with WavPack? (moved from General Audio) (Read 5467 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • conta69
  • [*]
How to compress DSD with WavPack? (moved from General Audio)
Hello there,
hope, that i am here in the proper subforum. It's my first thread...  O:)

Shortly i realized, that it's possible to compress DSD with the new WavPack 5.0 (Alpha5) and that it's already implemented in Foobar2000. So i downloaded the latest WavPack Alpha5 and tested it with several dsf-files and also direct from sacd-ISOs. The results looked very good - fine compression-rate and not very CPU-demanding.

BUT there seems to be something wrong with the conversation. When i invert the process with wvunpack.exe i don't get back a DSD-file in any form, but a PCM-file.
When i instead convert a dsf-file with the original wavpack.exe and uncompress it with wvunpack.exe, i get the original dsf-file back.

I tried to find an appropriate setting in the Foobar-convert dialog, but i am not able to find any.
So what i am missing? Is there something important to consider? As far as i can create wv-files from DSD in Foobar now, it seems, that they get converted to PCM at some point, what is - of course - not wanted.

  • Fairy
  • [*][*][*]
Re: How to compress DSD with WavPack?
Reply #1
Do you encode via foobar? Possibly foobar converts the dsd stream to PCM.

  • conta69
  • [*]
Re: How to compress DSD with WavPack?
Reply #2
Yes, i encode with Foobar, but i don't tell Foobar to convert DSD to PCM.  :)
At least i hope so...
I choose the dsf-file in the list, right-click it, then open the 'convert'-dialog and convert the file directly to wv.

Edit:
No DSPs activated.
  • Last Edit: 01 December, 2016, 06:28:14 AM by conta69

  • mpuzirew
  • [*]
Re: How to compress DSD with WavPack?
Reply #3
Yes, i encode with Foobar, but i don't tell Foobar to convert DSD to PCM.  :)
At least i hope so...
Foobar2000 (as any other player) always converts audio to PCM while playing or converting,  it can not be otherwise :) Least because this "format" is only that is "understood" by soundcard.
P.S. Maybe, you could use smth like "Run services" plugin instead of convertion function,  to avoid DSD -> PCM. I don't know exactly.  
  • Last Edit: 01 December, 2016, 07:49:23 AM by mpuzirew

  • conta69
  • [*]
Re: How to compress DSD with WavPack?
Reply #4
Foobar2000 (as any other player) always converts audio to PCM while playing or converting,  it can not be otherwise :) Least because this "format" is only that is "understood" by soundcard.

Hmmm...
I'm not really sure about that...   ::)
As far as i know, in the meantime it is possible to playback in DSD when you have a DSD-DACs.
And for the compression to DSD-WavPack - when the contet is converted to PCM i could use flac also.

  • mpuzirew
  • [*]
Re: How to compress DSD with WavPack?
Reply #5
When you have a DSD-DACs it is quite possible to playback in DSD, I think. Because that is a DSD-DAC :)
  • Last Edit: 01 December, 2016, 07:58:50 AM by mpuzirew

  • conta69
  • [*]
Re: How to compress DSD with WavPack?
Reply #6
Uhhh....
I've just seen, that here is a spezific Foobar-forum too. Maybe a Mod could move the thread to this forum?
Sorry, about the wrong entry. First time...   :-[

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack?
Reply #7
Thanks for trying out WavPack DSD!

What Fairy says is correct. Foobar (last time I checked) does not handle DSD internally. I believe the DSD is converted to PCM when the file is read, and so if you convert to any other format you're going to get PCM. You can always use wvunpack.exe -ss to find out what format the WavPack files are.

At this point you'll have to convert the DSF files using the command-line program to WavPack files, which Foobar can play (again, by converting immediately into PCM). That will also give you a real idea of the compression ratio. I don't know enough about sacd-ISO files yet to know how to handle them.

It would be possible for Foobar to have a “DSD path” internally and do the conversion to PCM later in the chain. That would allow it to convert to DSD formats and also allow it to play through DACs that handled DSD natively, but I don't know where that stands in priority.


  • conta69
  • [*]
Re: How to compress DSD with WavPack?
Reply #8
Thanks a lot for your enlightening answer!
And thanks a lot for your effort about the DSD-format. Your WavPack seems to be the first and only possibility to compress DSD to a 'standard' container-format, which can be tagged by nearly all tagging-progs.   ;D

But it seems, the playback of DSD-WavPack is not yet really suitable. When the DSD-data internally gets converted to PCM, it would be the same to convert the dsf-files to flac directly.
Hopefully this defict gets removed soon. Especially the converting SACD-ISOs to WavPack would be a pleasant thing. It's actually already working (with sacd-decoder plugin), but as you say, with the internal PCM-conversation.

To provide some concrete informations, here is the output of 'wvunpack.exe -ss' from a converted dsf-file:

Code: [Select]
file name:            xyz.wv
file size:               36457290 bytes
source:                24-bit ints at 88200 Hz
channels:             2 (stereo)
duration:             0:01:50.43
modalities:           lossless
compression:       37.61%
ave bitrate:          2641 kbps
encoder version:  5
original md5:       14rz138rz701837rz10837rz2108374erz134
source format:     Microsoft RIFF with 'wav' extension
file wrapper:         44 byte RIFF header

So it seems, that Foobar internally first creates a wav-file and after that the WavPack-file?

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack?
Reply #9
Yes, what you say is correct. Foobar converts DSD to PCM as the first step. Whether they want to handle DSD “natively” is up to them; it would be a lot of work.

BTW, that's not the real md5 sum right??

As for SACD-ISO, I'm not sure of the right answer. Obviously converting them to DSF would work, but some people don't want to do that. If the ISOs use DST compression already, then they will get bigger with WavPack because DST compression is better than WavPack (but uses more CPU to decode).

When I was working on this I looked for SACD-ISOs online and could not find any. If you don't have a PlayStation, how does one get them?

  • conta69
  • [*]
Re: How to compress DSD with WavPack?
Reply #10
BTW, that's not the real md5 sum right??

Good eye...  :D
Had to type the text, because copy/paste doese not work with the command-prompt in Win7.

And you are right with the converting from sacd-iso to dsf. The DST-data consumes a lot of disc-space when extracted. Nearly every multichannel-SACD is compressed to DST. So it would be a very pleasant innovation to have a easy-to-use file-format where all DSD-data is compressed.
The second point, why converting from iso's to dsf-files can be problematic, is a technical thing: sometimes a loud crack is audible between the tracks when playing back the single files. It seems nobody knows exactly whats the reason for that. So many of the users leave the iso's untouched, which, on the other hand, is quite impractical.

For getting SACD-images, maybe this article is of interest for you:
link to computeraudiophile.com

For further information i send you a private mail.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack?
Reply #11
That's interesting news to me! I was planning on getting a PlayStation one day, but I already have an Oppo BDP-103, so I think I'll give that a try. Thanks!

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Re: How to compress DSD with WavPack?
Reply #12
When I was working on this I looked for SACD-ISOs online and could not find any. If you don't have a PlayStation, how does one get them?
Hello David, legal ISO files are rare but you can buy some on Pentatone webshop:
http://www.pentatonemusic.com/catalog
(not the whole catalog but only "Buy & download" titles)

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack?
Reply #13
Thanks, guru, I think I'll get one or two of those. Unfortunately my Oppo player won't play them directly (unless it's hacked), but I can certainly convert them to DSF. Anyway, it will give me some iso samples to investigate.

  • kode54
  • [*][*][*][*][*]
  • Administrator
Re: How to compress DSD with WavPack?
Reply #14
Have a look here:

https://github.com/jevinskie/sacd-ripper

This doesn't really step into the bounds of DRM extraction, since it requires a custom firmware on an old PS3 to actually break the DRM. The tools on that repository can extract ISO images to various file formats, including DSF. It can also remove the DST compression.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack?
Reply #15
Yeah, I became very familiar with that software when I was thinking of getting a PS3. I can't tell you how much time I spent on eBay looking for the right model, firmware, etc. I would have, but they were just a little too expensive to be unnoticed.

Now that the ripping can be done on lots of other machines, I wonder if those PS3s will become cheaper. I was also interested in installing Lunux and playing around with the cell processor (a guy I worked with wrote the book Programming the CELL Processor and he talked about it all the time).

  • conta69
  • [*]
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #16
Just for curiosity:
The handling with sacd-iso is quite impractical for different reasons. To split the iso to dsf/dff-files is a good alternative, but it seems, there are technical problems with that - there are some discussions, that there are obvious audible clicks which are not existing when listening to the iso.
Naturally there is the same situation, when the dsf-files are compressed with WavPack.
So, alltogether the situation is not too satisfying.  O:)

A possibility to avoid all these problems, would be to create a single wavpack-file with a cue-sheet from the iso, just like it is possible with audio-cd's. For audio-cd's this is a very common procedure to store the audio-data on the hdd.
For sacd-iso's such a possibility would be perfect, for different points:
1. no more discussions about audible clicks, because the "raw-material" did not change
2. the library stay in order - 1 iso = 2 files on the hdd
3. the tagging can easily be done with a lot of different software - tagging is a difficult thing regarding sacd-iso
4. the wavpack-image can be played back with every software which is capable of playing dsd-wavpack

So, do you think it is possible to create such a solution?
I am sure, a lot of users would highly appreciate this!  ;D

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #17
Yes, this certainly seems like a possibility. I will have a look once I get some sacd-iso files and become a little more familiar with them. WavPack also supports embedded cuesheets, so it might be that Foobar2000 would handle this without modification (the only thing I'm not sure about is how different sample rates are handled in cuesheets).

However, keep in mind that WavPack DSD compression, even in the high mode, is still noticeably worst than the DST compression that most sacd-iso's use, so these files are going to grow, maybe by 20% or more. It might make more sense to come up with a way of tagging them.

Another thing I don't really understand is why someone hasn't been able to put tags on DFF/DST files. The format uses the standard “chunk” format, and so it would be easy to simply put a ATAG chunk on the end with an APEv2 tag there, and the file would still be fully Philips compliant.

  • conta69
  • [*]
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #18
Quote
Another thing I don't really understand is why someone hasn't been able to put tags on DFF/DST files. The format uses the standard “chunk” format, and so it would be easy to simply put a ATAG chunk on the end with an APEv2 tag there, and the file would still be fully Philips compliant.

Thats interesting - i was convinced, that it is impossible to add tags to dff-files because of technical reasons. Strange then, that this deficit was ignored.
The reason might be, that both formats, dff and dsf, were both not regarded as very important formats, but more as freaky stuff.  :D
It is only about a year, that Mp3tag is capable to tag dsf and i dont think there are a lot more apps who can do that.

For the compression-rate:
You are (of course) right - DST is the format with the best rate. I tested a dsf-file anhd came to the following result:
conversation to wavpack        - 63,9% (from original)
conversation to wavpack high - 48,9%
conversation to DST                - 42,1%

So yes, DST is the best, BUT wavpack high is not so far away.  ;)
And the other thing to keep in mind is the fact, that very much SACDs are stereo only - i don't know the exact numbers, but in my view about 50%. And nearly all of them are stored in uncompressed DSD. So a transformation of these stereo-content to wavpack always brings a huge amount of hdd-saving.
I dont know, if there even exists any possibility to convert to DST and when there is a possibility, nobody uses it because of the tagging-problems of dff.

The cue-sheet support for iso's could be a good idea. Now there is the possibility to let Foobar with input-sacd-plugin create a xml-file which works like a cue sheet. The problem is, that it is not a real standard - so it works well in Foobar, but seems very choppy in the sacd-iso-fork of MPD. I don't know, if there is another app, which handle this iso-xml's.
To my opinion the important points for the playback of iso's are the following 3:

1. Ability of opening isos - Most of the apps cannot read them.
2. Integrating the iso-content in the database - I only know Foobar with it's xml-files, maybe there are some more apps which can handle this, but most of them cannot.
3. Playback dsd-content as dsd - also not a lot of apps are capable of this, but the number is increasing.

With Wavpack the first 2 points are unproblematic, because nearly all the playback-apps can open wavpack and most of them should recognize a integrated cue-sheet. So only the last point is a little tricky, but i am sure  with a little time all of the dsd-compatible apps will "learn" dsd-wavpack.  O:)

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #19
The reason might be, that both formats, dff and dsf, were both not regarded as very important formats, but more as freaky stuff.  :D
It is only about a year, that Mp3tag is capable to tag dsf and i dont think there are a lot more apps who can do that.

I think that both DSDIFF and DSF were designed to be SACD master formats, not consumer formats. Philips and Sony both assumed (I'm guessing) that unrippable SACDs would replace conventional CDs and that there would be no need for anyone outside of the industry to use these files. Unsurprisingly, they were wrong.

As for encoding DST, there are very expensive professional programs that can create them (presumably for SACD mastering), but I don't imagine that most people would have access to these (at least not legally). There are also sources for a reference encoder floating around, but (also also not unsurprisingly) it does not compress as well as the professional encoder, and in fact performs almost exactly the same as the WavPack "high" mode. Also, the only copy of it I used was so slow as to be unusable (like 0.1X realtime) although this could probably be optimized like the decoder was.

  • kode54
  • [*][*][*][*][*]
  • Administrator
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #20
The decoder wasn't optimized. It was brute forced through multithreading.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #21
Ah, well perhaps it was compiled differently then, because I have only 2 cores and foo_input_dsdiff is way more than twice as fast as the stand-alone decoder I tried. I've seen -O2 make a big difference.

Or maybe the stand-alone decoder has some other bottleneck, like some absurd I/O routines or something. I never looked at the code to figure out what was going on.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #22
I have experimented with creating a WavPack DSD image with embedded cuesheet, and they play great (and gapless) in the latest Foobar2000, both stereo and 5.1 multichannel versions! I used the program sacd_extract (on Linux) to extract the audio from an ISO as a DFF “edit master”, which is basically a whole-album DFF file with a separate cuesheet. I also used the option to decompress DST. Then I compressed that with WavPack, embedding the cuesheet, and also some cover artwork that I downloaded separately.

At first Foobar2000 would not recognize the cuesheet and I was afraid that this wasn't going to be easy, but then I discovered that the cuesheet started with a UTF-8 BOM marker (0xEF, 0xBB, 0xBF) that WavPack was not removing and Foobar2000 was not accepting. I deleted that with a text editor and it worked perfectly. It might be possible to just use Foobar2000 to embed the cuesheet, but I didn't try that.

I'll fix WavPack to correctly interpret and ignore that BOM marker, and then this should work for those who would like to have this option.

  • Jackal29a
  • [*][*]
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #23
I have tried compressing DSF files with Foo bar using the default settings changing only compression mode from normal to extra high but the wv's created playback as 24/352 PCM only wheras the ones created with wavpack frontend in "high" playback as DSD.
What parameters would I need to use in the custom mode command line in Foobar to get DSD output?
TIA

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How to compress DSD with WavPack? (moved from General Audio)
Reply #24
Currently there's no way to create native DSD files in Foobar2000 because the converter can only output PCM. For now you'll have to use the wavpack frontend, and add tags afterward with Foobar2000 (or an other tagger).

With the next wavpack release you will also be able to add an option in the frontend to have the ID3 tags in DSF files copied to the wavpack files. You can read a little about this here.