Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Error converting DSF to WavPack (Read 1835 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Error converting DSF to WavPack

After testing WavPack DSD support in Lyrion I set about converting my entire DSD library to lossless WavPack by walking the tree and running the following in each folder:
Code: [Select]
wavpack -d -h --import-id3 -m --threads=12 -v *.dsf

For a handful of albums folders I've encountered the following:
Code: [Select]
blocks not padded with NULLs, MD5 will not match!                                
original md5: b10299d0ecdcedf6c67ff98688f3cc9f                                
verified md5: bc3c5ea56adcf948f2e30ec2cb9ed56d                                
MD5 signatures should match, but do not!

Any idea what causes this, and whether there are any workarounds?

Re: Error converting DSF to WavPack

Reply #1 –
Files violate the DSF spec. Explained by David here: https://hydrogenaud.io/index.php/topic,113213.msg939507.html#msg939507

Does it have any consequences? If it is in the fade-out then - if I have understood it correctly - it is no worse than how two ripped CDs when offset-adjusted may differ at the very beginning or end (of the CD, not between-tracks), because of what is supposed to be digital silence still has dither ... or something like that.

If it is in between track with music across the track transition, then I am lost at what could happen - if it is supposed to be a thing at all. Should one even have used DSD then, as it cannot handle transitions? DSD should be used only for "full-disc images", and for stand-alone tracks that fade to silence across, or have I gotten that wrong?
Myself I have only "used" DSD on downloaded demonstration tracks, but even then they sometimes violate the spec this way: https://hydrogenaud.io/index.php/topic,124188.msg1030011.html#msg1030011 Kinda careless I think - posting "look how good this is" and then with garbled end, how do they know that <random player> won't end it with a noise tick? But DSD was never a useful end-user format anyway.

Re: Error converting DSF to WavPack

Reply #2 –
Thanks for the explanation and link.  Good to know.

Re: Error converting DSF to WavPack

Reply #3 –
One thing that hasn’t been mentioned is why some files are padded with values other than zero. It turns out that zero is a good padding value for PCM because it’s silence, but for DSD all zeros is full scale minus. So if a defective DSF player ignored the sample count in the header and “played” that solid block of zeros if would create a loud glitch. So someone had the bright idea of padding with a value (or string of values) that constitutes DSD silence instead, like for example 0x5a. Those files would play without a glitch in the defective player (although they would not be gapless). Too bad it wasn't Sony who had that bright idea.

To encode these files with WavPack simply skip the -m and -v options and they’ll be fine. Afterward, if you like, you can re-encode them with -m and -v using WavPack's transcoding feature so that they’ll have a new MD5 for the corrected data so you can fully verify them in the future.

The only possible issue is that in the future if you convert them back to DSF files and play them on such a defective player you’ll get the glitches. However, this seems unlikely because if you were using such a defective play then you would be getting glitches on your other, correctly padded files.

Re: Error converting DSF to WavPack

Reply #4 –
That's interesting. The 1990s called and reminded of its Xing headers and all the other MP3 format retro-hacks.

... is DFF the same? Potential consequences: can one actually transfer a DSD stream between DSF and DFF without creating issues at the end?

Quote from: bryant on
To encode these files with WavPack simply skip the -m and -v options and they’ll be fine.
Will that change anything to the audio but the feedback to the user? (I would need to find back one of those pesky files to check, hence the stupid question.)

(And, does WavPack anyway restore the file?)

Re: Error converting DSF to WavPack

Reply #5 –
So DFF does not use blocks at all; it’s interleaved channels, one byte each, all the way through the file. As it should be, IMHO.

DSF divides the data into blocks of 4096 bytes per channel. So for a stereo file you have 4096 bytes of left, then 4096 bytes of right, etc. Leftover bytes in the last block are zero, and you can see why the file couldn’t just terminate early because you still have to finish the other channels. You could terminate the last channel of the last block, but come on. The header indicates the exact sample count, so it’s easy to know how many bytes in the final block to ignore (despite some software not knowing).

Another wrinkle is that DSF files can have the bits in each byte ordered MSB first or LSB first. So there are really three native data formats for uncompressed DSD, and it’s possible to losslessly switch between them, but the same exact audio will have 3 different MD5 sums.

In WavPack, raw DSD is stored in one of two modes (normal and high), but there’s no distinction made based on the source format. The data is always stored exactly the same way and there’s no padding because each frame can have an arbitrary number of samples. There is some metadata stored that indicates which of the three output formats is used, and of course the source file’s headers and trailers are stored verbatim so that the exact source file can be restored.

So when you compress one of these DSF files with the bad padding, WavPack will report that the padding is wrong and warn that the MD5 (if you store one) will not match the original file (because we make no attempt to save that bad padding, and the bad padding is part of the MD5 data). So to compress them you want to omit the -m so that no MD5 will be generated and -v to make sure no verification is done.

At that point, you have a perfect WavPack file, except it has no MD5 checksum. And that’s not so bad because WavPack has two other ways to verify a file is not corrupted, but if you do want an MD5 in there for other reasons (like for cataloging) then you can add one by re-encoding the file using the transcode option, just like you would add an MD5 sum to any WavPack file (it’s a waste of energy because the encoding is redundant, but I don’t have a command or option to just decode a file and add an MD5). The MD5 will be based on the padding being zeros because we have no idea at this point what was there in the original file.

If you later restore the original file with wvunpack, it will have zeros for padding instead of whatever the original file had, so it's not technically lossless (which is why we warned in the first place), but it is now a compliant file.

I have attached a 1 second file with bad padding to experiment with.
 

Re: Error converting DSF to WavPack

Reply #6 –
Right ... so a bit on topic here, in case someone still reads:

wavpack -m uncompressed-sourcefile writes the "original" MD5 - this is a situation where they differ.
And - I just learned - then you cannot recompress.



Now as for the file spec ... no idea whether https://www.sonicstudio.com/pdf/dsd/DSDIFF_1.5_Spec.pdf  is the current, but that one says says "please fill ZERO(0x00) for unused sample data area in the block."
Where to put "please" in the MAY/SHOULD/MUST hierarchy?  ??? 
Anyway, it seems to require 1/75th of a second granularity, which must then mean 64*588 samples, that's not very 4096 ...
But offers ID3.

DSDIFF has eight samples (= one byte) per channel per "frame cluster" - it might be sensible to read bytes nor bits sure, I'd suppose that an 8 sample granularity is fine enough when they have to process the thing in PCM at 1/8 of the sample rate - and then it actually has metadata (artist etc.) chunks that that ... nobody reads?


Quote from: bryant on
Another wrinkle is that DSF files can have the bits in each byte ordered MSB first or LSB first. So there are really three native data formats for uncompressed DSD
*whistles* WSD ...
(FFmpeg can demux ... but into what?)

Re: Error converting DSF to WavPack

Reply #7 –
Quote from: Porcus on
Right ... so a bit on topic here, in case someone still reads:
I sincerely hope that nobody is still reading...    :D

Quote
wavpack -m uncompressed-sourcefile writes the "original" MD5 - this is a situation where they differ.
And - I just learned - then you cannot recompress.
Haha, I just discovered (or rediscovered) that myself. Apparently WavPack transcode will verify the MD5 whether you ask it to or not, and will delete the output on a failure. Not ideal. Found an ugly work around:
Code: [Select]
wvunpack dsf-bad-padding.wv -o - | wavpack - -o dsf-bad-padding-fixed.wv

Quote
Now as for the file spec ... no idea whether https://www.sonicstudio.com/pdf/dsd/DSDIFF_1.5_Spec.pdf  is the current, but that one says says "please fill ZERO(0x00) for unused sample data area in the block."
Where to put "please" in the MAY/SHOULD/MUST hierarchy?  ??? 
Anyway, it seems to require 1/75th of a second granularity, which must then mean 64*588 samples, that's not very 4096 ...
But offers ID3.
So I think you mixed up your metaphors here. The spec you link to is the DFF spec and the line pleading zeros is from the DSF spec, which is found for example here.

And that spec, BTW, is one of the most horribly written I have encountered, not to mention it's a poorly thought out format in the first place.

Specifically note that the bit-order option is contained in a field called Bits per sample with the value "1" meaning LSB first and "8" meaning MSB first. And in case that didn't make sense, [annotation4] makes it very clear...not.

The ID3 is nice, although the sacd_extract program (part of sacd-ripper) puts ID3 tags in DFF files too because it's a reasonable chunked file format.

Quote
DSDIFF has eight samples (= one byte) per channel per "frame cluster" - it might be sensible to read bytes nor bits sure, I'd suppose that an 8 sample granularity is fine enough when they have to process the thing in PCM at 1/8 of the sample rate - and then it actually has metadata (artist etc.) chunks that that ... nobody reads?
I've never seen the other DFF metadata fields used, just like they don't seem to be ever used in WAV files (although I did find out recently that FFmpeg writes, and presumably reads, them).

Quote
Quote from: bryant on
Another wrinkle is that DSF files can have the bits in each byte ordered MSB first or LSB first. So there are really three native data formats for uncompressed DSD
*whistles* WSD ...
(FFmpeg can demux ... but into what?)
I think it always comes out as PCM. FFmpeg does not pass around DSD in any form, AFAIK.

Re: Error converting DSF to WavPack

Reply #8 –
Yeah, I sat with both the documents and pasted wrong URL. Meh. Yes that spec would have given me a headache if I had to do something about it.