Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Another HDCD thread... (Read 7195 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Another HDCD thread...

While updating the release information for the albums in my collection.  I discovered one of my CDs is an HDCD.  Nowhere on the case does it mention anything about being HDCD.  After scanning my collection, I have almost 2 dozen CDs that claim to be HDCD, and only one has the HDCD logo on the case.  It seems for most of these CDs, they were probably unknowingly mastered on a Pacific Microsonics ADC, and therefore don't use any of the HDCD features.  In my current collection, 4 use the Peak Extend feature, one of those uses the Low Level Extend feature (though it only seems to affect the silence between tracks), and none use the transient filters.  Seems the transient filters were never used due to patent issues.

The 4 albums that use Peak Extend are:
Green Day - Nimrod (the only one labeled as HDCD)
KMFDM - Symbols
KMFDM - MDFMK (Remixes from Symbols)
Orgy - Candyass (2 tracks don't use PE)

(Open the images in a new tab to see the original size)

Spectrograms showing where PE and LLE are active on the Nimrod album




Spectrograms showing where PE and LLE are active on the Candyass album.  You can see where tracks 2 and 5 are, since they don't use PE.




I wanted to see a visual of what the different is between the original audio and the audio after decoding with the HDCD filter.  The filter lowers the gain by -6dBFS, and outputs to 24-bit WAV.  I used ffmpeg to apply the filter, and to draw the waveforms.  The waveforms are the original audio (orange, lowered by 6dBFS) over the decoded waveform (light orange).

This is the track "Stitches" on Candyass, which doesn't actually use Peak Extend.


This is the track "Blue Monday" that does use Peak Extend.


I attached a couple 30 second samples of what the hdcd filter reconstructed.  Seems to affect mostly percussion, but the vocals and other things can be faintly heard.

I also attached the results of what I found in my collection.  There are HDCD lists of varying details and completion floating around on other forums and wikis.  Some of the albums I have I didn't see on any lists. 

Re: Another HDCD thread...

Reply #1 –
I see a couple of "detectable errors", they might be false positives (the Kenny Wayne Sheppard track?)
Lateralus is a well-known example. IIRC the HDCD manages to switch on for the lead-in silence. A mistake.

Lucky for those compilation tracks that you might keep HDCD off.

Re: Another HDCD thread...

Reply #2 –
I didn't bother trying to decode any tracks that didn't have "peak_extend: enabled permanently" detected.

I wonder if it's really necessary to keep the audio as 24-bit when Peak Extend only uses 1 bit of the 4 extra bits that hdcd adds.  Decoding with the hdcd filter drops the gain 6dB and gives back the headroom for that 1-bit, right?

24-bit track
Code: [Select]
Overall     Left      Right
DC offset  -0.000000 -0.000000 -0.000000
Min level  -0.750519 -0.750137 -0.750519
Max level   0.750900  0.750900  0.750900
Pk lev dB      -2.49     -2.49     -2.49
RMS lev dB    -17.98    -18.10    -17.87
RMS Pk dB     -12.77    -12.80    -12.77
RMS Tr dB     -52.42    -52.42    -46.86
Crest factor       -      6.03      5.88
Flat factor     0.00      0.00      0.00
Pk count           6         3         9
Bit-depth      24/24     24/24     24/24
Num samples    16.2M
Length s     367.640
Scale max   1.000000
Window s       0.050

16-bit track
Code: [Select]
Overall     Left      Right
DC offset  -0.000000 -0.000000 -0.000000
Min level  -0.750519 -0.750122 -0.750519
Max level   0.750916  0.750916  0.750885
Pk lev dB      -2.49     -2.49     -2.49
RMS lev dB    -17.98    -18.10    -17.87
RMS Pk dB     -12.77    -12.80    -12.77
RMS Tr dB     -52.42    -52.42    -46.86
Crest factor       -      6.03      5.88
Flat factor     0.00      0.00      0.00
Pk count        5.50         3         8
Bit-depth      16/16     16/16     16/16
Num samples    16.2M
Length s     367.640
Scale max   1.000000
Window s       0.050

Re: Another HDCD thread...

Reply #3 –
You can decode them on-the-fly, and keep the 16-bit files.

Decoding with 1 bit drop will decimate away the LSB yes, and if you convert to 16 bits you will lose 1 bits resolution - remember that also the quiet parts of the track will lose the LSB. Not saying that 15 is too little.

Whether "24" ... Peak Extend increases the range by one bit, LLE less than one. The rest of the "20 bits" resolution is dithering in the mastering - that is something you would have to do anyway, so it is not unique to the HDCD process, but it was just taken onboard the "HDCD process" to make it sound more impressive (or well it could maybe be defended by "fixes it so you don't need to remember it", but that left us with all the fake HDCDs).
Then IIRC from their white paper, they got up to 19.alittle bits after some nice calculations which included a never-implemented DAC response to choice of transient filter.  And "more than 19 bits" require 20 bits in the integer domain, justifying rounding up rather than to closest  ;D

"24" over "20" then:
Sure FLAC allows 20 bits natively, but storing in 24 bits could make more compatible (still! Look at file 37 at https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench ) - and at negligible compression penalty of less than 0.1 kbit/s or something compared to a 20-bit FLAC file. (Note, ALAC and Monkey's will both take a performance hit.)
As for WAVE/AIFF, there is no size difference between 17 bits and 23 bits - both are just 24 with a flag saying "use the top 17" or "use the top 23".

Re: Another HDCD thread...

Reply #4 –
Quote from: Replica9000 on
I wonder if it's really necessary to keep the audio as 24-bit when Peak Extend only uses 1 bit of the 4 extra bits that hdcd adds.  Decoding with the hdcd filter drops the gain 6dB and gives back the headroom for that 1-bit, right?

24-bit track
Code: [Select]
Overall     Left      Right
...
Bit-depth      24/24     24/24     24/24
Looks like according to SoX the LSB in the 24-bit sample is used.

Re: Another HDCD thread...

Reply #5 –
About the bit-depth indication, for example, here are the scanning result of my oldsCool software of the two HDCD decoded files in this post:
https://hydrogenaud.io/index.php/topic,122263.msg1009263.html#msg1009263
Code: [Select]
-------------------------------------------------------------------------------
E:\download\hdcdexe.wav
00:00:15 = 1323000 samples / 2-ch @ 44100 Hz
24-bit fixed point
Bit    Count            Percent
0      118          0.008919124
8      199           0.01504157
9      426           0.03219955
10     847           0.06402116
11     1730           0.1307634
12     3303           0.2496599
13     6834           0.5165533
14     13527           1.022449
15     27112           2.049282
16     54488           4.118518
17     107067          8.092744
18     205345          15.52116
19     346154          26.16432
20     382560           28.9161
21     161670          12.21995
22     11612          0.8777022
23     8           0.0006046863
Round Trip: 20
-------------------------------------------------------------------------------
E:\download\ffmpeg.wav
WAVE_FORMAT_EXTENSIBLE
00:00:15 = 1323000 samples / 2-ch @ 44100 Hz
24-bit fixed point
Bit    Count            Percent
0      118          0.008919124
8      199           0.01504157
9      426           0.03219955
10     847           0.06402116
11     1730           0.1307634
12     3303           0.2496599
13     6834           0.5165533
14     13527           1.022449
15     27112           2.049282
16     54488           4.118518
17     107067          8.092744
18     205345          15.52116
19     346154          26.16432
20     382560           28.9161
21     161670          12.21995
22     11612          0.8777022
23     8           0.0006046863
File saved in F:\Programming\oldsCool\oldsCool\bin\Release\download
Use WAVE_FORMAT_PCM
-------------------------------------------------------------------------------
oldsCool 1.0.0.4
Quote from: bennetng on
My guess is SoX only sees the used highest bit and lowest bit, without checking anything in-between. My software on the other hand also looks for missing bits in-between.
Despite what is being shown in SoX, oldsCool only sees 17 bits from these two decoded files.

Re: Another HDCD thread...

Reply #6 –
Quote from: Porcus on
(still! Look at file 37 at https://wiki.hydrogenaud.io/index.php?title=FLAC_decoder_testbench )
The mentioned file:
https://github.com/ietf-wg-cellar/flac-test-files/blob/main/subset/37%20-%2020%20bit%20per%20sample.flac
In this case my software indeed shows 20 bits in the sample distribution table.
Code: [Select]
WAVE_FORMAT_EXTENSIBLE
00:00:07.9962187 = 1535274 samples / 2-ch @ 96000 Hz
24-bit fixed point
Bit    Count            Percent
0      83           0.005406201
5      104          0.006774035
6      212           0.01380861
7      385           0.02507696
8      715           0.04657149
9      1409          0.09177515
10     2609           0.1699371
11     4864           0.3168164
12     9306           0.6061459
13     17478           1.138429
14     32195            2.09702
15     56957           3.709892
16     100675          6.557461
17     177774           11.5793
18     290413          18.91604
19     400960          26.11651
20     335253          21.83669
21     85494           5.568648
22     17652           1.149762
23     736           0.04793933
Round Trip: 20
Use WAVE_FORMAT_PCM
BitSort 1.0.0.4
Hopefully a fully implemented HDCD decoder can show a similar result if the HDCD specs itself is not a scam. The 20/20 and 24/24 files reported by SoX indeed require 20 and 24 bits to store losslessly, but they have not utilized all 20 or 24 bits of quantization levels properly.

Re: Another HDCD thread...

Reply #7 –
I was imprecise. "Look at file 37" I wrote, but I meant "look at that column": Several players choke on 20-bit FLAC files, so storing the signal in 24-bit FLAC might still be "safer".

Re: Another HDCD thread...

Reply #8 –
Quote from: Porcus on
I was imprecise. "Look at file 37" I wrote, but I meant "look at that column": Several players choke on 20-bit FLAC files, so storing the signal in 24-bit FLAC might still be "safer".
I understood your intent of posting this file was to demonstrate a compatibility issue. I just conveniently borrowed this file to answer @Replica9000 's doubt about the required bit-depth to store the decoded file. Different decoders generate files with different rendered bit-depth, however the distribution of samples still different from a "normal" 20-bit file.

BTW, Adobe Audition 1.5 failed to open this 20-bit flac file, Audacity 2.3.3 decoded incorrectly with very low volume level.

I also agree that if possible, decode the file on the fly during playback instead of render to a file. If one needs to render to a file, don't delete the original source.

Re: Another HDCD thread...

Reply #9 –
Quote from: bennetng on
BTW, Adobe Audition 1.5 failed to open this 20-bit flac file, Audacity 2.3.3 decoded incorrectly with very low volume level.

@ktf : How do you propose that such software be entered in the testbench table? After all, "doing playback" isn't their primary function.
Pretend it is, and just check whether they can decode to appropriate output?
Anyway, one may consider a separate section, rather than put them into the alphabet with software players?

Re: Another HDCD thread...

Reply #10 –
Attached some other results decoded using foo_hdcd with settings "Halve output volume only if peak extension is enabled".

Re: Another HDCD thread...

Reply #11 –
Quote from: Porcus on
@ktf : How do you propose that such software be entered in the testbench table? After all, "doing playback" isn't their primary function.
Pretend it is, and just check whether they can decode to appropriate output?
Anyway, one may consider a separate section, rather than put them into the alphabet with software players?
I don't see the difference really. For players, there are four categories, plays correctly, rejects, fails (i.e. plays incorrectly) or crashes. The same holds for such software, imports correctly, rejects, fails (imports incorrectly) or crashes. To test whether it imports correctly or not, the files have to be played back, so no difference there. Sure, they can go in a separate category, but I'd say not because they're different but to keep the table well organized (with not too many items under one category).
Music: sounds arranged such that they construct feelings.

Re: Another HDCD thread...

Reply #12 –
Quote from: bennetng on
Attached some other results decoded using foo_hdcd with settings "Halve output volume only if peak extension is enabled".

It looks like none of those files are using more than 18 bits.


FLAC might support 20-bit files, but neither sox or ffmpeg would let me create a 20-bit file.

I'll probably leave them as 24-bit files since I don't use any media players that will do on-the-fly hdcd decoding that I'm aware of.  I always keep the original files backup up.

Re: Another HDCD thread...

Reply #13 –
https://foobar.hyv.fi/?view=foo_dsp_mdadither
This plugin for example can shave off the residue bits (which I consider they are useless processing leftovers) to reduce encoded flac file size. Since HDCD claims 20 bits there is no need to dither, as dither will increase file size.
X
SoX only requires one sample in each file to determine the bit-depth, like the attached examples.

Re: Another HDCD thread...

Reply #14 –
Looks like I can make a 20-bit file with SoX.

Code: [Select]
sox input_hdcd.flac -b 24 output_20b.flac dither -p 20
Code: [Select]
             Overall     Left      Right
DC offset  -0.000000 -0.000000 -0.000000
Min level  -0.750521 -0.750137 -0.750521
Max level   0.750900  0.750900  0.750900
Pk lev dB      -2.49     -2.49     -2.49
RMS lev dB    -17.98    -18.10    -17.87
RMS Pk dB     -12.77    -12.80    -12.77
RMS Tr dB     -52.42    -52.42    -46.86
Crest factor       -      6.03      5.88
Flat factor     0.00      0.00      0.00
Pk count           3         3         3
Bit-depth      20/20     20/20     20/20
Num samples    16.2M
Length s     367.640
Scale max   1.000000
Window s       0.050


FLAC shows that not every frame has wasted bits.  Maybe it's just dither?
Code: [Select]
flac -ac input_hdcd.flac | awk '/wasted_bits/ {print $2}' | sort | uniq -c

input_hdcd.flac: done
   4786 wasted_bits=0
     73 wasted_bits=1
     25 wasted_bits=2
     10 wasted_bits=3
      3 wasted_bits=4
      3 wasted_bits=5
    281 wasted_bits=6
   1857 wasted_bits=7

Re: Another HDCD thread...

Reply #15 –
Quote from: Replica9000 on
Looks like I can make a 20-bit file with SoX.

Code: [Select]
sox input_hdcd.flac -b 24 output_20b.flac dither -p 20
This means dither to 20 bits, because SoX uses a traditional approach (contrast to e.g. Case's smart dither), the LSBs will always be filled, even when the input is digital silence (all zero).

One more thing. People usually say 6dB = 1 bit shift, but this is not accurate enough for actual processing in DAWs etc. A factor of 0.5x is the exact value, and requires -6.0206dB for software doing 32-bit float math, and -6.020599913279624dB for 64-bit float. So if people treat a bit shift as 6dB and do a subtraction test, there will be more leftovers than using a factor of exactly 0.5x.

Re: Another HDCD thread...

Reply #16 –
Quote from: bennetng on
One more thing. People usually say 6dB = 1 bit shift, but this is not accurate enough for actual processing in DAWs etc. A factor of 0.5x is the exact value, and requires -6.0206dB for software doing 32-bit float math, and -6.020599913279624dB for 64-bit float. So if people treat a bit shift as 6dB and do a subtraction test, there will be more leftovers than using a factor of exactly 0.5x.

Matters if you ask it to attenuate "6 dB" and expect flat out zero yes. The HDCD whitepaper has the same ambiguity, actually: "allowing the average signal level to be increased as much as 6 dB or 1 bit for material with very high but infrequent peaks". Halving is enough, so you don't have to worry about those 0.02 dB running you past digital full scale :-)

(By the way: care to run the FLAC testbench with the DAW software?)

Re: Another HDCD thread...

Reply #18 –
In fact, foo_hdcd x64 will output int32 files which show 32/32 in SoX. Because flac 1.4 supports int32, one can try to use foobar2000 x64 and decode to int32 without dither, and compare the flac file size against a dithered 20-bit decode.

The frame order of animated gif below:
[1] Undecoded 16-bit
[2] Decoded to 32-bit
[3] Decoded to 20-bit without dither
[4] Decoded to 20-bit with flat TPDF dither

Unit in Y axis is LSBs relative to 16-bit quantization so the undecoded file snap to exactly 1 LSB interval, the decoded files won't due to PE and LLE. I adjusted the vertical zoom level of decoded and undecoded files (+/-10 LSB vs +/-30 LSB).

With frequency domain manipulation (filtering) you can have a lot of bits, like DSD to PCM conversion. Otherwise it is more or less like zooming the waveform in and out for LLE.
X

Re: Another HDCD thread...

Reply #19 –
Quote from: bennetng on
In fact, foo_hdcd x64 will output int32 files which show 32/32 in SoX. Because flac 1.4 supports int32, one can try to use foobar2000 x64 and decode to int32 without dither, and compare the flac file size against a dithered 20-bit decode.
Did you find much bloat?

Nothing that happens down those bits would be accurate anyway. IIRC, we got the following generations in succession:
* Pacific's own chips. Since HDCD wasn't an open thing, then what the encoding and decoding chips actually did, would be the reference, unless one could demonstrate that the decoder didn't match the encoder. (The white paper includes features never implemented ... and who knows whether the patent fully matches the implementation.)
* Microsoft's software implementation in WMP. Microsoft then had acquired Pacific. WMP is as good as we get, unless there is evidence to the contrary. But note that at this stage, there could have been good reasons to deviate on when to switch on and off; switching from a HDCD to a non-HDCD does no longer take the time of ejecting and physically replacing it.
(Also, at this stage one would have known that compilations could have mixed content ...)
* hdcd.exe: reverse engineering the WMP output.
* foo_hdcd: mimicking hdcd.exe . At this stage we are quite sure there are some discrepancies in how "doubtful" packets are interpreted.
* IIRC, ffmpeg's decoder built on foo_hdcd. But at least in some version, "doubtful" packets were treated different.

If any of these are implemented in near-accurate floating-point without round-off, then please remember there was no way to get that out of the original hardware. (That doesn't mean it was "wrong" in an application that would do the rounding itself. And anyway, those bits aren't audible anyway.)

Re: Another HDCD thread...

Reply #20 –
https://hydrogenaud.io/index.php/topic,126497.msg1049823.html#msg1049823
HDCD Sampler Volume 2
06. Finale from Sinfonietta

flac -8
Code: [Select]
  Length Name
  ------ ----
39012970 undecoded.flac
39981965 20bit-no-dither.flac
41250247 24bit-no-dither.flac
41710962 int32-no-dither.flac
43584576 int32-no-dither-gx4-optimize-int32.wv
43585540 int32-no-dither-gx4.wv
52749175 20bit-TPDF.flac
73587068 undecoded.wav
The very reason that I specifically mentioned dither. Of course, if The Human Centipede assumption is real (Pacific > Microsoft > hdcd.exe > foo_hdcd > ffmpeg) then there is not much to talk about.

Re: Another HDCD thread...

Reply #21 –
Quote from: Replica9000 on
In my current collection, 4 use the Peak Extend feature, one of those uses the Low Level Extend feature (though it only seems to affect the silence between tracks), and none use the transient filters.  Seems the transient filters were never used due to patent issues.
There are discs in the wild with Transient Filter enabled. I've found two so far: Chris Isaak: Speak of the Devil and Tim Reynolds: Chaos View. According to FB2k's HDCD scanner the Chris Isaak uses all three HDCD features but the Tim Reynolds only has Transient Filter enabled. FB2K's HDCD decoder does spit out different bits when decoding Chaos View.

Re: Another HDCD thread...

Reply #22 –
I know ffmpeg's HDCD filter if based on FB2K's filter, so I'm assuming they behave similarly.

I do have a couple albums that have a transient filter detected.  If I decode the original file with the HDCD filter vs just lowering the gain -6dB and outputting to 24-bit, they seem the same.  The waveform seems identical.  Sox stats shows a difference of 16/17 vs 23/24 with the LSB/MSB, but maybe this is a difference in dithering?  Unlike SoX, I don't think ffmpeg has any options to control dither for audio.

Other than lowering the gain -6dB, I don't know what the HDCD filter does to the audio if no HDCD features are actually present.

Re: Another HDCD thread...

Reply #23 –
Quote from: Lazlo Nibble on
Quote from: Replica9000 on
In my current collection, 4 use the Peak Extend feature, one of those uses the Low Level Extend feature (though it only seems to affect the silence between tracks), and none use the transient filters.  Seems the transient filters were never used due to patent issues.
There are discs in the wild with Transient Filter enabled. I've found two so far: Chris Isaak: Speak of the Devil and Tim Reynolds: Chaos View. [...]

Yeah, but it during ADC only, according to this post: https://hydrogenaud.io/index.php/topic,79427.msg836350.html#msg836350
If I have understood it correctly, the idea was to allow for
1) a different filter to be applied in the ADC when the analog signal had enough transients,
2) flag in the HDCD packets that this had been applied
3) ... so that the HDCD-aware DAC could treat those samples different.
And then it turned out, there was a patent in the way. And so part 3 was never implemented.

I do have a few CDs with "all three features" enabled according to fb2k; that would be TF as "Intermittent": The "HDCD Sampler Volume 2" from 1995, and a few Neil Young CDs.
Also, I find a couple with TF but no PE: Glenn Hughes: "Addiction" (has LLE) and Cypress Hill: "Skull & Bones" (does not have LLE).

Re: Another HDCD thread...

Reply #24 –
In this case a software decoder needs decode to a higher sample rate (e.g. 2x) for the transient filter to work. The filter response should change over time in the decoded file depending on the HDCD control signal. Here is a mockup:
X