Skip to main content

Topic: Extract HDCD (Read 164147 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
Extract HDCD
Reply #25
Thanks for looking into this.

I amplified the waveforms for this song to the same relative volume.
This didn't get as much peak extension as the other title, but still a good 3 dB increase in dynamics.


What exactly do you mean by this? You normalized them in Audacity? What do they look like without matching levels? Why was level matching required?

I didn't change my levels at all. If you could describe your procedure I will check my results against it.
  • Last Edit: 27 February, 2007, 12:59:25 AM by Eric Carroll

  • markanini
  • [*][*][*][*][*]
Extract HDCD
Reply #26
For the sake of visual comparison I adjusted the waveforms of Sister Andrea  to the same relative volume, the original would be 3 dB louder and the WMP output 3 dB quieter. I other words the WMP output for Sister Andrea was 6 dB quieter overall, but the peaks went up to -3 dB.

Extract HDCD
Reply #27
Well your results sure look like what I expected to see: clipped material becoming unclipped. Maybe my HDCD is just not encoded with the compression part enabled... Puzzling. I certainly see the 6dB quieter effect, but no enhancement on the peaks.

Just to confirm, you are using WMP9?
  • Last Edit: 27 February, 2007, 01:17:12 AM by Eric Carroll

  • markanini
  • [*][*][*][*][*]
Extract HDCD
Reply #28
I used WMP 10.

  • SebastianG
  • [*][*][*][*][*]
  • Developer
Extract HDCD
Reply #29
I recall someone mentioning on the audioasylum forum that this peak compression isn't used on every HDCD disc. (Sorry, no reference) This should explain your observations.

Cheers!
SG
  • Last Edit: 27 February, 2007, 03:51:00 AM by SebastianG

  • user
  • [*][*][*][*][*]
Extract HDCD
Reply #30
I have an alternative idea to capture the 2 streams (1. 15-bit+16th-HDCD-bit-as-noise --- 2. the-full-HDCD-decoded-wav).

If you have a soundcard with digital in and out, capable of 24 bit (and not resampling 44.1 to 48 kHz should be a basic, otherwise these tiny differences could be destroyed anyways),
plus an amp/receiver with HDCD decoder, you might be able to play the HDCD as normal CD and record it again,
and the next step:
Play digitally the HDCD, the amp decodes as true HDCD, and loops with some luck the decoded signals back to soundcard, which with some luck could record this.
But I am not sure, if soundcards and amps are capable of same time passing through digital in and out.

Or another alternative:
You have a DVD/CD player, which is able to decode HDCD to hdcd and give out it analouge.
Many cheap "no-name (lol, often better than well-known brand-name players, at least in features)" DVD-players should be able to do so nowadays.
You could record the analouge output of this DVDplayer, and get the HDCD content. If you have a good soundcard with some middle to good recording capabilities of 24bit/44.1/48/96 kHz or so, you could be successful.

edit addon:
I think, already my Cyberhome 635 DVD-A/V/SACD/CD player can decode HDCD and give out analouge. Together with terratec ewx 2496 I could record the wav. hm, but my time  Sooner, but more likely later, I could carry out this experiment
  • Last Edit: 21 March, 2007, 06:46:05 AM by user
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials

Extract HDCD
Reply #31
Quote
How I Can extract HDCD by software(EAC, WMP9...)?
[{POST_SNAPBACK}][/a]




Yes,

EAC will extract the HDCD bit for bit,
you need a Lossless compression format (wavpack, flac, ape), or the original waves. (see for Lossless extraction [a href="http://www.high-quality.ch.vu]www.high-quality.ch.vu - users' audio guides[/url]

eg. foobar2000 will play those files perfectly, if you set the bit depth to 16 bit, 24 bit will destroy the readability for an external HDCD decoder like built in amp.
(same for 5.1-dts-wav-CDDA).

If eg. foobar2000 plays the Lossless files, you need a soundcard, which ddoesn't upsample internally from 44.1 to 48 kHZ, and digital output to external hdcd decoder, like in amps.


Foobar 24 bit does not destroy the readability of an external decoder. Works just fine.

  • krabapple
  • [*][*][*][*][*]
Extract HDCD
Reply #32
Quote
I was always under the impression that HDCD was simply a better dithering algorithm.
[a href="index.php?act=findpost&pid=268929"][{POST_SNAPBACK}][/a]


HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG


Still interested in this (mainly so I can archive my HDCDs as 'decoded' flacs)....so, what is actually 'done' during the decoding step?  I presume the dither/noise shaping is already 'built into' the recording, and comes through regardless of whether an HDCD decoder is used.  So I'm guessing the decoding is really just the 'decompression' step (acting on the optional peak  compression) plus the choosing of an anti-aliasing filter.  (And regarding that, can we assum,e that because WMP has an HDCD decoder, that it also has a variety of virtual anti-aliasing filters to choose from?)

  • SebastianG
  • [*][*][*][*][*]
  • Developer
Extract HDCD
Reply #33
I reviewed the papers about HDCD you can find on the net again recently. Here's my current view on it:

The dithering/noise shaping part they were advocating is nothing special. In fact, it is inferior to what other programs do (like Foobar or my requant.jar tool) and not provably safe in the sense that it avoids nonlinear distortions. You can't even call it "noise shaping" because it only introduces a bit of colored dither which has more power above 16 kHz -- it doesn't shape quantization noise.

If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).

Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)

HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.

The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.

Since the "benefits" of HDCD seem rather small it's -- in my humble opinion -- not worth to decode the stream to something like 24/44 or 24/88. One should rather keep it like it is and/or use a HDCD capable post processing algorithm within your player of choice (which only is supported by Microsoft's Mediaplayer IIRC).

In terms of signal-to-noise ratio (I'm not referring to peak signal to noise ratio), proper HDCD decoding won't help you a thing. The only thing you can improve is restoring the slightly compressed dynamics.

Cheers!
SG

edit: fixed typos
  • Last Edit: 28 June, 2007, 06:26:11 AM by SebastianG

Extract HDCD
Reply #34
I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136


If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).


If range expansion is enabled then samples less than -3.11dBFS (<= 0x5980) are attenuated by 6dB, samples greater than -3.11dBFS are attentuation progressively less, giving the end mapping of 0dBFS to 0dBFS.

If range expansion is not enabled (including non hdcd's) all samples are simply attenuated by 6dB.

Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)


I'm not sure how this works exactly.  There are 3 bits left that could be used for controlling the reconstruction filters, although there were always zero in all my test cds, and wmp didn't seem to respond to them at all.

HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.


Exactly so.

The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.


Just needed a little bit of cryptanalysis.

On a candidate hdcd, during the silent intro at the start, the left and right channels matched exactly with a delay of a few hundred samples.  This delay was rather fortunate, as it was hence, possible to identify where an hdcd code had been inserted by looking for a section where the correlation between the two channels stopped briefly.  The locations of these sections were such that it was clear that the hdcd codes had been inserted into the left and right channels at the same time, and it was hence possible to assemble a set of encoded hdcd codes.  Guessing that hdcd codes would contain some sort of magic number to mark them, and that a maximal length LFSR would have been used, it was simply a matter of trying all maximal length LFSRs on the data and looking for some correlation between them.

Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris

  • SebastianG
  • [*][*][*][*][*]
  • Developer
Extract HDCD
Reply #35
Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.

Cheers!
SG

Extract HDCD
Reply #36
Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.


Thanks Sebastian,

I don't suppose you have a reference for the paper do you.  I worked from what I found freely available on the internet, which mentioned that LFSRs were used, but not what tap values.  Two example lengths were given, but the actual length was neither of those.

Chris

  • eevan
  • [*][*][*][*][*]
Extract HDCD
Reply #37
You can find this link at Wikipedia.

Take a look at the last page.

BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when  the decoder is decoding properly. Was it the same with previous WMP versions?
If age or weaknes doe prohibyte bloudletting you must use boxing

  • M
  • [*][*][*][*][*]
Extract HDCD
Reply #38
I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?

    - M.

Extract HDCD
Reply #39
You can find this link at Wikipedia.

Take a look at the last page.

I wish I'd seen that document before I attempted this, it would have made things quite a bit easier.  I had seen the LFSR diagrams however.

BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when  the decoder is decoding properly. Was it the same with previous WMP versions?


From memory, the hdcd logo did come and go with (wmp 10?) for a bit, then spontaneously and permanently vanished.  It hasn't reappeared with the upgrade to wmp 11.

Chris

  • eevan
  • [*][*][*][*][*]
Extract HDCD
Reply #40
I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?
  • Last Edit: 22 August, 2007, 12:14:32 PM by eevan
If age or weaknes doe prohibyte bloudletting you must use boxing

Extract HDCD
Reply #41

I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?


I'm afraid I know nothing about foobar2000, but I can't imagine any reason why such a plugin couldn't be written.  Monitoring an audio stream and looking for an hdcd marker is not a particularly expensive operation.  You would need some sort of look ahead to avoid playing the first samples of a track up to the first hdcd code without hdcd decoding however.

Chris

I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?


17 bits sounds about correct for a track just using peak expansion.  The source is 16 bits (15 bits audio + 1 bit dithering), and the peak expansion expands the top 3.1dB to 9.1dB giving you an extra bit.  The decoder will print out 'Detected HDCD' when it first sees a valid hdcd code in the audio.

Chris

  • eevan
  • [*][*][*][*][*]
Extract HDCD
Reply #42
Thanks!

Cheers
If age or weaknes doe prohibyte bloudletting you must use boxing

  • gabeg
  • [*]
Extract HDCD
Reply #43
Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris



So is this literally replicating what an HDCD filter supposed to do??

  • SebastianG
  • [*][*][*][*][*]
  • Developer
Extract HDCD
Reply #44
I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.

Cheers!
SG
  • Last Edit: 24 August, 2007, 04:17:01 AM by SebastianG

Extract HDCD
Reply #45



Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris



So is this literally replicating what an HDCD filter supposed to do??


It's replicating exactly what wmp does when playing back an hdcd with 24-bit output enabled.  I'd guess that that's pretty much identical to what happens in hardware, although I think there's may also be some upsampling done aswell.

Regards,

Chris



I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.


I calculated all my constants and the compression curve by passing test cases through wmp.  The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain. 

Reagrds,

Chris

  • dmckean
  • [*]
Extract HDCD
Reply #46
It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!

Extract HDCD
Reply #47
It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!


Glad to hear it's working.  The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

I've never used foobar, and don't really have the time to start developing a plugin myself.  However, if there are no objections from MS, I'll release the source code in a little while, and I'd hope it'd be pretty easy for someone to encorporate into a plugin.

Regards,

Chris

  • SebastianG
  • [*][*][*][*][*]
  • Developer
Extract HDCD
Reply #48
I calculated all my constants and the compression curve by passing test cases through wmp.  The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain.

Oh, right. Sorry, I forgot that you tested the 'black box' WMP.

The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

Let me just share my opinion on that with you all. I think it's not a good idea to decode HDCD "just for lossless storage" for the same reason you wouldn't download MP3 files and store them decoded as WAVs or FLACs on your harddisk. This decoding should only be done as preprocessing (prior lossy encoding) or as "postprocessing" (within a player, like the player decodes MP3). Otherweise you'll be wasting a lot of harddisk space.

I'm sure FLAC will have trouble catching the gaps (sample values not in use). For example: If you convert 16 bit samples to 24 bit samples and adjust the gain by -9dB to undo the dynamics compression only one out of 91 possible 24bit samples values is used (assuming you don't dither). Since 91 is not a power of two the least significant bits of the samples will look like random bits so that FLAC can't exploit the fact that the signal has an effective resolution of only 17.5 bits (16+1.5 due to the gain of -9 dB). So, approximately 573 kilobits/s (2*(24-17.5)*44100) are wasted (!!!).

Cheers!
SG
  • Last Edit: 27 August, 2007, 03:23:43 AM by SebastianG

Extract HDCD
Reply #49
Interesting stuff, I have 20~ HDCDs around here.

Going to have a play

EDIT:  How do you use this program, when I try and run it in the command prompt it just sits and does nothing.  I tried specifying the wav file name and it still does nothing.  What is the correct syntax?  Even tried drag and drop of the wav, didn't work either
  • Last Edit: 27 August, 2007, 04:55:08 AM by Walrusbonzo