HydrogenAudio

CD-R and Audio Hardware => CD Hardware/Software => Topic started by: SCIF on 2005-01-27 07:34:46

Title: Extract HDCD
Post by: SCIF on 2005-01-27 07:34:46
How I Can extract HDCD by software(EAC, WMP9...)?
Title: Extract HDCD
Post by: Garf on 2005-01-27 09:28:04
HDCD just uses the standard 16 bit CD Audio encoding, so EAC will rip the "HDCD" part too.

For playback, WMP9 can decode the HDCD.
Title: Extract HDCD
Post by: PoisonDan on 2005-01-27 09:50:48
Quote
For playback, WMP9 can decode the HDCD.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=268432")

But only from CD, not from the ripped audio files, apparently (according to [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=30200&view=findpost&p=261426]this post[/url]).

More about HDCD here (http://www.hydrogenaudio.org/forums/index.php?showtopic=517).
Title: Extract HDCD
Post by: Garf on 2005-01-27 10:44:45
Perhaps it could work when you mount the ripped wav with Deamontools.

I was thinking something like Virtual Audio Cable would let you get the HDCD decoded result, but I don't think those tools deal with >16 bit audio which makes them useless for this.
Title: Extract HDCD
Post by: SCIF on 2005-01-28 02:31:33
Quote
But only from CD, not from the ripped audio files

Is there programm for ripping HDCD in 24/44.1?
Title: Extract HDCD
Post by: precisionist on 2005-01-28 10:17:49
Quote
Quote
But only from CD, not from the ripped audio files

Is there programm for ripping HDCD in 24/44.1?
[a href="index.php?act=findpost&pid=268749"][{POST_SNAPBACK}][/a]

I don't think so, CD is 16bit.
More interesting:
Is there a program able to decode the HDCD info and to convert it to another (hIgher, 32bit float for example) format, and everything in digital way (no analogue step) ?

And what I'm always wondering:
Is HDCD a way to circumvent loudness; does a HDCD played as HDCD sound less clipressed compared to played as CD ?
Title: Extract HDCD
Post by: SebastianG on 2005-01-28 16:02:22
Quote
More interesting:
Is there a program able to decode the HDCD info and to convert it to another (hIgher, 32bit float for example) format, and everything in digital way (no analogue step) ?

And what I'm always wondering:
Is HDCD a way to circumvent loudness; does a HDCD played as HDCD sound less clipressed compared to played as CD ?
[a href="index.php?act=findpost&pid=268837"][{POST_SNAPBACK}][/a]


I seriously doubt that this would be worth the effort. Having read the technical infos about HDCD which are publicly available (the AES paper) I currently believe HDCD to be just a technology with quite low benefit compared to plain noise shaping. It's just a thing to make a bit money.

They are claiming 20 bits dynamic resolution. This is actually 16+1+3 bits where approximately one bit is gained via this peak compression and three bits are gained via noise shaping. While playing an HDCD signal WITHOUT a special decoder you'll roughly experience the same signal to noise ratio compared to the output of an HDCD decoder.  The peak decompression mechanism which is part of an HDCD decoder is fairly simple, but it won't increase the signal to noise ratio because most of the signal (except the peaks) will get attenuated by 6 dB.

IMHO the best thing would be to leave the format (44 kHz, 16 bit) the way it is. Even without a special  decoder you will still benefit from the noise shaping part and get roughly 19 bits dynamic resolution.


SebastianG

edit: grammar
Title: Extract HDCD
Post by: adlai on 2005-01-28 17:00:26
I was always under the impression that HDCD was simply a better dithering algorithm.
Title: Extract HDCD
Post by: SebastianG on 2005-01-28 17:59:15
Quote
I was always under the impression that HDCD was simply a better dithering algorithm.
[a href="index.php?act=findpost&pid=268929"][{POST_SNAPBACK}][/a]


HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG
Title: Extract HDCD
Post by: user on 2005-02-01 10:31:43
Quote
How I Can extract HDCD by software(EAC, WMP9...)?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=268419")




Yes,

EAC will extract the HDCD bit for bit,
you need a Lossless compression format (wavpack, flac, ape), or the original waves. (see for Lossless extraction [a href="http://www.high-quality.ch.vu]www.high-quality.ch.vu - users' audio guides[/url]

eg. foobar2000 will play those files perfectly, if you set the bit depth to 16 bit, 24 bit will destroy the readability for an external HDCD decoder like built in amp.
(same for 5.1-dts-wav-CDDA).

If eg. foobar2000 plays the Lossless files, you need a soundcard, which ddoesn't upsample internally from 44.1 to 48 kHZ, and digital output to external hdcd decoder, like in amps.
Title: Extract HDCD
Post by: ShowsOn on 2005-02-01 12:10:18
I  just noticed this thread, and given that I finally installed Windows XP last week I thought I would give try the HDCD playback in WMP XP.

I think the only HDCD I own is Pet Sounds by The Beach Boys, the 2001 remastered version.

I went into the devices menu and checked the box "play back audio CDs at 24 bit", is this all that is required to activate HDCD playback? I have a Chaintech 710 sound card set for 2 channel high sample rate mode.

I restarted WMP and when I started to playback the CD the first thing I noticed is that the volume was significantly quieter. Does this mean HDCD playback is working? Is there any other visible way to see that WMP is playing the CD back properly? I'm not really sure if playback sound better, but it is quieter than before I had checked the 24 bit playback box.
Title: Extract HDCD
Post by: user on 2005-02-01 12:45:48
as written above, normally 24 bit playback of audioCds destroys headers, like for HDCD, or 5.1-dts-CD

the HDCD has all information inside its standard 16 bits, the additional informations are stored in least significants bits, maybe inside the 16th.

The wmp player should show "HDCD" if it recognized a HDCD.
iirc, previous wmp version showed hdcd, but didn't decode ?
is it safe now, that wmp decodes HDCD ?

A safer way is,
as written above, play HDCD by foobar2000, 16 bit, and transmit the digital output to an external amp/hdcd-decoder.
which will show u the hdcd logo, if it recognized the true hdcd content an decodes it.

I think, most soundcards don't have such a good analogue output, that u will benefit from wmp hdcd decoding, better to let do the D/A conversion and  analogue site of the chain, by external amp.
Title: Extract HDCD
Post by: precisionist on 2005-02-02 12:26:06
Quote
I restarted WMP and when I started to playback the CD the first thing I noticed is that the volume was significantly quieter. Does this mean HDCD playback is working? Is there any other visible way to see that WMP is playing the CD back properly? I'm not really sure if playback sound better, but it is quieter than before I had checked the 24 bit playback box.
[a href="index.php?act=findpost&pid=270014"][{POST_SNAPBACK}][/a]

Sounds very hopeful. Usually quieter means better.
My question still has not yet been answered. Could it be that mastering engineers 'abuse' HDCD technology to hide a less clipressed version of the audio inside the clipressed one ?
Title: Extract HDCD
Post by: user on 2005-02-03 11:02:28
Quote
My question still has not yet been answered. Could it be that mastering engineers 'abuse' HDCD technology to hide a less clipressed version of the audio inside the clipressed one ?
[a href="index.php?act=findpost&pid=270324"][{POST_SNAPBACK}][/a]


No, I don't think, that's possible.
because they have only 16 bits stereo
So, they have maybe 15 bits for the CDDA wave, and the 16th, least significant bit contains a digital compressed information, which can only be decoded by HDCD decoder.
So, if you play a HDCD via a normal CDplayer, the 16th bits would be converted by normal DDA-converter to pure noise. (quite silent noise of course, it is the 16th bit...)

bryant (wavpack developer) offered some time ago a tool, which was able to remove the 16th bit with hdcd information. This should enable to ABX hdcd version against the 15 bit CDDA version.
Dunno, where he offered it (at r3mix forum iirc), ask him,
though I should have it somewhere on my HD or a DVD backup.
Title: Extract HDCD
Post by: Pio2001 on 2005-02-03 20:25:59
Quote
Quote
My question still has not yet been answered. Could it be that mastering engineers 'abuse' HDCD technology to hide a less clipressed version of the audio inside the clipressed one ?
[a href="index.php?act=findpost&pid=270324"][{POST_SNAPBACK}][/a]


No, I don't think, that's possible.[a href="index.php?act=findpost&pid=270555"][{POST_SNAPBACK}][/a]


It is possible. Why not ? Since they can play with the dynamics, and choose to have the player enhance it or not.
Title: Extract HDCD
Post by: precisionist on 2005-02-04 13:23:31
Quote
Quote
Quote
My question still has not yet been answered. Could it be that mastering engineers 'abuse' HDCD technology to hide a less clipressed version of the audio inside the clipressed one ?
[a href="index.php?act=findpost&pid=270324"][{POST_SNAPBACK}][/a]


No, I don't think, that's possible.[a href="index.php?act=findpost&pid=270555"][{POST_SNAPBACK}][/a]


It is possible. Why not ? Since they can play with the dynamics, and choose to have the player enhance it or not.
[a href="index.php?act=findpost&pid=270697"][{POST_SNAPBACK}][/a]

Then, again:
Is there a software (or any other way besides a digital-analogue-digital process), allowing me to convert HDCD to 20bit (or better) (or maybe only quieter 16bit) ?
Title: Extract HDCD
Post by: SebastianG on 2005-02-04 15:34:41
Quote
Then, again:
Is there a software (or any other way besides a digital-analogue-digital process), allowing me to convert HDCD to 20bit (or better) (or maybe only quieter 16bit) ?
[a href="index.php?act=findpost&pid=270891"][{POST_SNAPBACK}][/a]


To fully benefit from the HDCD encoded informations you have to convert the PCM stream to 88,2 kHz @ 24 bits. But I guess you only want to reverse the optional "peak compression" thingy which won't require upsampling. However, I don't think it'll be worth the effort. (!)

Is there a software ? Possibly. I don't know.
Is there a way ? Sure! Theoretically.
Does it make sense ? IMHO not.


SebastianG
Title: Extract HDCD
Post by: Pio2001 on 2005-02-04 22:29:46
No software, exept maybe the combination of Windows Media Player with something like Total Recorder.
Title: Extract HDCD
Post by: precisionist on 2005-02-07 16:01:25
Quote
Does it make sense ? IMHO not.
[a href="index.php?act=findpost&pid=270904"][{POST_SNAPBACK}][/a]

Depends on your personal priorities...In my opinion, 'manual SBR' wasn't worth the effort...  (I listened to those samples.)
Anyway, thank you for your help.
Title: Extract HDCD
Post by: krabapple on 2007-02-08 18:15:04
No software, exept maybe the combination of Windows Media Player with something like Total Recorder.



just to resurrect this from the dead (because I've been playing with HDCD + Total Recorder)  TR allows a max PCM recording format of 48 kHz/16 bit.  So if 88/24 is really required for proper PCM capture (as per SebastianG) , TR isn't the answer. 

I've approached this repeatedly but am still not quite certain how to do a proper HDCD decoded vs nondecoded data (not listening) comparison. I'd like to be able to compare .wav data using soemthing like Audition.  I've tried redigitizing the analog output from an HDCD-decoding player, and comparing it to the same file ripped, but in the end I'm not sure what I'm looking at.  Flat-topped peaks seem to go away but I'm not sure that isn't just due to capturing an analog output.

And btw I can't for the life of me get WMP 11 to recognize an HDCD (i.e., show the HDCD logo).  Anyone succeeded with this?
Title: Extract HDCD
Post by: Eric Carroll on 2007-02-26 05:56:52
There has been a long running thread (http://forums.slimdevices.com/showthread.php?t=32967) on the Slim Devices Audiophile forum where we have been working to decode HDCDs so we could play them back on our Squeezeboxen and Transporters. Best to read the thread backwards, its now 11 pages long!

We have reached the point where we definately have made progress. Using Windows Media Player 9, we can get the HDCD icon to light up, and we have successfully recorded bits using the Chronotron WAV Plugin for WMP9 (http://www.wmplugins.com/ItemDetail.aspx?ItemID=820) (using a 24b audio card and the 24b option enabled).

The issue at hand is if the plugin is actually recording 20b and writing a 24b WAV (actually its WAVEX format) or just recording 16b and padding zeros to 24b.

The only other alternative I have found to date is to use Virtual Audio Cable (VAC) (http://software.muzychenko.net/eng/vac.html) to see if it gets the same result as the plugin or something different. I have not yet tried this option.

If there is someone here who really understands HDCD, what is the expected result when comparing a 16b rip to a 24b WAV capture via WMP9? What should the waveforms look like if the decoder worked and the plugin actually captured 20b? Personally I would have expected increased dynamic range in accordance with the increased bit depth, and an unwinding of any range compression. So I should see spikier peaks and higher peak to average. Yes?

Here is my comparison of the 16b rip and the 24b capture:
(http://img401.imageshack.us/img401/4905/hdcddj2.th.jpg) (http://img401.imageshack.us/my.php?image=hdcddj2.jpg) The 16b rip is the trace on top and the 24b is the trace on the bottom. Closer inspection shows the peaks in the 24b capture are exactly 6dB down from the 16b capture. However, the peak to average seems about the same, thus the dynamic range did not increase. This suggests we don't actually have the HDCD decode captured.
Title: Extract HDCD
Post by: markanini on 2007-02-27 00:40:47
Wow, I wish I knew about this plugin before.
Finally I can automatically extract HDCD info!

@Eric Carroll:
All HDCD titles don't use peak extension, try some other titles.
Title: Extract HDCD
Post by: Eric Carroll on 2007-02-27 03:42:35
Wow, I wish I knew about this plugin before.
Finally I can automatically extract HDCD info!

@Eric Carroll:
All HDCD titles don't use peak extension, try some other titles.


Glad to help on the plugin... It still means you have to play the whole song in real time...

But we have two users who seperately captured via the plugin and all we saw was a decrease in peak levels.

I only have one HDCD that I know of... I was doing this as an experiment, hoping to get to ABX the before 16b and after 24b to see if it was worth buying more. While we have put alot of work in on this, we haven't yet gotten to the "is the difference audible" stage yet...

If you have some HDCDs, could you try doing a 16b rip, then a 24b capture and compare them in Audacity to see what result you get?
Title: Extract HDCD
Post by: markanini on 2007-02-27 04:26:26
Glad to help on the plugin... It still means you have to play the whole song in real time...

Not if you tick "Disable audio output(fast recording)". 

I will upload some waveforms as soon as I have the time.
Title: Extract HDCD
Post by: markanini on 2007-02-27 05:31:23
Since I got too excited about this, here are the waveforms:

Sound Chaser from Yes - Relayer


Sister Andrea from Mahavishu Orchestra - Lost Trident Sessions
[a href="http://img132.imageshack.us/my.php?image=sisterandreawv8.png" target="_blank"](http://img132.imageshack.us/img132/9457/sisterandreawv8.th.png) (http://img179.imageshack.us/my.php?image=soundchaserfa0.png)

I amplified the waveforms for this song to the same relative volume.
This didn't get as much peak extension as the other title, but still a good 3 dB increase in dynamics.
Title: Extract HDCD
Post by: Eric Carroll on 2007-02-27 05:57:31
Thanks for looking into this.

I amplified the waveforms for this song to the same relative volume.
This didn't get as much peak extension as the other title, but still a good 3 dB increase in dynamics.


What exactly do you mean by this? You normalized them in Audacity? What do they look like without matching levels? Why was level matching required?

I didn't change my levels at all. If you could describe your procedure I will check my results against it.
Title: Extract HDCD
Post by: markanini on 2007-02-27 06:08:37
For the sake of visual comparison I adjusted the waveforms of Sister Andrea  to the same relative volume, the original would be 3 dB louder and the WMP output 3 dB quieter. I other words the WMP output for Sister Andrea was 6 dB quieter overall, but the peaks went up to -3 dB.
Title: Extract HDCD
Post by: Eric Carroll on 2007-02-27 06:14:35
Well your results sure look like what I expected to see: clipped material becoming unclipped. Maybe my HDCD is just not encoded with the compression part enabled... Puzzling. I certainly see the 6dB quieter effect, but no enhancement on the peaks.

Just to confirm, you are using WMP9?
Title: Extract HDCD
Post by: markanini on 2007-02-27 06:17:29
I used WMP 10.
Title: Extract HDCD
Post by: SebastianG on 2007-02-27 08:49:17
I recall someone mentioning on the audioasylum forum that this peak compression isn't used on every HDCD disc. (Sorry, no reference) This should explain your observations.

Cheers!
SG
Title: Extract HDCD
Post by: user on 2007-03-21 10:41:23
I have an alternative idea to capture the 2 streams (1. 15-bit+16th-HDCD-bit-as-noise --- 2. the-full-HDCD-decoded-wav).

If you have a soundcard with digital in and out, capable of 24 bit (and not resampling 44.1 to 48 kHz should be a basic, otherwise these tiny differences could be destroyed anyways),
plus an amp/receiver with HDCD decoder, you might be able to play the HDCD as normal CD and record it again,
and the next step:
Play digitally the HDCD, the amp decodes as true HDCD, and loops with some luck the decoded signals back to soundcard, which with some luck could record this.
But I am not sure, if soundcards and amps are capable of same time passing through digital in and out.

Or another alternative:
You have a DVD/CD player, which is able to decode HDCD to hdcd and give out it analouge.
Many cheap "no-name (lol, often better than well-known brand-name players, at least in features)" DVD-players should be able to do so nowadays.
You could record the analouge output of this DVDplayer, and get the HDCD content. If you have a good soundcard with some middle to good recording capabilities of 24bit/44.1/48/96 kHz or so, you could be successful.

edit addon:
I think, already my Cyberhome 635 DVD-A/V/SACD/CD player can decode HDCD and give out analouge. Together with terratec ewx 2496 I could record the wav. hm, but my time  Sooner, but more likely later, I could carry out this experiment
Title: Extract HDCD
Post by: Night Rain on 2007-03-21 11:38:32
Quote
How I Can extract HDCD by software(EAC, WMP9...)?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=268419")




Yes,

EAC will extract the HDCD bit for bit,
you need a Lossless compression format (wavpack, flac, ape), or the original waves. (see for Lossless extraction [a href="http://www.high-quality.ch.vu]www.high-quality.ch.vu - users' audio guides[/url]

eg. foobar2000 will play those files perfectly, if you set the bit depth to 16 bit, 24 bit will destroy the readability for an external HDCD decoder like built in amp.
(same for 5.1-dts-wav-CDDA).

If eg. foobar2000 plays the Lossless files, you need a soundcard, which ddoesn't upsample internally from 44.1 to 48 kHZ, and digital output to external hdcd decoder, like in amps.


Foobar 24 bit does not destroy the readability of an external decoder. Works just fine.
Title: Extract HDCD
Post by: krabapple on 2007-06-27 20:44:06
Quote
I was always under the impression that HDCD was simply a better dithering algorithm.
[a href="index.php?act=findpost&pid=268929"][{POST_SNAPBACK}][/a]


HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG


Still interested in this (mainly so I can archive my HDCDs as 'decoded' flacs)....so, what is actually 'done' during the decoding step?  I presume the dither/noise shaping is already 'built into' the recording, and comes through regardless of whether an HDCD decoder is used.  So I'm guessing the decoding is really just the 'decompression' step (acting on the optional peak  compression) plus the choosing of an anti-aliasing filter.  (And regarding that, can we assum,e that because WMP has an HDCD decoder, that it also has a variety of virtual anti-aliasing filters to choose from?)
Title: Extract HDCD
Post by: SebastianG on 2007-06-28 11:21:49
I reviewed the papers about HDCD you can find on the net again recently. Here's my current view on it:

The dithering/noise shaping part they were advocating is nothing special. In fact, it is inferior to what other programs do (like Foobar or my requant.jar tool) and not provably safe in the sense that it avoids nonlinear distortions. You can't even call it "noise shaping" because it only introduces a bit of colored dither which has more power above 16 kHz -- it doesn't shape quantization noise.

If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).

Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)

HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.

The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.

Since the "benefits" of HDCD seem rather small it's -- in my humble opinion -- not worth to decode the stream to something like 24/44 or 24/88. One should rather keep it like it is and/or use a HDCD capable post processing algorithm within your player of choice (which only is supported by Microsoft's Mediaplayer IIRC).

In terms of signal-to-noise ratio (I'm not referring to peak signal to noise ratio), proper HDCD decoding won't help you a thing. The only thing you can improve is restoring the slightly compressed dynamics.

Cheers!
SG

edit: fixed typos
Title: Extract HDCD
Post by: Christopher Key on 2007-08-22 10:57:24
I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136 (http://forum.doom9.org/showthread.php?t=129136)


If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).


If range expansion is enabled then samples less than -3.11dBFS (<= 0x5980) are attenuated by 6dB, samples greater than -3.11dBFS are attentuation progressively less, giving the end mapping of 0dBFS to 0dBFS.

If range expansion is not enabled (including non hdcd's) all samples are simply attenuated by 6dB.

Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)


I'm not sure how this works exactly.  There are 3 bits left that could be used for controlling the reconstruction filters, although there were always zero in all my test cds, and wmp didn't seem to respond to them at all.

HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.


Exactly so.

The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.


Just needed a little bit of cryptanalysis.

On a candidate hdcd, during the silent intro at the start, the left and right channels matched exactly with a delay of a few hundred samples.  This delay was rather fortunate, as it was hence, possible to identify where an hdcd code had been inserted by looking for a section where the correlation between the two channels stopped briefly.  The locations of these sections were such that it was clear that the hdcd codes had been inserted into the left and right channels at the same time, and it was hence possible to assemble a set of encoded hdcd codes.  Guessing that hdcd codes would contain some sort of magic number to mark them, and that a maximal length LFSR would have been used, it was simply a matter of trying all maximal length LFSRs on the data and looking for some correlation between them.

Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris
Title: Extract HDCD
Post by: SebastianG on 2007-08-22 13:50:11
Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.

Cheers!
SG
Title: Extract HDCD
Post by: Christopher Key on 2007-08-22 14:04:36
Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.


Thanks Sebastian,

I don't suppose you have a reference for the paper do you.  I worked from what I found freely available on the internet, which mentioned that LFSRs were used, but not what tap values.  Two example lengths were given, but the actual length was neither of those.

Chris
Title: Extract HDCD
Post by: eevan on 2007-08-22 15:29:59
You can find this (http://web.archive.org/web/20020124220637/www.hdcd.com/partners/proaudio/AES_Paper.pdf) link at Wikipedia.

Take a look at the last page.

BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when  the decoder is decoding properly. Was it the same with previous WMP versions?
Title: Extract HDCD
Post by: M on 2007-08-22 15:33:51
I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136 (http://forum.doom9.org/showthread.php?t=129136)

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?

    - M.
Title: Extract HDCD
Post by: Christopher Key on 2007-08-22 16:28:05
You can find this (http://web.archive.org/web/20020124220637/www.hdcd.com/partners/proaudio/AES_Paper.pdf) link at Wikipedia.

Take a look at the last page.

I wish I'd seen that document before I attempted this, it would have made things quite a bit easier.  I had seen the LFSR diagrams however.

BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when  the decoder is decoding properly. Was it the same with previous WMP versions?


From memory, the hdcd logo did come and go with (wmp 10?) for a bit, then spontaneously and permanently vanished.  It hasn't reappeared with the upgrade to wmp 11.

Chris
Title: Extract HDCD
Post by: eevan on 2007-08-22 17:12:34
I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?
Title: Extract HDCD
Post by: Christopher Key on 2007-08-22 17:54:03

I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136 (http://forum.doom9.org/showthread.php?t=129136)

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?


I'm afraid I know nothing about foobar2000, but I can't imagine any reason why such a plugin couldn't be written.  Monitoring an audio stream and looking for an hdcd marker is not a particularly expensive operation.  You would need some sort of look ahead to avoid playing the first samples of a track up to the first hdcd code without hdcd decoding however.

Chris

I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?


17 bits sounds about correct for a track just using peak expansion.  The source is 16 bits (15 bits audio + 1 bit dithering), and the peak expansion expands the top 3.1dB to 9.1dB giving you an extra bit.  The decoder will print out 'Detected HDCD' when it first sees a valid hdcd code in the audio.

Chris
Title: Extract HDCD
Post by: eevan on 2007-08-22 17:58:37
Thanks!

Cheers
Title: Extract HDCD
Post by: gabeg on 2007-08-24 01:15:50
Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris



So is this literally replicating what an HDCD filter supposed to do??
Title: Extract HDCD
Post by: SebastianG on 2007-08-24 09:15:56
I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.

Cheers!
SG
Title: Extract HDCD
Post by: Christopher Key on 2007-08-24 11:03:17



Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris



So is this literally replicating what an HDCD filter supposed to do??


It's replicating exactly what wmp does when playing back an hdcd with 24-bit output enabled.  I'd guess that that's pretty much identical to what happens in hardware, although I think there's may also be some upsampling done aswell.

Regards,

Chris



I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.


I calculated all my constants and the compression curve by passing test cases through wmp.  The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain. 

Reagrds,

Chris
Title: Extract HDCD
Post by: dmckean on 2007-08-25 01:24:40
It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!
Title: Extract HDCD
Post by: Christopher Key on 2007-08-25 16:42:08
It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!


Glad to hear it's working.  The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

I've never used foobar, and don't really have the time to start developing a plugin myself.  However, if there are no objections from MS, I'll release the source code in a little while, and I'd hope it'd be pretty easy for someone to encorporate into a plugin.

Regards,

Chris
Title: Extract HDCD
Post by: SebastianG on 2007-08-27 08:21:58
I calculated all my constants and the compression curve by passing test cases through wmp.  The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain.

Oh, right. Sorry, I forgot that you tested the 'black box' WMP.

The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

Let me just share my opinion on that with you all. I think it's not a good idea to decode HDCD "just for lossless storage" for the same reason you wouldn't download MP3 files and store them decoded as WAVs or FLACs on your harddisk. This decoding should only be done as preprocessing (prior lossy encoding) or as "postprocessing" (within a player, like the player decodes MP3). Otherweise you'll be wasting a lot of harddisk space.

I'm sure FLAC will have trouble catching the gaps (sample values not in use). For example: If you convert 16 bit samples to 24 bit samples and adjust the gain by -9dB to undo the dynamics compression only one out of 91 possible 24bit samples values is used (assuming you don't dither). Since 91 is not a power of two the least significant bits of the samples will look like random bits so that FLAC can't exploit the fact that the signal has an effective resolution of only 17.5 bits (16+1.5 due to the gain of -9 dB). So, approximately 573 kilobits/s (2*(24-17.5)*44100) are wasted (!!!).

Cheers!
SG
Title: Extract HDCD
Post by: Walrusbonzo on 2007-08-27 09:45:27
Interesting stuff, I have 20~ HDCDs around here.

Going to have a play

EDIT:  How do you use this program, when I try and run it in the command prompt it just sits and does nothing.  I tried specifying the wav file name and it still does nothing.  What is the correct syntax?  Even tried drag and drop of the wav, didn't work either
Title: Extract HDCD
Post by: adamjk on 2007-08-27 11:36:09
Interesting stuff, I have 20~ HDCDs around here.

Going to have a play

EDIT:  How do you use this program, when I try and run it in the command prompt it just sits and does nothing.  I tried specifying the wav file name and it still does nothing.  What is the correct syntax?  Even tried drag and drop of the wav, didn't work either


Follow strictly commandline from doom9.org including <, > and spaces.
Title: Extract HDCD
Post by: Raiden on 2007-08-27 12:02:05
hdcd.exe < 16bit.wav > 24bit.wav
Title: Extract HDCD
Post by: Walrusbonzo on 2007-08-27 13:44:22
Thanks guys....  Got it working.
Title: Extract HDCD
Post by: Walrusbonzo on 2007-08-27 15:20:09
I've started to run all my known HDCD encoded CDs through the HDCD decoder and I'm then compressing using WavPack 4.42 (-hhx6).  I'll post more results once I get them.

WavPack compression results.

Buffalo Springfield, Last Time Around.


Original 16bit - 197mb - 806kbps
HDCD Decoded 24bit - 248mb - 1013kbps

Joni Mitchell, Clouds.


Original 16bit - 202mb - 749kbps
HDCD Decoded 24bit - 222mb - 827kbps

Joni Mitchell, Blue.


Original 16bit - 187mb - 724kbps
HDCD Decoded 24bit - 210mb - 811kbps

Joni Mitchell, Ladies of the Canyon.

Original 16bit - 259mb - 806kbps
HDCD Decoded 24bit - 282mb - 878kbps

Joni Mitchell, Song to a Seagull.


Original 16bit - 210mb - 771kbps
HDCD Decoded 24bit - 229mb - 841kbps

Van Halen, 1984.


Original 16bit - 246mb - 1031kbps
HDCD Decoded 24bit - 244mb - 1022kbps

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

Van Halen, Van Halen I.


Original 16bit - 260mb - 1025kbps
HDCD Decoded 24bit - 258mb - 1018kbps

Same again here for Van Halen.

Neil Young, American Stars 'n Bars.

Original 16bit - 240mb - 879kbps
HDCD Decoded 24bit - 295mb - 1082kbps

Neil Young, On The Beach.


Original 16bit - 216mb - 763kbps
HDCD Decoded 24bit - 265mb - 936kbps

King Crimson, Islands.


Original 16bit - 208mb - 640kbps
HDCD Decoded 24bit - 214mb - 680kbps

King Crimson, In The Wake Of Poseidon.

Original 16bit - 222mb - 757kbps
HDCD Decoded 24bit - 233mb - 796kbps

King Crimson, Larks' Tongues In Aspic.

Original 16bit - 241mb - 724kbps
HDCD Decoded 24bit - 251mb - 754kbps

King Crimson, Starless and Bible Black.

Original 16bit - 246mb - 739kbps
HDCD Decoded 24bit - 255mb - 764kbps
Title: Extract HDCD
Post by: skamp on 2007-08-27 16:38:41
FLAC seems to support (http://flac.sourceforge.net/format.html#frame_header) 20 bit audio. I don't suppose there would be much of a compression gain from encoding 20 bit audio instead of 24 bit?
Title: Extract HDCD
Post by: Walrusbonzo on 2007-08-27 17:43:31
FLAC seems to support (http://flac.sourceforge.net/format.html#frame_header) 20 bit audio. I don't suppose there would be much of a compression gain from encoding 20 bit audio instead of 24 bit?


Would make no odds.  This software HDCD decoder I'm using only outputs 24bit audio.
Title: Extract HDCD
Post by: SebastianG on 2007-08-27 18:10:57
One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

Quote
Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


Cheers!
SG
Title: Extract HDCD
Post by: Walrusbonzo on 2007-08-27 18:30:41
Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!


The point is easy to see.  I want to be able to play these CDs' songs at the highest quality I can with the equipment I have.  I don't have a HDCD decoder and I like to play all my music through my PC via Foobar.  So to me it makes perfect sense.  Don't be so narrow minded.

Quote

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


It could well be the reason.  I might try re-ripping it in case I made a mistake somewhere.
Title: Extract HDCD
Post by: skamp on 2007-08-27 18:42:55
Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

I agree, HDCD content should be decoded on playback. But in the real world, I doubt I'll see a HDCD decoder plugin for my player of choice (Audacious (http://audacious-media-player.org/Main_Page)) any time soon (and I don't own an audio receiver, let alone a HDCD-compatible one).
Title: Extract HDCD
Post by: lucas1985 on 2007-08-27 20:09:44
I'm with SebastianG. This decoder should be released as a player plug-in.
and I don't own an audio receiver, let alone a HDCD-compatible one

Cheap universal player (http://www.oppodigital.com/dv980h/)
Title: Extract HDCD
Post by: edwardar on 2007-08-27 23:05:18
Can I just say I'm very excited about this tool!

Did a quick ABX on a beach boys HDCD disc, and couldn't tell the decoded one from the standard one, but I'll keep trying... perhaps with a more modern CD.

Wouldn't it be great if foobar was the second player (after WMP) to support decoding of HDCDs!

Ed
Title: Extract HDCD
Post by: Christopher Key on 2007-08-27 23:17:46

The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

Let me just share my opinion on that with you all. I think it's not a good idea to decode HDCD "just for lossless storage" for the same reason you wouldn't download MP3 files and store them decoded as WAVs or FLACs on your harddisk. This decoding should only be done as preprocessing (prior lossy encoding) or as "postprocessing" (within a player, like the player decodes MP3). Otherweise you'll be wasting a lot of harddisk space.


The analogy to mp3 decoding is a good one.  Conceptually, hdcd is really just a lossy way of encoding 20 bit audio with (by design) the very useful property that the compressed output is very similar to a 16 bit version of the original.

I'm inclined to agree with you in principal that the decoding process should be done during playback or transcoding.  In practice however, one doesn't always have the means to do this, and converting before hand is a good alternative.  I would *strongly* recommend never deleting the original rip however, as this is the master copy.  Hopefully plugins will become available, removing the need to store a preconverted version though.

I'm sure FLAC will have trouble catching the gaps (sample values not in use). For example: If you convert 16 bit samples to 24 bit samples and adjust the gain by -9dB to undo the dynamics compression only one out of 91 possible 24bit samples values is used (assuming you don't dither). Since 91 is not a power of two the least significant bits of the samples will look like random bits so that FLAC can't exploit the fact that the signal has an effective resolution of only 17.5 bits (16+1.5 due to the gain of -9 dB). So, approximately 573 kilobits/s (2*(24-17.5)*44100) are wasted (!!!).


I've not looked closely at how FLAC operates, but this certainly sounds highly likely.  Huffman coding would work relatively well for this sort of situation, but would be fairly pointless for most audio, so is likely not used.

One point to note however is that wmp does truncate the output to 20 bits; maybe FLAC would be able to take advantage of that.

Regards,

Chris


Quote

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed.  I'd guess:

Peak extend enabled at some point during the audio
Peak extend disabled at some point during the audio
Peak extend used (|sample|>0x5980 while peak extend used)
Range of gain adjustment values used.
Title: Extract HDCD
Post by: Dynamic on 2007-08-28 10:06:54
The older foobar v0.8 series do allow Playback/Bit Depth to be specified using "Additional Options" which includes 20-bit padded to 24-bit. FLAC and Wavpack are supposed to take advantage if they detect that the bottom four bits are set to zero. If they aren't zeroed already using hdcd.exe then fb2k might save you a few kbps in your FLACs if you can be bothered to output to 20-bit padded to 24.
Title: Extract HDCD
Post by: Christopher Key on 2007-08-29 19:12:30
I'll have a go tomorrow at adding some more diagnostics


Well there's a new version available with a few more options and some more detailed information.  For now it's available from,

http://www.srcf.ucam.org/~cjk32/hdcd/hdcd-new.zip (http://www.srcf.ucam.org/~cjk32/hdcd/hdcd-new.zip)

but it'll replace the existing version,

http://www.srcf.ucam.org/~cjk32/hdcd/hdcd.zip (http://www.srcf.ucam.org/~cjk32/hdcd/hdcd.zip)

fairly shortly.


To get usage information, run,

hdcd.exe -h

I won't explain here as I'd like the usage information to be self explanatory. If anything's unclear, let me know and I'll explain and update it.

Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris
Title: Extract HDCD
Post by: SiriusB on 2007-08-30 03:38:16
One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!



Hi guys, great thread.  I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs;  the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD.  You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism? 


Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe.  FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example).  I couldn't tell them apart in an ABX.

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs).  THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction.  see this post (http://www.stevehoffman.tv/forums/showpost.php?p=2817631&postcount=1) for examples.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them.  Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.


All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')

I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed.  I'd guess:

Peak extend enabled at some point during the audio
Peak extend disabled at some point during the audio
Peak extend used (|sample|>0x5980 while peak extend used)
Range of gain adjustment values used.


From what I've read (e.g. John Marks' self-interview about his HDCD releases)  it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion.  So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).
Title: Extract HDCD
Post by: eevan on 2007-08-30 11:39:56
Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs).  THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction.  see this post (http://www.stevehoffman.tv/forums/showpost.php?p=2817631&postcount=1) for examples.


You're right. I have the japanese HDCD release of Malmsteen's Concerto Suite for Electric Guitar and Orchestra. Here is the output of the hdcd.exe:

Code: [Select]
D:\>hdcd.exe -o track08.wav 08_-_Track08.wav
HDCD Detected
Features Used:
  Peak extend   : Not enabled
  Minimum gain  : 0.0dB
  Maximum gain  : 0.0dB

D:\>

The Audition says that the actual bit depth of decoded wav is 17 bits, and the peak is -6.01 dB. This is the same for all tracks on the CD.

 
From what I've read (e.g. John Marks' self-interview about his HDCD releases)  it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion.  So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

When I play this CD in WMP11 without 24 bit playback enabled, I see the HDCD logo. But when I enable 24 bit playback (as it should be), the logo doesn't show up.
Title: Extract HDCD
Post by: skelly831 on 2007-08-30 15:33:44
When I play this CD in WMP11 without 24 bit playback enabled, I see the HDCD logo. But when I enable 24 bit playback (as it should be), the logo doesn't show up.

Same here, for a while the logo wouldn't show up at all, but after switching the 24-bit option on and off a few times it finally appears while the 24-bit option is off 

But I guess the 24-bit option is working because it sounds noticeably quieter.
Title: Extract HDCD
Post by: Christopher Key on 2007-08-30 16:33:24
Hi guys, great thread.  I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs;  the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD.  You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?


My view is that theoretically, you should only ever store the 16 bit master.  That can then be played back as 16 bit or 24 bit dependent upon the listening environment.  In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.

Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe.  FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example).  I couldn't tell them apart in an ABX.


That's rather worrying.  The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC.  (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly.  (Not strictly true.  A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested.  I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you?  I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs).  THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction.  see this post (http://www.stevehoffman.tv/forums/showpost.php?p=2817631&postcount=1) for examples.


Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes.  The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit.  Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them.  Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.


From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels.  If you're so inclined, this can be fixed with a hex editor.

All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')


There are 3 unidentified bits in the hdcd codes that might be used to switch filters.  My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them.  I'd be very grateful to hear from anyone who finds a CD in which they're used.

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps).  Is this what's referring to as 'low level extension'?

From what I've read (e.g. John Marks' self-interview about his HDCD releases)  it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion.  So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).


That certainly sounds sensible from a marketing point of view.  As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

Regards,

Chris
Title: Extract HDCD
Post by: krabapple on 2007-08-30 19:56:07
My view is that theoretically, you should only ever store the 16 bit master.  That can then be played back as 16 bit or 24 bit dependent upon the listening environment.  In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.


Theoretically very desirable, but practically not do-able in any easy way....so far.  I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)


Quote
That's rather worrying.  The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC.  (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly.  (Not strictly true.  A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested.  I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you?  I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia



It was Joni Mitchell 'Mingus'.  FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around.  To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks.  For hdcd.exe of course all I have to do is feed it the individual .wavs directly.  I can do it all again and compare the files.  Also  I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.


Quote
Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes.  The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit.  Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.




Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'.    The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

Code: [Select]
undecoded
    Left    Right
Min Sample Value:    -32452    -32429
Max Sample Value:    32440    32415
Peak Amplitude:    -.08 dB    -.09 dB
Possibly Clipped:    0    0
DC Offset:    0     0
Minimum RMS Power:    -138.15 dB    -139.94 dB
Maximum RMS Power:    -2.58 dB    -4.66 dB
Average RMS Power:    -11.14 dB    -11.51 dB
Total RMS Power:    -10.54 dB    -10.9 dB
Actual Bit Depth:    16 Bits    16 Bits

Using RMS Window of 50 ms



decoded
    Left    Right
Min Sample Value:    -16226    -16214.5
Max Sample Value:    16220    16207.5
Peak Amplitude:    -6.1 dB    -6.11 dB
Possibly Clipped:    0    0
DC Offset:    0     0
Minimum RMS Power:    -144.17 dB    -145.97 dB
Maximum RMS Power:    -8.6 dB    -10.68 dB
Average RMS Power:    -17.16 dB    -17.54 dB
Total RMS Power:    -16.56 dB    -16.92 dB
Actual Bit Depth:    17 Bits    17 Bits

Using RMS Window of 50 ms


I can send this track to you too if you like.


Quote
From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels.  If you're so inclined, this can be fixed with a hex editor.



Thanks. I'd much rather just use hdcd.exe, so long as it's creating files that are identical in all the ways that matter, to what the WMP WAV plugin generates.


Quote
There are 3 unidentified bits in the hdcd codes that might be used to switch filters.  My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them.  I'd be very grateful to hear from anyone who finds a CD in which they're used.


Check out the link I made, to my Steve Hoffman forum post  There I document a variety of hdcd behavior  -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor).  How can I tell if the 3 bits are in use?


Quote
Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps).  Is this what's referring to as 'low level extension'?



Beats me.  I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D  (I have yet to determine if these are applied at decoding too.. there was a huge flamewar (http://www.audioasylum.com/scripts/t.pl?f=dvda&m=24549) about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters).  Maybe there's more options?





Quote
That certainly sounds sensible from a marketing point of view.  As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.


IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'.  But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light. 

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').
Title: Extract HDCD
Post by: krabapple on 2007-08-30 20:20:51
Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris


here's that variety I mentioned, detected by hdcd.exe

Beck "Midnite Vultures', 02 - Nicotine & Gravy
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Van Halen "1984" 02 - Jump
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

King Crimson "Starless & Bible Black" 02 - Lament
HDCD Detected
Peak extend: Not enabled
Minimum gain: -4.0dB
Maximum gain : 0.0.dB

Joni Mitchell "Mingus" 09- The Dry Cleaner from Des Moines
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: -0.5 dB
Maximum gain : 0.0.dB

Yes "Yessongs" 05 - And You And I
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Yes "Big Generator" 04 - Almost Like Love
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB



(is there any way to get hdcd.exe to generate a log file?  I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)
Title: Extract HDCD
Post by: bryant on 2007-08-31 05:36:13
I've started to run all my known HDCD encoded CDs through the HDCD decoder and I'm then compressing using WavPack 4.42 (-hhx6).  I'll post more results once I get them.

Thanks for posting your compression results, however I was very surprised that some albums would actually get smaller after the HDCD decoding. I can only imagine two reasons this could happen. One is if the original data was not compressed with the same settings (say, -hhx instead of -hhx6), and the other was if in some cases the HDCD program is actually throwing away a bit (but that wouldn't make any sense).

I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations! 

BTW, I'm sure you have thought of this, but a very interesting complementary project would be an HDCD encoder. Then any band could release their own HDCD CDs, although Microsoft might have something to say about that! 
Title: Extract HDCD
Post by: SebastianG on 2007-08-31 08:27:33
I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Here's a possible explanation: The peak compression is heavily non-linear. It creates harmonic distortions which could be harder to predict than the original or decoded data.

Cheers!
SG
Title: Extract HDCD
Post by: Christopher Key on 2007-08-31 11:37:28
[quote name='krabapple' date='Aug 30 2007, 19:56' post='513502']
[quote name='Christopher Key' post='513457' date='Aug 30 2007, 11:33']

My view is that theoretically, you should only ever store the 16 bit master.  That can then be played back as 16 bit or 24 bit dependent upon the listening environment.  In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.
[/quote]

Theoretically very desirable, but practically not do-able in any easy way....so far.  I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)
[/quote]

Hopefully plugins will become available.  Storage isn't really the problem, HDCDs will always be a very small percentage of one's music library, it's more the incovenience of having two versions available stored, keeping them both tagged correctly, making the correct one gets selected etc.  It depends on your particular set up as to whether this is a problem

Quote

Quote

That's rather worrying.  The software was designed to be bit identical to wmp, and is for my 5 test cds.

Which CD was it that wasn't matching for you?


It was Joni Mitchell 'Mingus'.  FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around.  To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks.  For hdcd.exe of course all I have to do is feed it the individual .wavs directly.  I can do it all again and compare the files.  Also  I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.


If you get a chance, would you mind rechecking just one of the tracks.  You may need to watch out for sample offsets, I did find that wmp rather bizarrely advanced to first 2m by 4 samples, then corrected itself.  I've not been abl to get it to do it again however.

I'll see if I can acquire a copy of 'Mingus', along with a few others mentioned later to test this myself.

Quote

Quote

Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes.  The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit.  Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.


Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'.    The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

Code: [Select]
undecoded
    Left    Right
Min Sample Value:    -32452    -32429
Max Sample Value:    32440    32415
Peak Amplitude:    -.08 dB    -.09 dB

Actual Bit Depth:    16 Bits    16 Bits

Using RMS Window of 50 ms



decoded
    Left    Right
Min Sample Value:    -16226    -16214.5
Max Sample Value:    16220    16207.5
Peak Amplitude:    -6.1 dB    -6.11 dB

Actual Bit Depth:    17 Bits    17 Bits

Using RMS Window of 50 ms



That's exactly as expected.  If neither peak extend nor gain adjustment are used, the least significant seven bits of each sample will always be zero (as will the one most significant), hence the bit depth is detected as 17 bits.  I don't know why the peak amplitude values aren't exactly 6dB lower in the decoded case, (0.09dB -> 6.11dB, not just the way the values are rounded), but the sample values are *exactly* halved.


Quote

Quote

There are 3 unidentified bits in the hdcd codes that might be used to switch filters.  My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them.  I'd be very grateful to hear from anyone who finds a CD in which they're used.


Check out the link I made, to my Steve Hoffman forum post  There I document a variety of hdcd behavior  -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor).  How can I tell if the 3 bits are in use?


Thanks for the link, you initial post covers the behaviour nicely.  Category 3 and 4 are essentially the same from a technical point of view, they both have peak extend enabled, category 4 has larger sample value before decoding that go further up the expansion curve.  Given that I've got the data, I really should produce an accurate plot of the curve to demonstrate how it works.

Quote

Quote
Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps).  Is this what's referring to as 'low level extension'?



Beats me.  I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D  (I have yet to determine if these are applied at decoding too.. there was a huge flamewar (http://www.audioasylum.com/scripts/t.pl?f=dvda&m=24549) about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters).  Maybe there's more options?


http://web.archive.org/web/20030605091427/...o/overview.html (http://web.archive.org/web/20030605091427/www.hdcd.com/partners/proaudio/overview.html)

Mentions those three, and only those three, so I suspect that those are the options.  I'll try to use their terminology from now on to avoid confusion.

Interestingly, the low level range extension referred to there (20dB in 4dB steps) is much different to and stronger that used by wmp (7.5dB in 0.5 dB steps).

I did catch the AA thread but got lost fairly quickly as a result of the strength of their convictions.


Quote

Quote


That certainly sounds sensible from a marketing point of view.  As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.


IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'.  But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light. 

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').


Quite agree, particularly the loudness point.  You can justify the behaviour from a technial point of view, if normal CDs weren't attenuated, and an hdcd code was spuriouusly detected during playback, you'd get a sudden jump in volume.  On the other hand, when they are already attenuated by 6dB, a spuriusly detected hdcd code will just give you slightly more dynamic for a short period, which is much less offensive.  I doubt this is the primary motivation however!


[quote name='krabapple' date='Aug 30 2007, 20:20' post='513505']
[quote name='Christopher Key' post='513258' date='Aug 29 2007, 14:12']
Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris
[/quote]

here's that variety I mentioned, detected by hdcd.exe

Beck "Midnite Vultures', 02 - Nicotine & Gravy
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Van Halen "1984" 02 - Jump
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Yes "Yessongs" 05 - And You And I
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

[/quote]

I'll try and get a copy of those that have the unknown codes.  The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes.  This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing.  If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.

Quote

(is there any way to get hdcd.exe to generate a log file?  I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)


Just redirect stderr to a file.  Using the newer version, it should be as simple as,

hdcd.exe -o out.wav in.wav 2>mylog.txt

and with older version that only supports stdin / stdout,

hdcd.exe < in.wav > out.wav 2>mylog.txt


Also, take a look at the slimdevices thread, that's got some guidance on how to use this in a script,

http://forums.slimdevices.com/showthread.p...967&page=24 (http://forums.slimdevices.com/showthread.php?t=32967&page=24)


I did a quick scan of my library and was interested to find that all my Mark Knopfler CDs are HDCD, despite not being marked as such, as was *one* track from a Bond themes compilation CD.  I'm guessing the compiler probably didn't realise either!!!
Title: Extract HDCD
Post by: Christopher Key on 2007-08-31 11:55:00
Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations! 



Ouch, trying to do it with only an analogue reference was going to be seriously hard!

I'd be very interested to know how you went about deriving the tap values used for the LFSR.  There's a breif description of how I went about it earlier in this thread, but I can't believe that was the easiest way.


BTW, I'm sure you have thought of this, but a very interesting complementary project would be an HDCD encoder. Then any band could release their own HDCD CDs, although Microsoft might have something to say about that! 


I would certainly be technically possible, although to do it well would require a lot more knowledge of audio processing than I have at present.  The decoding is easy, there's no decisions to be made.  Encoding isn't quite as straighforward.  Peak extend is fine, low level range extend is fine with a bit of care to get a suitable level of hysteresis, but getting the dithering right is probably quite hard.  I also have no idea yet quite how the filter switching works, given that wmp doesn't appear to do that.

It'd certainly be something to speak to Microsoft about, they may well not object.  No HDCD encoders are being sold at present, so it's not really providing any competition, and electronic releases of HDCD music might well encourage some people to switch to wmp for playback.

Regards,

Chris
Title: Extract HDCD
Post by: SiriusB on 2007-08-31 12:14:07
Quote

(is there any way to get hdcd.exe to generate a log file?  I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)


Just redirect stderr to a file.  Using the newer version, it should be as simple as,

hdcd.exe -o out.wav in.wav 2>mylog.txt

and with older version that only supports stdin / stdout,

hdcd.exe < in.wav > out.wav 2>mylog.txt


Also, take a look at the slimdevices thread, that's got some guidance on how to use this in a script,

http://forums.slimdevices.com/showthread.p...967&page=24 (http://forums.slimdevices.com/showthread.php?t=32967&page=24)


I did a quick scan of my library and was interested to find that all my Mark Knopfler CDs are HDCD, despite not being marked as such, as was *one* track from a Bond themes compilation CD.  I'm guessing the compiler probably didn't realise either!!!


Thanks for addressing all the points in my (long) post, Christopher!

Regarding logs/scanning, before reading your reply I did some research and managed on my own to write two batch files that do the job for me...all of my files are .flacs, so one batch file decodes the first second of track 01 of each album to .wav, the second batch file runs hdcd.exe -a (i.e., .wav output suppressed) on those 1-sec wavs and directs the text output and stderr to a log file. Then I go back and delete all the tiny .wav files.  Pretty kludgy but it worked fine! 

Doing that I discovered some interesting things in my collection too...some things I'd either forgotten were HDCD, or never realized in the first place! (marked with asterisks).  I've always thought the J. McLaughlin and J. Cash CDs sounded way loud, and maybe now I know why.    Also interesting to find that some, but not all, King Crimson discs do employ peak extension.

(And because of your Bond result, I'm thinking maybe I need to do more than just 'sample' each album!)


Code: [Select]
Beck    1999 - Midnite Vultures    Peak extend   : Not enabled  
Beck    2002 - Sea Change    Peak extend   : Not enabled  
Grateful Dead    1968 - Anthem of the Sun    Peak extend   : Not enabled  
*John McLaughlin    1970 - My Goal's Beyond    Peak extend   : Enabled permanently  
*Johnny Cash    1999 - 16 Biggest Hits    Peak extend   : Enabled permanently  
Joni Mitchell    1974 - Court And Spark    Peak extend   : Enabled permanently  
Joni Mitchell    1976 - Hejira    Peak extend   : Enabled permanently  
Joni Mitchell    1979 - Mingus    Peak extend   : Enabled permanently  
King Crimson    1970.1 - In the Wake of Poseidon    Peak extend   : Enabled permanently  
King Crimson    1970.2 - Lizard    Peak extend   : Enabled permanently  
King Crimson    1971 - Islands    Peak extend   : Enabled permanently  
King Crimson    1972 - Earthbound    Peak extend   : Not enabled  
King Crimson    1973 - Larks' Tongues In Aspic    Peak extend   : Not enabled  
King Crimson    1974.1 - Starless And Bible Black    Peak extend   : Not enabled  
King Crimson    1974.2 - Red    Peak extend   : Not enabled  
King Crimson    1981 - Discipline    Peak extend   : Not enabled  
King Crimson    1982 - Beat    Peak extend   : Not enabled  
King Crimson    1984 - Three Of A Perfect Pair    Peak extend   : Not enabled  
Mahavishnu Orchestra    1972 - The Lost Trident Sessions    Peak extend   : Enabled permanently  
*McDonald & Giles    1969 - McDonald & Giles    Peak extend   : Not enabled  
Roxy Music    1972 - Roxy Music    Peak extend   : Not enabled  
Roxy Music    1973 - For Your Pleasure    Peak extend   : Not enabled  
Roxy Music    1976 - Viva!    Peak extend   : Not enabled  
The B-52's    1998 - Time Capsule    Peak extend   : Not enabled  
The Flaming Lips    1999 - The Soft Bulletin    Peak extend   : Not enabled  
Van Halen    1983 - 1984    Peak extend   : Not enabled  
*Weather Report    2002 - Live & Unreleased    Peak extend   : Enabled permanently  
Yes    1973 - Yessongs (JPN)     Peak extend   : Enabled permanently
Title: Extract HDCD
Post by: Christopher Key on 2007-08-31 12:50:50
Regarding logs/scanning, before reading your reply I did some research and managed on my own to write two batch files that do the job for me...all of my files are .flacs, so one batch file decodes the first second of track 01 of each album to .wav, the second batch file runs hdcd.exe -a (i.e., .wav output suppressed) on those 1-sec wavs and directs the text output and stderr to a log file. Then I go back and delete all the tiny .wav files.  Pretty kludgy but it worked fine!


You can pipe input to hdcd.exe if you want to avoid intermediate files.  Something like,

flac [options required for wav to stdout] file.flac | hdcd.exe -a 2>>mylog.txt

should work.  It's probably worth using -a rather than -i, as at least one hdcd track didn't have any hdcd information in the first second.  I'll modify the code to default to ten seconds, but allow this to be changed from the command line, and will also have it exit as soon as it detects a valid code.

Regards,

Chris
Title: Extract HDCD
Post by: SiriusB on 2007-08-31 13:20:22
Yes, piping makes more sense than the way I did it.

I too noticed that 1 second was cutting it too close -- as a value for the flac decoder, and as a result hdcd.exe -a missed a few known HDCD tracks in my collection (e.g., part one of 'Ommadawn').  When I changed the flac value to decode 5 sec, hdcd.exe -a  detects ones it missed before.  Do you think ten seconds will add still more?
Title: Extract HDCD
Post by: Egor on 2007-08-31 14:38:47
FLAC --best compression results, 1.2.0 ICL compile

Brian Hughes - Along The Way (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Not enabled
  Minimum gain  : 0.0dB
  Maximum gain  : 0.0dB

Brian Hughes - Along The Way.wav: wrote 367162927 bytes, ratio=0,688
Brian Hughes - Along The Way 24.wav: wrote 368801906 bytes, ratio=0,461
(0.45% size increase)


Keiko Matsui - Dream Walk (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Enabled permanently
  Minimum gain  : -4.0dB
  Maximum gain  : 0.0dB

Keiko Matsui - Dream Walk.wav: wrote 281913513 bytes, ratio=0,585
Keiko Matsui - Dream Walk 24.wav: wrote 308084696 bytes, ratio=0,426
(9.28% size increase)
Title: Extract HDCD
Post by: Christopher Key on 2007-08-31 17:35:50
I'll try and get a copy of those that have the unknown codes.  The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes.  This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing.  If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.


I've looked again more closely at exactly what wmp does.  It does in fact extract and store the third bit from the hdcd codes, but then does absolutely nothing with it.  I'm hence guessing that this single bit is used to switch between two alternate filters, which seems to fit in pretty closely with the last paragraph of,

http://www.hydrogenaudio.org/forums/index....mp;#entry333032 (http://www.hydrogenaudio.org/forums/index.php?showtopic=37653&pid=333032&mode=threaded&show=&st=0&#entry333032)


On the other hand, the AES paper,

http://www.paulita.lt/HDCD_AES_Paper.pdf (http://www.paulita.lt/HDCD_AES_Paper.pdf)

does refer to 'several' filters in a few places, and,

http://www.ednasia.com/article-511-destina...rtion-Asia.html (http://www.ednasia.com/article-511-destinationdistortion-Asia.html)

refers to four filters.

I'm not sure quite what the truth is, but am inclined to think there are only two filters, given that over the sample of tested CDs, the only currently unused bit that was ever set is the one extracted by wmp.


I'd be quite keen to add an option to do the correct upsampling as well.  I've managed to extract a little bit of information; the AES paper specifically says that its a symmetric FIR filter, and another link that I've lost for now claims that it has less than 256 taps.

If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs.  With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.

I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

Regards,

Chris
Title: Extract HDCD
Post by: Fool_on_the_hill on 2007-08-31 21:19:25
I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac

18:05:31 : Test started.
18:09:54 : 01/01  50.0%
18:10:15 : 02/02  25.0%
18:10:40 : 03/03  12.5%
18:11:13 : 04/04  6.3%
18:11:49 : 05/05  3.1%
18:12:34 : 06/06  1.6%
18:13:49 : 07/07  0.8%
18:14:28 : 08/08  0.4%
18:15:12 : 09/09  0.2%
18:16:02 : 10/10  0.1%
18:16:15 : 11/11  0.0%
18:16:19 : Test finished.

----------
Total: 11/11 (0.0%)
Title: Extract HDCD
Post by: Christopher Key on 2007-08-31 21:52:24
I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

Regards,

Chris


Van Halen's 1984 yields some interesting results.  I set up the decoder to output just the value of the third bit from each hdcd code to the left channel, and to leave the right channel untouched.  Attached is a screenshot showing the start of track 3, Panama.

(http://www.srcf.ucam.org/~cjk32/Van%20Halen%20-%201984%20-%2003%20Panama.PNG)

This shows bit3 being turned on for a brief period spanning each drum beat during the intro.  Although much less clear, a similar pattern is visible throughout the rest of the cd.  bit3 is also always turned on for a brief period covering the very start of each track.

My guess is that this is exactly as described in the last paragraph of,

http://www.hydrogenaudio.org/forums/index....mp;#entry333032 (http://www.hydrogenaudio.org/forums/index.php?showtopic=37653&pid=333032&mode=threaded&show=&st=0&#entry333032)

and that setting bit3 is supposed to switch the decoder from the standard, sharp recontruction filter to one with a more gentle roll off that better preserves transients.

Does this seem to make sense?

Chris

Edit:  Just done the same experiment with Joni Mitchell - Both Sides Now, and this seems to be quite clearly what's going on.  It's not used at all in tracks 4, 6, 12, which seem to feature much softer drum beats, and perfectly follows the percussion at about 2:22 into the first track.
Title: Extract HDCD
Post by: Christopher Key on 2007-08-31 22:40:37
FLAC --best compression results, 1.2.0 ICL compile

Brian Hughes - Along The Way (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Not enabled
  Minimum gain  : 0.0dB
  Maximum gain  : 0.0dB

Brian Hughes - Along The Way.wav: wrote 367162927 bytes, ratio=0,688
Brian Hughes - Along The Way 24.wav: wrote 368801906 bytes, ratio=0,461
(0.45% size increase)


Keiko Matsui - Dream Walk (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Enabled permanently
  Minimum gain  : -4.0dB
  Maximum gain  : 0.0dB

Keiko Matsui - Dream Walk.wav: wrote 281913513 bytes, ratio=0,585
Keiko Matsui - Dream Walk 24.wav: wrote 308084696 bytes, ratio=0,426
(9.28% size increase)


Thanks for those results and to everyone else who's posted.  They do seem to show that the lossless codecs are doing their job well, and getting rid of most of the redundancy in the files.  The decoded files that compress to a smaller size are particularly interesting, and as Sebastian points out, are much closer to 'real' music.

Regards,

Chris
Title: Extract HDCD
Post by: Triza on 2007-09-01 00:34:09
Excellent work. Christopher. Well done.

Any chance for having the source released?

Triza
Title: Extract HDCD
Post by: SiriusB on 2007-09-01 00:36:20
Quote
If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs.  With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.


I have captured the analog output of a hardware HDCD decoder digitally if that's what you mean.  I know of no consumer hardware decoder that digitally outputs the decoded stream.
Title: Extract HDCD
Post by: Dynamic on 2007-09-01 01:13:05
I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac


Successfully ABXed but what was the 16-bit file? Was it the undecoded file straight from the CD, or was it the decoded file (20 bits of more) converted back to 16-bit with flat dither?

I'm interested in the latter, because I doubt it's ABXable without going insanely loud on the volume control.
Title: Extract HDCD
Post by: bryant on 2007-09-01 05:49:31

I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Here's a possible explanation: The peak compression is heavily non-linear. It creates harmonic distortions which could be harder to predict than the original or decoded data.

Cheers!
SG

Hi Sebastian! 

Actually, I think it's a lot simpler than that. At least on the tracks I tried, the peak extension is triggered very infrequently, so most of the time you have only 16 bits and WavPack can encode that as 16 bits. But when the peak extension is triggered in a block, that entire block must be encoded in 20 bits (or whatever the worst sample is, bits-wise). When you're using the rather large default blocks (0.5 - 1.0 seconds) then a high percentage of the blocks have to go 20 bit, but when using smaller blocks (0.05 - 0.10 seconds) most can be just 16 bit.

David
Title: Extract HDCD
Post by: bryant on 2007-09-01 06:14:48

Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations! 


Ouch, trying to do it with only an analogue reference was going to be seriously hard!

I'd be very interested to know how you went about deriving the tap values used for the LFSR.  There's a breif description of how I went about it earlier in this thread, but I can't believe that was the easiest way.

I had read the patent and the AES paper, so I knew that it was [probably] based on the modulo-1 sum of 3 taps from a delay line of the audio MSB. So I wrote a simple program to scan entire tracks trying all possible combinations of [reasonably short] tap delays and then analyzing the resulting bitstream for patterns that repeat far more often than chance would predict. When I put in the correct taps, a 48-bit pattern jumped out!

However, as I mentioned, I never could find anything else. The 48-bit pattern was always the same on every HDCD track I tried, and the bits on either side (and everywhere else I looked) were always perfectly random. I knew I had found the right thing because if I changed a single bit of the 48 then my converter wouldn't light the HDCD indicator and if I changed other bits it still would (and I had to burn CDs for every experiment because I didn't have spdif output!)

My suspicion at the time was that there was some fixed dynamic range expansion implemented and that it was either on or off. Either I missed something, or the CD tracks I tried all happened to just use the same simple scheme.

I'll be interested to eventually see where my error was... 

David

edit: added details
Title: Extract HDCD
Post by: Christopher Key on 2007-09-01 16:09:38
Quote

If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs.  With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.


I have captured the analog output of a hardware HDCD decoder digitally if that's what you mean.  I know of no consumer hardware decoder that digitally outputs the decoded stream.


I believe that there are HDCD decoding chips that will output 88.2kHz 24 bit PCM.  With any consumer device based upon one of these, it should be physically possible to tap into this signal.  I doubt it's doable without specialised equipment in practice though.

If you've access to an decoder with an analogue out, it might be possible to apporiximate the correct behaviour.  I'd propose creating a test file with a slow 10kHz - 22.5kHz sweep, an impulse train, and a low frequency square wave.  This file would then be duplicated, and each version would have HDCD codes added such that one version played back through the normal upsampling filter, and the other, through the 'transient' filter.

By then looking at the captured output from the two filters, it should be possible to have a good at reproducing them.

I'm away all of next week, but I'll see if I can have a go a creating them when I get back.

Regards,

Chris


Any chance for having the source released?


I do intend to release the source code when possible.  At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Regards,

Chris
Title: Extract HDCD
Post by: skamp on 2007-09-01 18:50:14
I do intend to release the source code when possible.  At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Isn't it because LAME is distributed only as source code, that it doesn't require paying royalties?
Title: Extract HDCD
Post by: Fool_on_the_hill on 2007-09-01 20:32:41
Quote
Dynamic     Posted Yesterday, 18:13
     
QUOTE(Fool_on_the_hill @ Aug 31 2007, 21:19) *

I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac

Successfully ABXed but what was the 16-bit file? Was it the undecoded file straight from the CD, or was it the decoded file (20 bits of more) converted back to 16-bit with flat dither?

I'm interested in the latter, because I doubt it's ABXable without going insanely loud on the volume control.


Compared: Grabbed with EAC 16-bit track VS this track converted to 24-bit using hdcd.exe. Both files were converted to FLAC and Replay Gain (Track Gain) was applied. ABXed with Headphones in quiet part in the middle of the song.
Title: Extract HDCD
Post by: Dynamic on 2007-09-02 18:37:13
Thanks for the info, FotH. I hadn't noticed that Wag The Dog was HDCD, and that's an enjoyable track, so I might give it a go myself one day when I've got all my gear assembled again. I'd fancy comparing the 24-bit decoded output to the same output converted to 16-bit (flat dither) by fb2k, and perhaps converted to 14-bit padded (flat dither) also.
Title: Extract HDCD
Post by: SebastianG on 2007-09-03 12:27:45
My guess is that (...) setting bit3 is supposed to switch the decoder from the standard, sharp recontruction filter to one with a more gentle roll off that better preserves transients.

Does this seem to make sense?

Perfectly. I guess this 'transient bit' can be ignored then.

Cheers!
SG
Title: Extract HDCD
Post by: gabeg on 2007-09-04 17:31:09
Perfectly. I guess this 'transient bit' can be ignored then.

Cheers!
SG



So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?
Title: Extract HDCD
Post by: SebastianG on 2007-09-05 08:29:26
So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Although I don't know the exact specs of the reconstruction filters I feel I can safely say that decoding this (non-)feature isn't worth the trouble.

Cheers!
SG
Title: Extract HDCD
Post by: Cavaille on 2007-09-06 18:55:36
for heavens sake!!! it´s finally happening... i´ve waited so long for this.

a year ago or so, i posted a wish here for someone to program a little tool, that would be capable of decoding hdcd´s. everyone except one was saying: why? it´s microsoft, it´s pointless, you can´t hear it... etc. etc.

sadly, i´m lacking the skills of programming something, so that i would be able to do it myself... but i can´t.

my statement always was, that a hdcd only sounds the way the artist or the engineer intended, when decoded.

with the help of the chronotron plug-in i found out, that a hdcd really CAN be 20 bit - at least, wavelab is showing this to me. i still don´t know how this works (i never completeley understood all posts here in this thread), but here comes a picture:

(http://img526.imageshack.us/img526/8651/untitled1vj8yt6.jpg)

i opened one not-decoded wave-file & the decoded one. i reduced the volume of the not-decoded file about 6dB. i pressed play, then i simply made two screenshots when the play-marker reached a transient and merged them together with photoshop. you can see the peak extend (white) and the changing bit-depth in the bit-meter (dark pink). when transients are emerging, the resolution jumps from 17 to 20 bit. i don´t know, if this information is new to someone...
Title: Extract HDCD
Post by: Christopher Key on 2007-09-13 13:19:32
I had read the patent and the AES paper, so I knew that it was [probably] based on the modulo-1 sum of 3 taps from a delay line of the audio MSB. So I wrote a simple program to scan entire tracks trying all possible combinations of [reasonably short] tap delays and then analyzing the resulting bitstream for patterns that repeat far more often than chance would predict. When I put in the correct taps, a 48-bit pattern jumped out!

However, as I mentioned, I never could find anything else. The 48-bit pattern was always the same on every HDCD track I tried, and the bits on either side (and everywhere else I looked) were always perfectly random. I knew I had found the right thing because if I changed a single bit of the 48 then my converter wouldn't light the HDCD indicator and if I changed other bits it still would (and I had to burn CDs for every experiment because I didn't have spdif output!)

My suspicion at the time was that there was some fixed dynamic range expansion implemented and that it was either on or off. Either I missed something, or the CD tracks I tried all happened to just use the same simple scheme.

I'll be interested to eventually see where my error was... 

David

edit: added details


Sounds like you got to exactly the same point I did, albeit by a rather simpler approach.  I too found that I has exactly the same repeating through all my 4 test cds, with about a very few occurences of a slight variation of the pattern on one cd.  Fortunately, having wmp as a reference, it was possible to deduce what the patterns meant.

The 48bit patten is made from a 32bit marker, an 8 bit code and an 8 bit checksum.  (Actually, it seems to be a 31bit marker, following by 1 bit that determines the format of the code and whether there is a checksum).  The pattern I was seeing was the code for disabling the transient filter, enabling peak extend, and not performing any gain adjustment (0x10).  The few slight variations were use of the gain adjustment feature.

Regards,

Chris
Title: Extract HDCD
Post by: Christopher Key on 2007-09-13 14:02:16
I do intend to release the source code when possible.  At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Isn't it because LAME is distributed only as source code, that it doesn't require paying royalties?


Thanks,

I wasn't aware of that.  I'll have a chat with the LAME guys and see if their argument has been tested / accepted.

Regards,

Chris
Title: Extract HDCD
Post by: Christopher Key on 2007-09-13 14:13:48

So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Although I don't know the exact specs of the reconstruction filters I feel I can safely say that decoding this (non-)feature isn't worth the trouble.


Indeed, it probably isn't worth the bother of decoding, but it would be nice to know.  I've created four wav files, an impulse train and a 10kHz -> 22.05kHz sweep, each set to use both the normal and transient filter:

http://www.srcf.ucam.org/~cjk32/hdcd/transient_filter/ (http://www.srcf.ucam.org/~cjk32/hdcd/transient_filter/)

Sirius, do you think you could pipe these four through your hardware decoder and capture the output.  I'd strongly recommend against feeding the decoded signal into anything other than a capture device, as I doubt speakers (amp too???) would fare very well with them!

Regards,

Chris
Title: Extract HDCD
Post by: MRC01 on 2007-10-18 06:14:54
I'm not sure how this works exactly.  There are 3 bits left that could be used for controlling the reconstruction filters, although there were always zero in all my test cds, and wmp didn't seem to respond to them at all.

I just downloaded your HDCD.EXE and gave it a whirl. Works great - THANKS. I think it's MUCH nicer to have all that stuff decoded to play in a non-HDCD player - a lower average signal level is a small price to pay for decoding (expanding) the dynamic compression, and most recordings don't use the full 90+ dB that redbook CD offers anyway.

On a couple of my HDCD recordings it said "Unknown codes seen: 0x03" These CDs are from Reference Recordings which apparently employs Johnson, one of the coinventors of HDCD.

Here's a link to a sample WAV file. I picked the shortest - only 5 MB in size - because my FTP server has limited bandwidth.
ftp://mclements.net/HDCDTest1.wav (http://ftp://mclements.net/HDCDTest1.wav)
Title: Extract HDCD
Post by: ProtectYaNeck36 on 2007-10-20 17:41:53
Quote
I was always under the impression that HDCD was simply a better dithering algorithm.
[a href="index.php?act=findpost&pid=268929"][{POST_SNAPBACK}][/a]


HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG


I just came across this post and I had a question for.  #2 threw me off when you said "reversible dynamic compressor."  are the signal peaks actually compressed (or limited?) or is it a compressor acting as an expander (the "reversible" part is what lead me to believe you were describing a compressor reversed)?
Title: Extract HDCD
Post by: krabapple on 2007-10-24 17:09:16
I just came across this post and I had a question for.  #2 threw me off when you said "reversible dynamic compressor."  are the signal peaks actually compressed (or limited?) or is it a compressor acting as an expander (the "reversible" part is what lead me to believe you were describing a compressor reversed)?



Compression during encoding (in the studio), expansion during decoding  (in the player) -- it's enabled by the 'peak extension' option available to the mastering engineer.
Title: Extract HDCD
Post by: MRC01 on 2007-10-30 15:16:03
are the signal peaks actually compressed (or limited?)

When playing an HDCD encoded CD on a normal (non-HDCD) player, yes - you will usually get dynamic compression (if the HDCD encoding used this feature).

The $64k question is: why? They're not using the full 90+ dB of dynamic range that CD offers, and even after expanding the compression, most recordings still don't use that 90+ dB of dynamic range. In other words, the compression (or "peak extension" as they call it) is pointless because standard redbook CD already offers more dynamic range than they are using.

My cynical side says it's a way to make money by licensing a technology to make recordings "sound better". I use the words "sound better" loosely, since what they're really doing is making the redbook CD version sound worse so when played through the licensed HDCD decoder it sounds better. But ultimately, when played on an HDCD decoder, it should sound the same as it would have sounded if they had never used HDCD or dynamic compression at all.
Title: Extract HDCD
Post by: Wombat on 2008-02-07 19:05:45
This tool is pretty interesting. I haven´t much HDs but one of them "Yim Hok-Man - Poems Of Thunder, The Master Chinese Percussionist" i bought for 7.95 at Amazon is a HD showing no sign of it on the box. Only a small info in the booklet about mastering equipment using it.
In the first song are some strong Kudo drumhits that are real distorted.
Decoding it with the peak extension found makes these drumhits absolut clear!
Looking in an editor shows the non decoded one as totally smashed.
I replagained both and did an abx in foobar with no problem.
Title: Extract HDCD
Post by: skamp on 2008-03-04 04:51:25
I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

These are the results I get for the Joni Mitchell "Blue" HDCD, with Monkey's Audio 3.99-u4-b5, wavpack 4.41.0, flac 1.2.1 (all under linux):

16-bit WAVs (not HDCD decoded)
Code: [Select]
176.9 MiB    mac -c5000 (Monkey's Audio insane)
185.4 MiB    wavpack -hhx6
191.1 MiB    flac --best

As expected, here Monkey's Audio compresses much better than WavPack, which itself is substantially better than FLAC.

24-bit WAVs (HDCD decoded), default blocksizes
Code: [Select]
338.9 MiB    mac -c5000 (Monkey's Audio insane)
210.6 MiB    wavpack -hhx6
198.8 MiB    flac --best (default blocksize: 4096)

I thought there was something wrong with my setup, until I saw Walrusbonzo's results (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=30999&view=findpost&p=512838), confirming mine. I can't believe how bad Monkey's Audio fared.

24-bit WAVs (HDCD decoded), custom blocksizes
Code: [Select]
196.6 MiB    wavpack -hhx6 --blocksize=4410
197.1 MiB    wavpack -hhx6 --blocksize=2205
196.1 MiB    mix of the above (smallest encodes from each)
197.1 MiB    flac -l 12 -b 2048 -m -e -r 6 (equivalent to --best)

Now we're talking! Overall, wavpack with a blocksize of 4410 fares better than with a blocksize of 2205, although some tracks (4, 6, and 7) are indeed a bit smaller with the latter.
Note: Monkey's Audio doesn't provide the ability to change blocksizes.
Title: Extract HDCD
Post by: foo_lover on 2008-03-25 07:29:00
I was thinking of having HDCD decoder as foobar2000 plugin and came to conclusion that it won't fit in player's model... It would break replaygain, for example. So, the only reasonable implementation of HDCD decoder would be as a part of player's core.. IMHO, of course.
Title: Extract HDCD
Post by: kremb on 2008-05-24 07:24:08
Here is what I did. I'm not sure if I notice any appreciable difference in quality but what the heck.

EAC to .wav files

hdcd.exe .wav files to 24 bit .wav files

import 24 bit .wav files into itunes at 320kbps AAC

Burn itunes playlist to CD


*****

here are the size of the various wave files I burned to CD

original EAC Rip - 23 870 KB - lame mp3 via foobar - 3816 KB
24 bit .wav via hdcd.exe - 35 804 KB - lame mp3 via foobar - 3754 KB
.wav from itunes burned cd via EAC - 23 870 KB - lame mp3 via foobar - 3704 KB
.wav from nero burning 24bit .wav to cd - 23 870 KB - lame mp3 via foobar - 3762 KB

*****

I think I have answer my own question, I didn't think the AAC was going to be so lossy. Looks like simply burning the 24 bit wav files using nero audio selection is the best choice if a person is determined to try and include all 20 bits of data in their NON-hdcd CD backup.

*****

Feel free to tell me where my logic is flawed and should just settle with the backwards compatible hdcd playback.

Does anyone have another suggestion to try and use the most of the 20 bits of data on a regular CD? Is this a job for soundforge? Or into some sort of DVD format that will decode ALL 20 bits?

*****

I've decided to author a dvd with still frame photos for video and the 24 bit wav files for the audio. I'm using pretty simple authoring software called TMPGEnc DVD Author. I think it's going to work out pretty well, other than I'll have to play the tracks on a DVD player.

Hope my long winded process helps someone else.

Enjoy your tunes!

*****

easier said than done. dvd author program insists on reencoding some black box method to AC3.

Thus ....

I decided to compress to 24 bit flac and listen to them via my Ipod using Rockbox. I THINK i can notice a difference with my cans on vs a good little cd player I have. We'll have to see how much of an audiophile I really am as I only have 2GB on my ipod.
Title: Extract HDCD
Post by: Nubben on 2008-06-04 10:44:33
dBpoweramp R13 (just released) now contains an option to rip these into 24-bit wav/aiff files.

Nubben
Title: Extract HDCD
Post by: odyssey on 2008-06-04 11:52:01
I was thinking of having HDCD decoder as foobar2000 plugin and came to conclusion that it won't fit in player's model... It would break replaygain, for example. So, the only reasonable implementation of HDCD decoder would be as a part of player's core.. IMHO, of course.

Did you see the latest changelog? They implemented some replaygain stuff for 3rd party components. What exactly it is or how it works, i have no idea about, but maybe it might come in handy?
Title: Extract HDCD
Post by: Walrusbonzo on 2009-02-14 12:57:18
POST RESURRECTION! 

Any news on a HDCD decoding plug-in for Foobar?
Title: Extract HDCD
Post by: Tropican on 2009-03-09 00:11:24
Hey, does anyone know anything about the new "0.2" version of HDCD.exe? http://www.srcf.ucam.org/~cjk32/hdcd/ (http://www.srcf.ucam.org/~cjk32/hdcd/)

It's been there since November. I can't find any indication of what's been changed/improved.

The fact that http://www.srcf.ucam.org/~cjk32/hdcd/hdcd.zip (http://www.srcf.ucam.org/~cjk32/hdcd/hdcd.zip) points to this new build makes me think its stable?
Title: Extract HDCD
Post by: markanini on 2009-05-03 02:24:32
The 0.2 dev build contains API which should make it easier to create a foobar2000 plugin...
Title: Extract HDCD
Post by: TheodoreG on 2009-07-15 18:35:02
I am trying to capture the decoded HDCD audio from Windows Media Player using Total Recorder Professional 7.1.  I can only get Total Recorder to recognize 16-bit, not 24-bit formats played from Windows Media Player.  This is using Windows XP SP3.  Therefore I cannot seem to obtain 24-bit output from HDCD's, even with 24-bit CD audio enabled and the HDCD logo displayed.

I created in Adobe Audition some Windows Media Audio files in 24-bit format.  When I play them in WinAmp through Total Recorder's driver, Total Recorder correctly recognizes the audio format (44.1 kHz, 24-bit).  If I play the same Windows Audio-format files in Windows Media Player (version 9 or 11), Total Recorder identifies the audio format at 44.1 kHz 16-bit (not 24-bit).

Similarly, when I play an HDCD in WMP (version 9 or 11), Total Recorder recognizes 44.1 kHz, 16-bit format regardless of whether 24-bit CD playback is selected for the Speakers device in WMP.

Note: Total Recorder does support 24-bit recording and playback in its current version, as I mentioned in the example above involving 24-bit playback from Winamp.  In fact, I use Total Recorder to record live music when I am doing sound in 96 kHz / 24-bit format.

Note 2: I am aware of the utility hdcd.exe version 0.2.  But I would like to do a reality check using a few physical CDs that are encoded with HDCD, playing through Windows Media Player.
Title: Extract HDCD
Post by: Cavaille on 2009-07-15 20:34:01
I am trying to capture the decoded HDCD audio from Windows Media Player using Total Recorder Professional 7.1.  I can only get Total Recorder to recognize 16-bit, not 24-bit formats played from Windows Media Player.  This is using Windows XP SP3.  Therefore I cannot seem to obtain 24-bit output from HDCD's, even with 24-bit CD audio enabled and the HDCD logo displayed.

I created in Adobe Audition some Windows Media Audio files in 24-bit format.  When I play them in WinAmp through Total Recorder's driver, Total Recorder correctly recognizes the audio format (44.1 kHz, 24-bit).  If I play the same Windows Audio-format files in Windows Media Player (version 9 or 11), Total Recorder identifies the audio format at 44.1 kHz 16-bit (not 24-bit).
Why so complicated? Just use a WAV-output Plug-in (http://www.chronotron.com/content.php?page=downloads). With that you´ll be able to extract HDCD to extensible .wav. A tip: create an image of the disc with EAC first and mount that image as a virtual drive - that way you can go around the crappy grabbing engine in WMP. WMP will still recognize the image as a CD (and therefore as a HDCD) if it is mounted as such. But note: if you activate the 24-bit-disc playback, WMP will also change non-HDCDs (volume will be reduced for 6 dB and the bit resolution will be 17 Bit).

Even less complicated is the HDCD decoder in dbPowerAmp. You can use that one with the already extracted wave-files.
Title: Extract HDCD
Post by: krabapple on 2009-07-15 21:04:27
.

Even less complicated is the HDCD decoder in dbPowerAmp. You can use that one with the already extracted wave-files.



dbpoweramp's decoder is simply hdcd.exe, I think.
Title: Extract HDCD
Post by: greynol on 2009-07-15 21:10:24
http://www.hydrogenaudio.org/forums/index....st&p=571890 (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=63674&view=findpost&p=571890)
Title: Extract HDCD
Post by: TheodoreG on 2009-07-16 03:34:49
Why so complicated? Just use a WAV-output Plug-in (http://www.chronotron.com/content.php?page=downloads). With that you´ll be able to extract HDCD to extensible .wav.



I did not try the plugin because I was doing a reality check of the playback of 24-bit audio -- which would mean that Windows Media Player playing 24-bit files or CDs in 24-bit mode should pass 24-bit data to the audio device.  It only passed 16-bit data, hence my concern.  If it does not do that correctly, then I would not necessarily assume any plugins in Windows Media Player are getting true 24-bit data.
Title: Extract HDCD
Post by: Cavaille on 2009-07-16 04:07:23
I did not try the plugin because I was doing a reality check of the playback of 24-bit audio -- which would mean that Windows Media Player playing 24-bit files or CDs in 24-bit mode should pass 24-bit data to the audio device.  It only passed 16-bit data, hence my concern.  If it does not do that correctly, then I would not necessarily assume any plugins in Windows Media Player are getting true 24-bit data.
Oh. Well, for that there could be many reasons. 1. your soundcard isn´t really capable of playing 24 bit - or more precise, its driver does not report 24 bit (although I don´t know if WMP really goes back to 16 bits if the card driver reports so) 2. the kernel mixer could prevent the stream going unaltered from WMP to the card. 3. HDCD isn´t 24 bit. It merely is 17 bits when decoded (and only because the volume is lowered for -6dB) and 'quasi' 20 bit when some of the switches kick in. Maybe the software you use for recording misinterprets this as 16 bit.

Someone may correct me on that but I think that the kernel mixer is your problem here. If my memory serves me right it will use the first audio signal as some kind of reference for every other audio signal that will be played afterwards. Meaning: if you played or recorded a 16 bit signal it will stay that way unless you turn it off. Kernel mixer won´t be switching bit depth automatically during playing or recording them. Other members can certainly clarify this if I made a mistake.

I can tell you from my experience that WMP really is capable of decoding HDCD and there is plenty of evidence for this here in this thread (if I´m not mistaken) and on the web. Why do you want to find it out in the first place?
Title: Extract HDCD
Post by: Weeping Robot on 2009-09-13 22:01:18
Ah, the fun old HDCD question. Was browsing on another forum today and the topic came up, which reminded me of an old, unanswered question I had regarding HDCD decoding. Specifically, from this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=64618&hl=). Money quote: "Any idea if [pregaps actually missing] affects HDCD playback?" From what I understand, the HDCD information is stored in the "LCBs" (least significant bits) and in only a minority of the... samples? I suppose, then, the key question is whether any HDCD "flags" are set in pregaps. My guess is no (are pregaps even audio information, really?), but it would be nice to know for sure.

By the way, something else I was wondering: what if your rips aren't offset correctly (in EAC, drive offset I mean). Does the HDCD "flag" rely on correct alignment (or cue sheets) as a reference? If each sample (or whatever) contains the HDCD information, than I suppose none of that would matter... and that might solve my missing pregaps question too.
Title: Extract HDCD
Post by: greynol on 2009-09-14 20:41:31
There is no reason that pregaps cannot contain HDCD information since pregaps (like any other part of a track) contain nothing more than audio information and subcode data*.  It is my understanding that offsets have no bearing on HDCD decoding either.

(*) subcode data has absolutely no bearing on HDCD decoding.
Title: Extract HDCD
Post by: Axon on 2009-09-14 20:47:29
Is HDCD 100% stateless? If not, then pregaps matter.
Title: Extract HDCD
Post by: greynol on 2009-09-14 20:49:34
Suppose I press skip half-way through the playback of a track, does that mean what I have heard up to that point was not decoded properly?  Of course it doesn't.

Does this mean that the next track will not be decoded properly?  I seriously doubt it.
Title: Extract HDCD
Post by: Axon on 2009-09-14 21:10:56
Suppose I press skip half-way through the playback of a track, does that mean what I have heard up to that point was not decoded properly?  Of course it doesn't.
True, but it's the answer to the wrong question. The question really is "is the playback after the skip always decoded properly?", and the HDCD patent very unambiguously implies the answer to this is "no". There is no reason why the same answer cannot also apply for ripped tracks that lack pregaps.
Title: Extract HDCD
Post by: greynol on 2009-09-14 21:12:29
It should be quite trivial to test and my guess about the outcome is that removal of pregaps will not matter.
Title: Extract HDCD
Post by: Axon on 2009-09-14 21:15:18
Yeah. Somebody who actually cares about HDCD should probably look into this.
Title: Extract HDCD
Post by: greynol on 2009-09-14 21:26:49
I want to add that such a test need not be constrained to the removal of pregaps.  XX frames of data could be removed from the middle of a track instead.  My point is to drive home the fact that audio data marked off as a pregap is no different than audio data that isn't marked off as a pregap.
Title: Extract HDCD
Post by: user on 2009-09-16 12:24:16
The answer is satisfying,
don't worry, be happy


Pregaps don't matter, offsets don't matter, and directly skipping into a track does not matter also, all for positive HDCD decode.

I tell this by practical experience with HDCD,

I transmit normally flac song track files via optical digital connection into a Harman kardon receiver/amp, which has the ability to deal with eg. up to 24/96 kHz PCM, and HDCD decode for this topic.

or a HDCD playing from CD/DVD-player digitally connected to that amp.

With both HDCD sources it does not matter, you can directly skip (either by foobar's time slider or by fast forwarding eg. with the CD/DVD-player) into the mid of a track, and after very short time the Harman Kardon amp/decoder detects it as HDCD and shows the HDCD symbol.

If somebody could prove the opposite eg. by the test of cutting an existing HDCD flac/wave file, and the cut part of this wave would not show HDCD anymore, I would be surprised.
Title: Extract HDCD
Post by: krabapple on 2009-09-16 18:01:31
OK, I did the experiment.

ran hdcd.exe -a on this undecoded track:
03 - Mixed Bizness.wav (from "Midnite Vultures" by Beck)
results:
HDCD detected
Packets Processed  3345
Peak extend never enabled
Min gain  0.0dB
Max gain 0.0dB
Transient filter enabled intermittently


Copied and saved a 1-minute excerpt from the middle, using Audition ('snap to zero crossings' enabled); ran hdcd.exe -a on it
03 - Mixed Bizness_excerpt.wav
results:
HDCD detected
Packets Processed  860
Peak extend never enabled
Min gain  0.0dB
Max gain 0.0dB
Transient filter enabled intermittently


Q.E.D.
Title: Extract HDCD
Post by: Aquares on 2009-11-22 23:25:28
This tool is pretty interesting. I haven´t much HDs but one of them "Yim Hok-Man - Poems Of Thunder, The Master Chinese Percussionist" i bought for 7.95 at Amazon is a HD showing no sign of it on the box. Only a small info in the booklet about mastering equipment using it.
In the first song are some strong Kudo drumhits that are real distorted.
Decoding it with the peak extension found makes these drumhits absolut clear!
Looking in an editor shows the non decoded one as totally smashed.
I replagained both and did an abx in foobar with no problem.

Same e.g. with "Rupa & the April Fishes - Extraordinary Rendition". The album is rather dynamic and not very loud anyway, but there is some clipping that is reduced audibly and measurably after decoding.

I measured with ClippingAnalyzer (http://www.ber-sd.com/dl_clipping_eng.html) to get some hard facts:

Track: Une Americaine à Paris

Normal:
RMS (after normalization), L/R:  -13.2 / -12.2
No. of occurrencies, L/R:  97 / 270
Ø clipped samples per sec:  4.0    
No. of samples in longest clipping sequence:  21

HDCD Decoded:
RMS (after normalization), L/R:  -15.9 / -14.9
No. of occurrencies, L/R:  0 / 11   
Ø clipped samples per sec:  0.1
No. of samples in longest clipping sequence:  6


Title: Extract HDCD
Post by: Gumboot on 2011-12-11 22:41:43
So I've been looking at hacking code to be bit exact with hdcd.exe (since I don't want to mess around with WMP plugins), and in that process I have to make sure I know what it's doing.  To put aside any doubt, I made up a test file which extracts the envelope function by encoding a DC signal and injecting various HDCD codes to move the gain around.  [Can I attach that?  I don't see a button.]

When gain changes, it steps in multiples of 1/4096.  This quantisation is being multiplied into the audio signal as the gain changes.  If this is the complement of the gain function applied by the encoder then there's no problem, except for the non-HDCD listeners who would have that error function multiplied into their version with no correction.

What worries me is the behaviour of HDCD.EXE (and presumably WMP too).  The point at which these quantised steps are taken depends on the starting value of the gain.  For example, a ramp from -0dB to -4.5dB proceeds through a fixed set of gain values for each audio sample, but a ramp from -0.5dB to -4.5dB is not the obvious subset of those.  I don't know for sure if this behaviour is bit-exact with WMP, but I'm doubtful that it would be bit-exact with a hardware implementation, because it would cost extra resources to be that erratic.

When there's a mis-step in an implementation's ramp, that's a transient miscalculation of a single sample by a factor of 1/4096, reducing the meaningful resolution in that area to (worst case) 12 bits.  It's a trivial error and likely completely imperceptible, but with all this effort going into lossless storage and high-resolution DACs and post-processing on 16-bit audio to get extra resolution out, it's not really what we want to happen.

So I have to ask: Is HDCD.EXE correct?  Is WMP correct?  Is the PMD-100 correct?

My inclination is to assume that the ramp should be perfectly smooth, as that would minimise loss for both HDCD and non-HDCD listeners, and that the quantisation is a numerical simplification for a desktop implementation only.  I don't know for sure, though, and finding out would require more signal analysis than I can be bothered with, or a reference encoder.
Title: Extract HDCD
Post by: Gumboot on 2011-12-12 17:21:01
Lest I be accused of doing my necromancy by half measures...


Ah, the fun old HDCD question. Was browsing on another forum today and the topic came up, which reminded me of an old, unanswered question I had regarding HDCD decoding. Specifically, from this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=64618&hl=). Money quote: "Any idea if [pregaps actually missing] affects HDCD playback?" From what I understand, the HDCD information is stored in the "LCBs" (least significant bits) and in only a minority of the... samples? I suppose, then, the key question is whether any HDCD "flags" are set in pregaps. My guess is no (are pregaps even audio information, really?), but it would be nice to know for sure.

By the way, something else I was wondering: what if your rips aren't offset correctly (in EAC, drive offset I mean). Does the HDCD "flag" rely on correct alignment (or cue sheets) as a reference? If each sample (or whatever) contains the HDCD information, than I suppose none of that would matter... and that might solve my missing pregaps question too.


The HDCD flags are typcially repeated ten times a second.  The bitstream decoder is non-recursive, so to destroy one you have to cut a file somewhere within a 72 sample window.

The filter is not stateless, though.  You can begin playing with an incorrect setting for gain or peak extension.  Changing the state of peak extension is instantaneous, and so if the signal came right in the middle of a peak then you might get a bit of a pop.  The gain is ramped from wherever the decoder thought it was previously, so it'll be smooth(ish), but it can take up to 44 milliseconds to find the right value.


In short, if you cut an HDCD stream, you can expect up to the first 145ms to be less than perfect, but not significantly less than perfect.