Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Extract HDCD (Read 209096 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Extract HDCD

Reply #50
Interesting stuff, I have 20~ HDCDs around here.

Going to have a play

EDIT:  How do you use this program, when I try and run it in the command prompt it just sits and does nothing.  I tried specifying the wav file name and it still does nothing.  What is the correct syntax?  Even tried drag and drop of the wav, didn't work either


Follow strictly commandline from doom9.org including <, > and spaces.

Extract HDCD

Reply #51
hdcd.exe < 16bit.wav > 24bit.wav

Extract HDCD

Reply #52
Thanks guys....  Got it working.

Extract HDCD

Reply #53
I've started to run all my known HDCD encoded CDs through the HDCD decoder and I'm then compressing using WavPack 4.42 (-hhx6).  I'll post more results once I get them.

WavPack compression results.

Buffalo Springfield, Last Time Around.


Original 16bit - 197mb - 806kbps
HDCD Decoded 24bit - 248mb - 1013kbps

Joni Mitchell, Clouds.


Original 16bit - 202mb - 749kbps
HDCD Decoded 24bit - 222mb - 827kbps

Joni Mitchell, Blue.


Original 16bit - 187mb - 724kbps
HDCD Decoded 24bit - 210mb - 811kbps

Joni Mitchell, Ladies of the Canyon.

Original 16bit - 259mb - 806kbps
HDCD Decoded 24bit - 282mb - 878kbps

Joni Mitchell, Song to a Seagull.


Original 16bit - 210mb - 771kbps
HDCD Decoded 24bit - 229mb - 841kbps

Van Halen, 1984.


Original 16bit - 246mb - 1031kbps
HDCD Decoded 24bit - 244mb - 1022kbps

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

Van Halen, Van Halen I.


Original 16bit - 260mb - 1025kbps
HDCD Decoded 24bit - 258mb - 1018kbps

Same again here for Van Halen.

Neil Young, American Stars 'n Bars.

Original 16bit - 240mb - 879kbps
HDCD Decoded 24bit - 295mb - 1082kbps

Neil Young, On The Beach.


Original 16bit - 216mb - 763kbps
HDCD Decoded 24bit - 265mb - 936kbps

King Crimson, Islands.


Original 16bit - 208mb - 640kbps
HDCD Decoded 24bit - 214mb - 680kbps

King Crimson, In The Wake Of Poseidon.

Original 16bit - 222mb - 757kbps
HDCD Decoded 24bit - 233mb - 796kbps

King Crimson, Larks' Tongues In Aspic.

Original 16bit - 241mb - 724kbps
HDCD Decoded 24bit - 251mb - 754kbps

King Crimson, Starless and Bible Black.

Original 16bit - 246mb - 739kbps
HDCD Decoded 24bit - 255mb - 764kbps

Extract HDCD

Reply #54
FLAC seems to support 20 bit audio. I don't suppose there would be much of a compression gain from encoding 20 bit audio instead of 24 bit?

Extract HDCD

Reply #55
FLAC seems to support 20 bit audio. I don't suppose there would be much of a compression gain from encoding 20 bit audio instead of 24 bit?


Would make no odds.  This software HDCD decoder I'm using only outputs 24bit audio.

Extract HDCD

Reply #56
One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

Quote
Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


Cheers!
SG

Extract HDCD

Reply #57
Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!


The point is easy to see.  I want to be able to play these CDs' songs at the highest quality I can with the equipment I have.  I don't have a HDCD decoder and I like to play all my music through my PC via Foobar.  So to me it makes perfect sense.  Don't be so narrow minded.

Quote

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


It could well be the reason.  I might try re-ripping it in case I made a mistake somewhere.

Extract HDCD

Reply #58
Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

I agree, HDCD content should be decoded on playback. But in the real world, I doubt I'll see a HDCD decoder plugin for my player of choice (Audacious) any time soon (and I don't own an audio receiver, let alone a HDCD-compatible one).


Extract HDCD

Reply #60
Can I just say I'm very excited about this tool!

Did a quick ABX on a beach boys HDCD disc, and couldn't tell the decoded one from the standard one, but I'll keep trying... perhaps with a more modern CD.

Wouldn't it be great if foobar was the second player (after WMP) to support decoding of HDCDs!

Ed

Extract HDCD

Reply #61

The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

Let me just share my opinion on that with you all. I think it's not a good idea to decode HDCD "just for lossless storage" for the same reason you wouldn't download MP3 files and store them decoded as WAVs or FLACs on your harddisk. This decoding should only be done as preprocessing (prior lossy encoding) or as "postprocessing" (within a player, like the player decodes MP3). Otherweise you'll be wasting a lot of harddisk space.


The analogy to mp3 decoding is a good one.  Conceptually, hdcd is really just a lossy way of encoding 20 bit audio with (by design) the very useful property that the compressed output is very similar to a 16 bit version of the original.

I'm inclined to agree with you in principal that the decoding process should be done during playback or transcoding.  In practice however, one doesn't always have the means to do this, and converting before hand is a good alternative.  I would *strongly* recommend never deleting the original rip however, as this is the master copy.  Hopefully plugins will become available, removing the need to store a preconverted version though.

I'm sure FLAC will have trouble catching the gaps (sample values not in use). For example: If you convert 16 bit samples to 24 bit samples and adjust the gain by -9dB to undo the dynamics compression only one out of 91 possible 24bit samples values is used (assuming you don't dither). Since 91 is not a power of two the least significant bits of the samples will look like random bits so that FLAC can't exploit the fact that the signal has an effective resolution of only 17.5 bits (16+1.5 due to the gain of -9 dB). So, approximately 573 kilobits/s (2*(24-17.5)*44100) are wasted (!!!).


I've not looked closely at how FLAC operates, but this certainly sounds highly likely.  Huffman coding would work relatively well for this sort of situation, but would be fairly pointless for most audio, so is likely not used.

One point to note however is that wmp does truncate the output to 20 bits; maybe FLAC would be able to take advantage of that.

Regards,

Chris


Quote

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed.  I'd guess:

Peak extend enabled at some point during the audio
Peak extend disabled at some point during the audio
Peak extend used (|sample|>0x5980 while peak extend used)
Range of gain adjustment values used.

Extract HDCD

Reply #62
The older foobar v0.8 series do allow Playback/Bit Depth to be specified using "Additional Options" which includes 20-bit padded to 24-bit. FLAC and Wavpack are supposed to take advantage if they detect that the bottom four bits are set to zero. If they aren't zeroed already using hdcd.exe then fb2k might save you a few kbps in your FLACs if you can be bothered to output to 20-bit padded to 24.
Dynamic – the artist formerly known as DickD

Extract HDCD

Reply #63
I'll have a go tomorrow at adding some more diagnostics


Well there's a new version available with a few more options and some more detailed information.  For now it's available from,

http://www.srcf.ucam.org/~cjk32/hdcd/hdcd-new.zip

but it'll replace the existing version,

http://www.srcf.ucam.org/~cjk32/hdcd/hdcd.zip

fairly shortly.


To get usage information, run,

hdcd.exe -h

I won't explain here as I'd like the usage information to be self explanatory. If anything's unclear, let me know and I'll explain and update it.

Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris

Extract HDCD

Reply #64
One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!



Hi guys, great thread.  I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs;  the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD.  You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism? 


Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe.  FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example).  I couldn't tell them apart in an ABX.

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs).  THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction.  see this post for examples.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them.  Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.


All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')

I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed.  I'd guess:

Peak extend enabled at some point during the audio
Peak extend disabled at some point during the audio
Peak extend used (|sample|>0x5980 while peak extend used)
Range of gain adjustment values used.


From what I've read (e.g. John Marks' self-interview about his HDCD releases)  it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion.  So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

Extract HDCD

Reply #65
Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs).  THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction.  see this post for examples.


You're right. I have the japanese HDCD release of Malmsteen's Concerto Suite for Electric Guitar and Orchestra. Here is the output of the hdcd.exe:

Code: [Select]
D:\>hdcd.exe -o track08.wav 08_-_Track08.wav
HDCD Detected
Features Used:
  Peak extend   : Not enabled
  Minimum gain  : 0.0dB
  Maximum gain  : 0.0dB

D:\>

The Audition says that the actual bit depth of decoded wav is 17 bits, and the peak is -6.01 dB. This is the same for all tracks on the CD.

 
From what I've read (e.g. John Marks' self-interview about his HDCD releases)  it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion.  So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

When I play this CD in WMP11 without 24 bit playback enabled, I see the HDCD logo. But when I enable 24 bit playback (as it should be), the logo doesn't show up.
If age or weaknes doe prohibyte bloudletting you must use boxing

Extract HDCD

Reply #66
When I play this CD in WMP11 without 24 bit playback enabled, I see the HDCD logo. But when I enable 24 bit playback (as it should be), the logo doesn't show up.

Same here, for a while the logo wouldn't show up at all, but after switching the 24-bit option on and off a few times it finally appears while the 24-bit option is off 

But I guess the 24-bit option is working because it sounds noticeably quieter.
we was young an' full of beans

Extract HDCD

Reply #67
Hi guys, great thread.  I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs;  the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD.  You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?


My view is that theoretically, you should only ever store the 16 bit master.  That can then be played back as 16 bit or 24 bit dependent upon the listening environment.  In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.

Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe.  FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example).  I couldn't tell them apart in an ABX.


That's rather worrying.  The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC.  (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly.  (Not strictly true.  A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested.  I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you?  I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs).  THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction.  see this post for examples.


Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes.  The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit.  Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them.  Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.


From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels.  If you're so inclined, this can be fixed with a hex editor.

All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')


There are 3 unidentified bits in the hdcd codes that might be used to switch filters.  My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them.  I'd be very grateful to hear from anyone who finds a CD in which they're used.

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps).  Is this what's referring to as 'low level extension'?

From what I've read (e.g. John Marks' self-interview about his HDCD releases)  it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion.  So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).


That certainly sounds sensible from a marketing point of view.  As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

Regards,

Chris

Extract HDCD

Reply #68
My view is that theoretically, you should only ever store the 16 bit master.  That can then be played back as 16 bit or 24 bit dependent upon the listening environment.  In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.


Theoretically very desirable, but practically not do-able in any easy way....so far.  I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)


Quote
That's rather worrying.  The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC.  (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly.  (Not strictly true.  A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested.  I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you?  I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia



It was Joni Mitchell 'Mingus'.  FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around.  To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks.  For hdcd.exe of course all I have to do is feed it the individual .wavs directly.  I can do it all again and compare the files.  Also  I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.


Quote
Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes.  The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit.  Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.




Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'.    The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

Code: [Select]
undecoded
    Left    Right
Min Sample Value:    -32452    -32429
Max Sample Value:    32440    32415
Peak Amplitude:    -.08 dB    -.09 dB
Possibly Clipped:    0    0
DC Offset:    0     0
Minimum RMS Power:    -138.15 dB    -139.94 dB
Maximum RMS Power:    -2.58 dB    -4.66 dB
Average RMS Power:    -11.14 dB    -11.51 dB
Total RMS Power:    -10.54 dB    -10.9 dB
Actual Bit Depth:    16 Bits    16 Bits

Using RMS Window of 50 ms



decoded
    Left    Right
Min Sample Value:    -16226    -16214.5
Max Sample Value:    16220    16207.5
Peak Amplitude:    -6.1 dB    -6.11 dB
Possibly Clipped:    0    0
DC Offset:    0     0
Minimum RMS Power:    -144.17 dB    -145.97 dB
Maximum RMS Power:    -8.6 dB    -10.68 dB
Average RMS Power:    -17.16 dB    -17.54 dB
Total RMS Power:    -16.56 dB    -16.92 dB
Actual Bit Depth:    17 Bits    17 Bits

Using RMS Window of 50 ms


I can send this track to you too if you like.


Quote
From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels.  If you're so inclined, this can be fixed with a hex editor.



Thanks. I'd much rather just use hdcd.exe, so long as it's creating files that are identical in all the ways that matter, to what the WMP WAV plugin generates.


Quote
There are 3 unidentified bits in the hdcd codes that might be used to switch filters.  My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them.  I'd be very grateful to hear from anyone who finds a CD in which they're used.


Check out the link I made, to my Steve Hoffman forum post  There I document a variety of hdcd behavior  -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor).  How can I tell if the 3 bits are in use?


Quote
Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps).  Is this what's referring to as 'low level extension'?



Beats me.  I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D  (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters).  Maybe there's more options?





Quote
That certainly sounds sensible from a marketing point of view.  As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.


IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'.  But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light. 

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').

Extract HDCD

Reply #69
Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris


here's that variety I mentioned, detected by hdcd.exe

Beck "Midnite Vultures', 02 - Nicotine & Gravy
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Van Halen "1984" 02 - Jump
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

King Crimson "Starless & Bible Black" 02 - Lament
HDCD Detected
Peak extend: Not enabled
Minimum gain: -4.0dB
Maximum gain : 0.0.dB

Joni Mitchell "Mingus" 09- The Dry Cleaner from Des Moines
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: -0.5 dB
Maximum gain : 0.0.dB

Yes "Yessongs" 05 - And You And I
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Yes "Big Generator" 04 - Almost Like Love
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB



(is there any way to get hdcd.exe to generate a log file?  I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)

Extract HDCD

Reply #70
I've started to run all my known HDCD encoded CDs through the HDCD decoder and I'm then compressing using WavPack 4.42 (-hhx6).  I'll post more results once I get them.

Thanks for posting your compression results, however I was very surprised that some albums would actually get smaller after the HDCD decoding. I can only imagine two reasons this could happen. One is if the original data was not compressed with the same settings (say, -hhx instead of -hhx6), and the other was if in some cases the HDCD program is actually throwing away a bit (but that wouldn't make any sense).

I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations! 

BTW, I'm sure you have thought of this, but a very interesting complementary project would be an HDCD encoder. Then any band could release their own HDCD CDs, although Microsoft might have something to say about that! 

Extract HDCD

Reply #71
I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Here's a possible explanation: The peak compression is heavily non-linear. It creates harmonic distortions which could be harder to predict than the original or decoded data.

Cheers!
SG

Extract HDCD

Reply #72
[quote name='krabapple' date='Aug 30 2007, 19:56' post='513502']
[quote name='Christopher Key' post='513457' date='Aug 30 2007, 11:33']

My view is that theoretically, you should only ever store the 16 bit master.  That can then be played back as 16 bit or 24 bit dependent upon the listening environment.  In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.
[/quote]

Theoretically very desirable, but practically not do-able in any easy way....so far.  I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)
[/quote]

Hopefully plugins will become available.  Storage isn't really the problem, HDCDs will always be a very small percentage of one's music library, it's more the incovenience of having two versions available stored, keeping them both tagged correctly, making the correct one gets selected etc.  It depends on your particular set up as to whether this is a problem

Quote

Quote

That's rather worrying.  The software was designed to be bit identical to wmp, and is for my 5 test cds.

Which CD was it that wasn't matching for you?


It was Joni Mitchell 'Mingus'.  FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around.  To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks.  For hdcd.exe of course all I have to do is feed it the individual .wavs directly.  I can do it all again and compare the files.  Also  I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.


If you get a chance, would you mind rechecking just one of the tracks.  You may need to watch out for sample offsets, I did find that wmp rather bizarrely advanced to first 2m by 4 samples, then corrected itself.  I've not been abl to get it to do it again however.

I'll see if I can acquire a copy of 'Mingus', along with a few others mentioned later to test this myself.

Quote

Quote

Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes.  The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit.  Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.


Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'.    The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

Code: [Select]
undecoded
    Left    Right
Min Sample Value:    -32452    -32429
Max Sample Value:    32440    32415
Peak Amplitude:    -.08 dB    -.09 dB

Actual Bit Depth:    16 Bits    16 Bits

Using RMS Window of 50 ms



decoded
    Left    Right
Min Sample Value:    -16226    -16214.5
Max Sample Value:    16220    16207.5
Peak Amplitude:    -6.1 dB    -6.11 dB

Actual Bit Depth:    17 Bits    17 Bits

Using RMS Window of 50 ms



That's exactly as expected.  If neither peak extend nor gain adjustment are used, the least significant seven bits of each sample will always be zero (as will the one most significant), hence the bit depth is detected as 17 bits.  I don't know why the peak amplitude values aren't exactly 6dB lower in the decoded case, (0.09dB -> 6.11dB, not just the way the values are rounded), but the sample values are *exactly* halved.


Quote

Quote

There are 3 unidentified bits in the hdcd codes that might be used to switch filters.  My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them.  I'd be very grateful to hear from anyone who finds a CD in which they're used.


Check out the link I made, to my Steve Hoffman forum post  There I document a variety of hdcd behavior  -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor).  How can I tell if the 3 bits are in use?


Thanks for the link, you initial post covers the behaviour nicely.  Category 3 and 4 are essentially the same from a technical point of view, they both have peak extend enabled, category 4 has larger sample value before decoding that go further up the expansion curve.  Given that I've got the data, I really should produce an accurate plot of the curve to demonstrate how it works.

Quote

Quote
Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps).  Is this what's referring to as 'low level extension'?



Beats me.  I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D  (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters).  Maybe there's more options?


http://web.archive.org/web/20030605091427/...o/overview.html

Mentions those three, and only those three, so I suspect that those are the options.  I'll try to use their terminology from now on to avoid confusion.

Interestingly, the low level range extension referred to there (20dB in 4dB steps) is much different to and stronger that used by wmp (7.5dB in 0.5 dB steps).

I did catch the AA thread but got lost fairly quickly as a result of the strength of their convictions.


Quote

Quote


That certainly sounds sensible from a marketing point of view.  As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.


IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'.  But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light. 

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').


Quite agree, particularly the loudness point.  You can justify the behaviour from a technial point of view, if normal CDs weren't attenuated, and an hdcd code was spuriouusly detected during playback, you'd get a sudden jump in volume.  On the other hand, when they are already attenuated by 6dB, a spuriusly detected hdcd code will just give you slightly more dynamic for a short period, which is much less offensive.  I doubt this is the primary motivation however!


[quote name='krabapple' date='Aug 30 2007, 20:20' post='513505']
[quote name='Christopher Key' post='513258' date='Aug 29 2007, 14:12']
Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris
[/quote]

here's that variety I mentioned, detected by hdcd.exe

Beck "Midnite Vultures', 02 - Nicotine & Gravy
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Van Halen "1984" 02 - Jump
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

Yes "Yessongs" 05 - And You And I
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen:  0x03

[/quote]

I'll try and get a copy of those that have the unknown codes.  The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes.  This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing.  If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.

Quote

(is there any way to get hdcd.exe to generate a log file?  I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)


Just redirect stderr to a file.  Using the newer version, it should be as simple as,

hdcd.exe -o out.wav in.wav 2>mylog.txt

and with older version that only supports stdin / stdout,

hdcd.exe < in.wav > out.wav 2>mylog.txt


Also, take a look at the slimdevices thread, that's got some guidance on how to use this in a script,

http://forums.slimdevices.com/showthread.p...967&page=24


I did a quick scan of my library and was interested to find that all my Mark Knopfler CDs are HDCD, despite not being marked as such, as was *one* track from a Bond themes compilation CD.  I'm guessing the compiler probably didn't realise either!!!

Extract HDCD

Reply #73
Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations! 



Ouch, trying to do it with only an analogue reference was going to be seriously hard!

I'd be very interested to know how you went about deriving the tap values used for the LFSR.  There's a breif description of how I went about it earlier in this thread, but I can't believe that was the easiest way.


BTW, I'm sure you have thought of this, but a very interesting complementary project would be an HDCD encoder. Then any band could release their own HDCD CDs, although Microsoft might have something to say about that! 


I would certainly be technically possible, although to do it well would require a lot more knowledge of audio processing than I have at present.  The decoding is easy, there's no decisions to be made.  Encoding isn't quite as straighforward.  Peak extend is fine, low level range extend is fine with a bit of care to get a suitable level of hysteresis, but getting the dithering right is probably quite hard.  I also have no idea yet quite how the filter switching works, given that wmp doesn't appear to do that.

It'd certainly be something to speak to Microsoft about, they may well not object.  No HDCD encoders are being sold at present, so it's not really providing any competition, and electronic releases of HDCD music might well encourage some people to switch to wmp for playback.

Regards,

Chris

Extract HDCD

Reply #74
Quote

(is there any way to get hdcd.exe to generate a log file?  I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)


Just redirect stderr to a file.  Using the newer version, it should be as simple as,

hdcd.exe -o out.wav in.wav 2>mylog.txt

and with older version that only supports stdin / stdout,

hdcd.exe < in.wav > out.wav 2>mylog.txt


Also, take a look at the slimdevices thread, that's got some guidance on how to use this in a script,

http://forums.slimdevices.com/showthread.p...967&page=24


I did a quick scan of my library and was interested to find that all my Mark Knopfler CDs are HDCD, despite not being marked as such, as was *one* track from a Bond themes compilation CD.  I'm guessing the compiler probably didn't realise either!!!


Thanks for addressing all the points in my (long) post, Christopher!

Regarding logs/scanning, before reading your reply I did some research and managed on my own to write two batch files that do the job for me...all of my files are .flacs, so one batch file decodes the first second of track 01 of each album to .wav, the second batch file runs hdcd.exe -a (i.e., .wav output suppressed) on those 1-sec wavs and directs the text output and stderr to a log file. Then I go back and delete all the tiny .wav files.  Pretty kludgy but it worked fine! 

Doing that I discovered some interesting things in my collection too...some things I'd either forgotten were HDCD, or never realized in the first place! (marked with asterisks).  I've always thought the J. McLaughlin and J. Cash CDs sounded way loud, and maybe now I know why.    Also interesting to find that some, but not all, King Crimson discs do employ peak extension.

(And because of your Bond result, I'm thinking maybe I need to do more than just 'sample' each album!)


Code: [Select]
Beck    1999 - Midnite Vultures    Peak extend   : Not enabled  
Beck    2002 - Sea Change    Peak extend   : Not enabled  
Grateful Dead    1968 - Anthem of the Sun    Peak extend   : Not enabled  
*John McLaughlin    1970 - My Goal's Beyond    Peak extend   : Enabled permanently  
*Johnny Cash    1999 - 16 Biggest Hits    Peak extend   : Enabled permanently  
Joni Mitchell    1974 - Court And Spark    Peak extend   : Enabled permanently  
Joni Mitchell    1976 - Hejira    Peak extend   : Enabled permanently  
Joni Mitchell    1979 - Mingus    Peak extend   : Enabled permanently  
King Crimson    1970.1 - In the Wake of Poseidon    Peak extend   : Enabled permanently  
King Crimson    1970.2 - Lizard    Peak extend   : Enabled permanently  
King Crimson    1971 - Islands    Peak extend   : Enabled permanently  
King Crimson    1972 - Earthbound    Peak extend   : Not enabled  
King Crimson    1973 - Larks' Tongues In Aspic    Peak extend   : Not enabled  
King Crimson    1974.1 - Starless And Bible Black    Peak extend   : Not enabled  
King Crimson    1974.2 - Red    Peak extend   : Not enabled  
King Crimson    1981 - Discipline    Peak extend   : Not enabled  
King Crimson    1982 - Beat    Peak extend   : Not enabled  
King Crimson    1984 - Three Of A Perfect Pair    Peak extend   : Not enabled  
Mahavishnu Orchestra    1972 - The Lost Trident Sessions    Peak extend   : Enabled permanently  
*McDonald & Giles    1969 - McDonald & Giles    Peak extend   : Not enabled  
Roxy Music    1972 - Roxy Music    Peak extend   : Not enabled  
Roxy Music    1973 - For Your Pleasure    Peak extend   : Not enabled  
Roxy Music    1976 - Viva!    Peak extend   : Not enabled  
The B-52's    1998 - Time Capsule    Peak extend   : Not enabled  
The Flaming Lips    1999 - The Soft Bulletin    Peak extend   : Not enabled  
Van Halen    1983 - 1984    Peak extend   : Not enabled  
*Weather Report    2002 - Live & Unreleased    Peak extend   : Enabled permanently  
Yes    1973 - Yessongs (JPN)     Peak extend   : Enabled permanently