Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Most 'true' way to de-emphasize CD image (Read 78522 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Most 'true' way to de-emphasize CD image

Reply #25
even the studio machines didn´t apply the curve always correct and had tolerances.
Probably the majority of cd's with emphasis have been recorded on a Sony PCM 1600/1610/1630 processor (the one that used a U-matic video recorder to store the data). AFAIK the pre-emphasis and de-emphasis circuits were analog and there must have been tolerances between models. Semi pro models like the PCM-F1, 501, 601 and 701 were used too, but not by the major studios.
If you want to remove the emphasis digitally, there are two options IMO:
1) use a 15/50µs digital filter, since that are the theoretically correct values. This however doesn't take into account the imperfections of the ADC.
2) measure the original ADC pre-emphasis curve and apply an inverse version to the signal.

I think it's a mistake to emulate the de-emphasis circuit, since it isn't necessarily identical to the pre-emphasis circuit. It would be interesting to know if the differences are worth worrying about. IIRC Daniel Weiss (pro audio equipment manufacturer) investigated this long time ago and decided to use different values from 15/50µs.

Most 'true' way to de-emphasize CD image

Reply #26
Measured difference of 0.2 dB is not bloody likely to be audible for anything but a test tone in the most sensitive frequency range of human hearing.

Most 'true' way to de-emphasize CD image

Reply #27
Measured difference of 0.2 dB is not bloody likely to be audible for anything but a test tone in the most sensitive frequency range of human hearing.
Tolerances are fine, but IMO their audibility should be verified at the end of the chain. If the chain is unknown, you "might" have a problem .
More importantly, if higher accuracy comes at no or little additional cost/effort, there is no excuse for sloppiness, IMHO.
Don't forget the OP asked for the most "true" way.

Most 'true' way to de-emphasize CD image

Reply #28
One thing i wonder how accurate the Denon original sweep really is and how it was created (...)

Those tracks on the CD are marked D*, which means it was "produced by a high-precision digital 16-bit signal processor" "of Denon/Nippon Columbia" "controlled by a computer and recorded in digital form on the compact disc", "The precision of D* sometimes deviates from the theoretical value, 16-bit, at signal generation (distortion factor: less than 0.1%, level deviation: within +- 0.1dB)"

BTW, I used SoX at the end because it supports flac and the difference is practically not very relevant.

Most 'true' way to de-emphasize CD image

Reply #29
I tried one more thing.
Seems really like the created reference files differ beneeth each other already. For this test i took the reference file from radonmaster.de i linked to before and removed the silent parts.
So i believe the sox claim of being in +/- 0.06dB of the value as long we can´t be sure how exactly the reference files are created.



Edit: besides that. while applying this effect the resulting file should be dithered. waveemph doesn´t do that.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!


Most 'true' way to de-emphasize CD image

Reply #31
If you want to do it digitally the best way is to process the file at 24 bits (or higher) then either leave it at the higher bit depth or dither it back to 16.  If you process it at 16 bits you will definitely lose fidelity.


Well, I just tried that. Converted to 24 and SoX-deemph'ed the files.
Guess what, they became > 50 % larger. Not by converting to 24 bits (FLAC compresses away the padded zeroes at little extra space cost), but by the deemphasis filter.

Which is fairly strange, as the actual information is the same, but:
Do not the two facts combined, that (I) the EQ curve only hits appr. 10dB (< 2 bits, or just above 10 %) and (II) the files seem to grow by 50 %, (mildly) suggest that SoX de-emph introduces digital noise below the 18th bit, which actually should be dithered away for fidelity?


(SoX version 14.1.0, libflac version 1.2.1, flac -8 used)

Most 'true' way to de-emphasize CD image

Reply #32
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.


Most 'true' way to de-emphasize CD image

Reply #34
It should make little difference to a lossy encoder, but a lossless encoder doesn't care whether the extra bits are significant or not, it only knows that it is required to reproduce them exactly, and that requires extra bytes.

Most 'true' way to de-emphasize CD image

Reply #35
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.

But a parametric EQ shouldn't make it that much harder to compress? Or?

If you keep it at 16-bits, then it's no harder to compress.  If you upsample to 24 bits without processing, it's no harder to compress.  If you go to 24 bits and do any processing, then yes...  As pdq said the extra bits are "filled up".

You are working with integer files, and when you go from 16 to 24 bits, the additional bits are analogous to digits to the right of the decimal point.   The extra bits are not used to make your file "louder", they are used to give you more "detail".  (That's not strictly true...  24 bit audio is also used to get more headroom.)

Your data is integer form, but your scale factors are not...  So, when you apply a mathematical operation to your 16-bit integer, you need more precision, or you can round-off. 

For example, if you increase a sample value by 1dB, that's a scale factor of ~1.12202.  With integer data that's going to require rounding.  You will get more precision (less rounding error) with 24 bits.  And, the higher precision means it's "harder" to compress.

Most 'true' way to de-emphasize CD image

Reply #36
So long as sox does the internal math at a higher bit depth and then dithers to get back to 16 bits then I would do it that way.  As has been discussed over and over, 16-bits is adequate for final delivery.  You'll probably notice that your de-emphasized tracks will compress better than their original sources in both lossy and lossless formats.

 

Most 'true' way to de-emphasize CD image

Reply #37
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.

In comparison, HDCD decoding (up to 19 point something bits, right?) seems not to add much -- naturally, since it is storeable in 16 bits (plus a description of the algorithm) in the first place.

And the HDCD results indicate that the performance of FLAC does not worsen disproportionally with the word length (unlike what others have reported with other codecs).



@ Greynol:
So long as sox does the internal math at a higher bit depth and then dithers to get back to 16 bits

Well does it? (No, I haven't read the ultimate documentation -- i.e. the source code.)


You'll probably notice that your de-emphasized tracks will compress better than their original sources in both lossy and lossless formats.

Actually, it has struck me that they usually come out a slight bit worse.
My hypothesis was that the original digital pre-master being 16 bits, and so increasing the treble by 10 dB would maybe lead to excess peak (and some information having to be cut away), maybe not, so the information content of the pre-emphed and de-emphed 16 bits would be more or less the same, but maybe the dithering noise being so idiosyncratic that it takes more space to compress.
But that is at odds with my results as of now.

Most 'true' way to de-emphasize CD image

Reply #38
Well does it?

Don't know.  Are you able to tell the difference in a double-blind test between 24 bit output and 16 bit output under normal listening conditions (ie. you don't crank the volume up to 11 during fade-outs)?

Most 'true' way to de-emphasize CD image

Reply #39
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.

Are you suggesting that the encoder try rescaling the data by 2^24 different divisors to see if one of them magically restores the low 8 bits to all zeros? This hardly seems worth the effort.

Besides, in this case we weren't even talking about scaling by a constant, but about applying an equalization. How many possible de-equalizations are we supposed to try in order to reduce the data to 16 bits?

Most 'true' way to de-emphasize CD image

Reply #40
Besides, in this case we weren't even talking about scaling by a constant, but about applying an equalization. How many possible de-equalizations are we supposed to try in order to reduce the data to 16 bits?

Hey, don't ask me, I am only one of those mathematicians who would say that passwordcracking is a trivial issue as there are only countably many combinations 

But nevertheless, the process of deemph'ing to 24 bits seems in practice to take disproportionally much more space than HDCD-decoding to ... at most > 19, which was my "gut feeling" benchmark for what I would expect. My HDCD files seem to encode to something fairly less than 25 % more than the original, but my 24 bit de-emphed files are some > 70 % more.
That's way more than the increase in wordlength.

("Test corpus": 15 discs, rock/darkwave.)


@ Greynol:
Objection! Not even hearsay argument 

Most 'true' way to de-emphasize CD image

Reply #41
For the ones that want a VST solution to do some de-emphasis i have a suggestion. There must have been a challenge to deliver VST plugins last year at KVR. There is one neat FREE Equalizer plugin from Matthew Lindsay named NCL Phase EQ.
http://www.kvraudio.com/developer_challenge_2009.php

If you disable all other filters but the HighShelf and set Freq: 5300, Q: 0.48, Gain: -9.4 you get a nice matching de-emphasis curve similar exact as sox. At least with the reference file i have.
I found no way to contact the author myself. He has several presets already into this plugin. Adding de-emphasis could be a gimmick. Maybe someone of you has an idea?

PS: switching to double-precision (2x) made it impossible for me to find a matching curve.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Most 'true' way to de-emphasize CD image

Reply #42
Well, not the most interesting thing but i wanted to mention that while i played with the NCL EQ i throwed in an iTunes 10 de-emphasis curve some recommend here and there.
It was obviously most away from the ideal curve of all the ones i tested. It must be dipping up to -1dB to much. There should the audibility start.



Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

Most 'true' way to de-emphasize CD image

Reply #43
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.


That's nonsense. In "theory" that's what happens in nondestructive editing, where information about the order and parameters of each applied algorithm is kept. Never wondered why it is called "destructive" editing once such a log isn't kept alongside an original file? It is not only a practical limitation, but especially "in theory", the problem space, of what you expect a lossless compressor to accomplish, is too large for any thinkable search algorithm.

For the rest, just reread DVDdoug's post #36. He pretty much nailed it.

Most 'true' way to de-emphasize CD image

Reply #44
That's nonsense. In "theory" that's what happens in nondestructive editing, where information about the order and parameters of each applied algorithm is kept. Never wondered why it is called "destructive" editing once such a log isn't kept alongside an original file? It is not only a practical limitation, but especially "in theory", the problem space, of what you expect a lossless compressor to accomplish, is too large for any thinkable search algorithm.

For the rest, just reread DVDdoug's post #36. He pretty much nailed it.

I dont understand this. Of course, a lossless encoder wont do an explicit blind modelling of the destructive processing done to its input. I took it more as an argument about information or entropy.

A lossless encoder contains a prediction filter that tries to predict future samples, right? Inserting new, unpredictable information (noise) means that the encoder cannot predict information, and the residue is larger. However, doing static, linear filtering, this should only concolve up to a new (larger) optimal prediction filter, and depending on the length of the prediction done by flac, as well as the original source autocorrelation, could work?

"Entropy" and "information" is beginning to be vague concepts for me.....

-k

Most 'true' way to de-emphasize CD image

Reply #45
As said, it's already all in post #36.

Digital filtering involves rounding. Rounding of 16 bit values at 24 bit precision will fill the full 8 additional bits with new information, even if a given filter was very simple. That new information might not be significant, i.e. noise, but that's up to a lossy encoder to decide. A lossless encoder has to preserve it.

Most 'true' way to de-emphasize CD image

Reply #46
As said, it's already all in post #36.

Post#36 does nothing to explain this. It merely states the obvious, that the 8 lsb will typically be non-zero when doing non-trivial operations on a 16-bit signal wrapped into a 24-bit container.
Quote
Digital filtering involves rounding. Rounding of 16 bit values at 24 bit precision will fill the full 8 additional bits with new information, even if a given filter was very simple. That new information might not be significant, i.e. noise, but that's up to a lossy encoder to decide. A lossless encoder has to preserve it.

This does not make sense. A filter does not create information (look up Shannon).

If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.

There are surely practical constraints that can prevent this (buffer sizes, processing power, truely random dither etc), but no-one has elaborated on those.

-k

Most 'true' way to de-emphasize CD image

Reply #47
This does not make sense. A filter does not create information (look up Shannon).


That's true under the condition that we have machines operating on real numbers, which we don't. Instead we have samples of finite width, which when processed, can to lead to new samples of theoretically infinite width, which we represent by rounding.

If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.


Maybe in math operating on real numbers, but not universally for an automaton processing samples of finite precision with finite storage space. The information might just be lost in some (if not most) cases. Yes, in other cases it might be recoverable. But claiming reconstructability "in principle" is definitely wrong.

Any re-quantization after a non-trivial operation destroys information by adding new information (quantization artifacts). By increasing the word length over the original length the information loss of re-quantization can be reduced. In the cited case the cost of that reduction of loss is 8 bits. It is not the cost of filtering, but the cost of reducing re-quantization artifacts which take up the 8 lsb.

Most 'true' way to de-emphasize CD image

Reply #48
Now we are getting somewhere. So I should have stated "an ideal filter" in my quick-and-dirty "in principle". And someone should have stated above that they referred to the post (ideal) filter quantization.

Lets say the filtering is carried out ideally (double precision float or whatever), the input is 16 bits and the output is rounded to 24 bits. The proper explanation for why flac cannot compress the 24-bit signal as well as the 16-bit signal is not that the 8 lsb are non-zero, but that flac contains a linear predictor that cannot model a (non-linear) 24-bit truncation? This truncation/quantization could be modelled as a random process but it really is not (just listen to quantization error after 4-bit quanttization).

We dont know (or explanations have not been provided) whether the information is really lost, or if some imaginary perfect non-linear predictor in flac could solve the general case outlined.

-k

Most 'true' way to de-emphasize CD image

Reply #49
I think it's far simpler than you think.

Consider the 16-bit file as 24-bits with the 8 LSB being zeros.
You filter it, and the 8 LSB become none zero. Aside from clipping, dithering, rounding etc, the information content has changed very little, if at all.

Why is the bitrate with FLAC so different? Easy:
1. FLAC has an absolutely perfect (or near as damn it perfect) way of expressing the 8 LSB zeros efficiently.
2. FLAC has a very imperfect method of expressing the filtering efficiently. In fact, it makes no special attempt to, since in the general case there is no utility in doing so, and even in this specific case, the prediction (to give a perfect result, recovering 8 zeroed LSBs) would have to be far more accurate than anything else it ever achieves.

If you take the 24-bits with 8 zero LSBs, and add a DC offset of 000000000000000010101011 to it, thus breaking the "perfect" trick FLAC can use on those 8 zero LSBs, you'll probably find the compression differential between that and the filtered version narrows dramatically (unless FLAC is smart enough to remove it - I haven't tried). This is despite the DC offset carrying exactly 8 extra bits of information for the entire file over and above the 24--bits with 8 zero LSBs.

FLAC isn't a form of artificial intelligence trying to find the absolute smallest lossless representation of a given set of data, including reverse engineering any algorithms that may have "bloated" that data; it's just trying to do a reasonable job in a reasonable time.

Shameless plug: if you want to throw away bits that don't contain "real" information, try lossyWAV  But then it won't be lossless.

Cheers,
David.