Hello!
Have a question related to de-emphasizing of images of audio CDs mastered with pre-emphasis. What I've already found on this topic:
1. Using Waves Q10 plugin. But it seems to me that equalizer plugin is not the best way to achieve originally mastered sound.
2. Using foobar2000 convolver with Waves_De-Emphasis pulses. Don't know anythig good/bad for this way.
So does anybody knows what way is 'true' to de-emphasize wave file?
PS: I know that using EAC or K3b to burn CD and delegate de-emphasizing to CD player will solve issue. But for some reasons I need to have images on my PC and ability to play them or several tracks on players that don't support 'pre' flag.
Are they still making CDs with pre-emphasis, or are these some pretty old discs?
You can de-emphasize using sox (a command line program).
Are they still making CDs with pre-emphasis, or are these some pretty old discs?
This is rather old discs. For example, first masters of early Pink Floyd or Deep Purple albums.
Egor, I've looked to side of sox. By some opinions it almost kills stereo from tracks
Egor, I've looked to side of sox. By some opinions it almost kills stereo from tracks
Eh? News to me. Could you post an example where SoX "kills" stereo, or refer us to some notes/opinions that explain how and why? SoX has a bit of a learning curve, so I suspect this is simply a case of someone's parameters not being quite right.
- M.
Egor, I've looked to side of sox. By some opinions it almost kills stereo from tracks
Eh? News to me. Could you post an example where SoX "kills" stereo, or refer us to some notes/opinions that explain how and why? SoX has a bit of a learning curve, so I suspect this is simply a case of someone's parameters not being quite right.
- M.
I don't remember exactly a link to tests of stereo damaging (I'll try to find it), but here http://www.hydrogenaudio.org/forums/lofive...php/t62156.html (http://www.hydrogenaudio.org/forums/lofiversion/index.php/t62156.html) I've found reference someone's opinion that sox can damage the sound durind de-emphasizing process.
CD de-emphasis was originally designed to be done in the analog domain. If you want to do it digitally the best way is to process the file at 24 bits (or higher) then either leave it at the higher bit depth or dither it back to 16. If you process it at 16 bits you will definitely lose fidelity. Unfortunately I'm not familiar enough with the SoX internals to know how its effects are applied.
Using Waves Q10 too. Tried Sox before but did hear a difference in stereo extensiveness (much better with Waves).
Surfi.
Yes, de-emphasis is meant to be done in the analog domain.
Two reasons:
1) That high 15us zero doesn't have enough resolution to do well in the digital domain.
2) The de-emphasis can produce peaks that would clip in the digital domain.
Using Waves Q10 too. Tried Sox before but did hear a difference in stereo extensiveness (much better with Waves).
How is a de-emphasis filter going to interfere with stereo information?
Yes, de-emphasis is meant to be done in the analog domain.
Two reasons:
1) That high 15us zero doesn't have enough resolution to do well in the digital domain.
2) The de-emphasis can produce peaks that would clip in the digital domain.
All true but once the file is ripped it has to be fixed. In Audition I use this FFT filter and I'm pretty happy with it. Log scale and spline curves on.
21.53 Hz 0dB
1241.7 Hz 0dB
1552.49 Hz 0dB
2386.17Hz -0.6dB
3330.31 Hz -2.4dB
6228.9Hz -6.3dB
8097.04 Hz -8.4dB
10851 Hz -9.6dB
22050 Hz -9.9dB
This filter will do nothing at all to stereo separation as it is done identicallly to the 2 channels with no 'blending'.
I asked this several months back. Is there any software that absolutely identifies pre-emphasis yes/no ?
G²
Egor, I've looked to side of sox. By some opinions it almost kills stereo from tracks
No need to worry! You've run in some misinformation.
Right now I've prepared a test audio cd with two tracks. Here is the cuesheet for burning:
FILE "test.wav" WAVE
TRACK 01 AUDIO
FLAGS PRE
INDEX 01 00:00:00
FILE "test-deemph.wav" WAVE
TRACK 02 AUDIO
INDEX 01 00:00:00
I don't have any real CDs with preemphasis, so I de-emphasized a normal CD track with sox:
C:\audio>sox test.wav test-deemph.wav deemph
When played back in a standalone CD player, one would hear "hardware" de-emphasis for the first track, and software de-emphasis for the second track.
So, I have just played back the freshly-burnt CD-RW in my Panasonic SL-CT820 portable CD player and guess what - I have not heard any difference between the 1st and 2nd tracks. Unfortunately I ain't got no SPDIF-in capture device to grab CD player's optical out, so here's my feedback in text form only.
It is all very interesting, thank you for information. So as far as I understood converting WAV to 24 bit, then de-emphasizing using SoX or Adobe Audition and then dithering to 16 bit could help levae fidelity almost intact. Interesting, is there information about de-emphasizing filter curve used in SoX anywhere ?
Found also information about tool written by acoustic engineer mr. Ahlersmeyer: that could be found at http://www.picosound.de/ (http://www.picosound.de/) Did someone tested this? I have Vista so this tool don't wanna run even in Win98 compatibility mode.
PS: what software could be recommebded for 16-24 and 24-16 bits transformations?
Interesting, is there information about de-emphasizing filter curve used in SoX anywhere ?
SoX is open source by the way Looking at the source may help somehow, even if don't speak C.
For bit-depth coversion use foobar2000.
@Glenn. Goldenhawk's CDRWIN can instantly display flags (incl. PRE) for a CD. Use the "Table of contents" command.
According to the changelog, the deemphasis filter was also changed from 1st order to 2nd order with the release of 14.0.0 (2007/09/11).
- M.
Interesting, is there information about de-emphasizing filter curve used in SoX anywhere ?
SoX is open source by the way
Already reading. I did not found any arrays with coefficients. It seems that it uses some implementation of digital filter and it is possible to reconstruct curve, but I need to extract formulae at first.
Did anyone check how well iTunes deemphasizes CDs?
I have tried Sox and WaveEmph and comparing the spectrum of a single impulse I saw the cutoff point for Sox de-emphasis process is higher and rolls of faster. Not sure which one is truer to spec but the developer of WaveEmph states his program is 0.2 db within spec while I found no documentation on Sox de-emphasis feature. I personally use the ImpulsEmph.wav files included in WaveEmph together with foobar2000 convolver plug-in for convenience. Hope this helps.
sox with --plot option and deemph will plot the filter function.
How is a de-emphasis filter going to interfere with stereo information?
I've no idea. It shouldn't if well implemented (trusted my ears those days). I've been comparing actual SoX and Waves and WaveEmph today and couldn't ABX the results. Either SoX has been improved or my ears have changed for the worse. If anyone's interested (and it doesn't violate the forum rules) I can upload the files and post the links.
Surfi
Either SoX has been improved or my ears have changed for the worse.
The sox changelog says that the deemph stereo bug was fixed in sox-12.18.2 (2006-09-03).
Also the latest doc on the sox website has some words on the accuracy:
The de-emphasis ?lter is implemented as a biquad; its maximum deviation from the ideal response is only 0.06dB (up to 20kHz).
-bandpass
Either SoX has been improved or my ears have changed for the worse.
The sox changelog says that the deemph stereo bug was fixed in sox-12.18.2 (2006-09-03).
Also the latest doc on the sox website has some words on the accuracy:
The de-emphasis ?lter is implemented as a biquad; its maximum deviation from the ideal response is only 0.06dB (up to 20kHz).
-bandpass
this is interesting, could you provide a link? Certainly more than WaveEmphs < 0.2 dB claim...
this is interesting, could you provide a link? Certainly more than WaveEmphs < 0.2 dB claim...
Homepage (I think you know that ...):
http://sox.sourceforge.net/ (http://sox.sourceforge.net/)
Effects manual:
http://sox.sourceforge.net/soxeffect.html (http://sox.sourceforge.net/soxeffect.html)
(do a search for biquad or deemph on that page)
Downloads:
http://sourceforge.net/project/showfiles.php?group_id=10706 (http://sourceforge.net/project/showfiles.php?group_id=10706)
Here's a link to an older (2006) german page that is providing files for testing a deemphasis filter.
http://www.radonmaster.de/robernd/tAFILTER.html (http://www.radonmaster.de/robernd/tAFILTER.html)
Surfi
Tested with Denon Audio Technical CD (1984), Tracks 36-39, WaveEmph is better.
SoX has a deviation of about +0.02/-0.18dB, whereas WaveEmph has +/- 0.03.
This is the sweep with emphasis (scale 2dB, original level):
(http://img109.imageshack.us/img109/6110/oriemphk.th.png) (http://img109.imageshack.us/i/oriemphk.png/)
The result from WaveEmph (scale 0.1dB, shifted to 0dB at 1kHz):
(http://img25.imageshack.us/img25/9804/wavemph.th.png) (http://img25.imageshack.us/i/wavemph.png/)
And from SoX (scale 0.1dB, shifted to 0dB at 1kHz):
(http://img4.imageshack.us/img4/448/sox.th.png) (http://img4.imageshack.us/i/sox.png/)
There is some measurement inaccuracy at lower frequencies.
Never saw someone offering such measurements, thanks for that.
One thing i wonder how accurate the Denon original sweep really is and how it was created. Reading a while back about pre-emphasis even the studio machines didn´t apply the curve always correct and had tolerances.
Out of curiosity it may be intersting if you can get an output of the foobar convolver plugin some recommended or the Waves Q Equalizer itself.
Edit: Here for example someone measured his Sony PCM-601 and it wasn´t 100% accurate by applying de-emphasis http://www.radonmaster.de/robernd/tAFILTER.html (http://www.radonmaster.de/robernd/tAFILTER.html)
even the studio machines didn´t apply the curve always correct and had tolerances.
Probably the majority of cd's with emphasis have been recorded on a Sony PCM 1600/1610/1630 processor (the one that used a U-matic video recorder to store the data). AFAIK the pre-emphasis and de-emphasis circuits were analog and there must have been tolerances between models. Semi pro models like the PCM-F1, 501, 601 and 701 were used too, but not by the major studios.
If you want to remove the emphasis digitally, there are two options IMO:
1) use a 15/50µs digital filter, since that are the theoretically correct values. This however doesn't take into account the imperfections of the ADC.
2) measure the original ADC pre-emphasis curve and apply an inverse version to the signal.
I think it's a mistake to emulate the de-emphasis circuit, since it isn't necessarily identical to the pre-emphasis circuit. It would be interesting to know if the differences are worth worrying about. IIRC Daniel Weiss (pro audio equipment manufacturer) investigated this long time ago and decided to use different values from 15/50µs.
Measured difference of 0.2 dB is not bloody likely to be audible for anything but a test tone in the most sensitive frequency range of human hearing.
Measured difference of 0.2 dB is not bloody likely to be audible for anything but a test tone in the most sensitive frequency range of human hearing.
Tolerances are fine, but IMO their audibility should be verified at the end of the chain. If the chain is unknown, you "might" have a problem .
More importantly, if higher accuracy comes at no or little additional cost/effort, there is no excuse for sloppiness, IMHO.
Don't forget the OP asked for the most "true" way.
One thing i wonder how accurate the Denon original sweep really is and how it was created (...)
Those tracks on the CD are marked D*, which means it was "produced by a high-precision digital 16-bit signal processor" "of Denon/Nippon Columbia" "controlled by a computer and recorded in digital form on the compact disc", "The precision of D* sometimes deviates from the theoretical value, 16-bit, at signal generation (distortion factor: less than 0.1%, level deviation: within +- 0.1dB)"
BTW, I used SoX at the end because it supports flac and the difference is practically not very relevant.
I tried one more thing.
Seems really like the created reference files differ beneeth each other already. For this test i took the reference file from radonmaster.de i linked to before and removed the silent parts.
So i believe the sox claim of being in +/- 0.06dB of the value as long we can´t be sure how exactly the reference files are created.
(http://img691.imageshack.us/img691/8004/deemphpcmpre.th.gif) (http://img691.imageshack.us/i/deemphpcmpre.gif/)
Edit: besides that. while applying this effect the resulting file should be dithered. waveemph doesn´t do that.
Are they still making CDs with pre-emphasis [...]?
Yes, unfortunately. My most-recent CD purchase: http://www.discogs.com/Lifelover-Konkurs/release/1513652 (http://www.discogs.com/Lifelover-Konkurs/release/1513652) from 2008.
If you want to do it digitally the best way is to process the file at 24 bits (or higher) then either leave it at the higher bit depth or dither it back to 16. If you process it at 16 bits you will definitely lose fidelity.
Well, I just tried that. Converted to 24 and SoX-deemph'ed the files.
Guess what, they became > 50 % larger. Not by converting to 24 bits (FLAC compresses away the padded zeroes at little extra space cost), but by the deemphasis filter.
Which is fairly strange, as the actual information is the same, but:
Do not the two facts combined, that (I) the EQ curve only hits appr. 10dB (< 2 bits, or just above 10 %) and (II) the files seem to grow by 50 %, (mildly) suggest that
SoX de-emph introduces digital noise below the 18th bit, which actually should be dithered away for fidelity?(SoX version 14.1.0, libflac version 1.2.1, flac -8 used)
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.
But a parametric EQ shouldn't make it that much harder to compress? Or?
It should make little difference to a lossy encoder, but a lossless encoder doesn't care whether the extra bits are significant or not, it only knows that it is required to reproduce them exactly, and that requires extra bytes.
ANY operation on the data other than scaling by exactly a power of two will fill all 24 bits.
But a parametric EQ shouldn't make it that much harder to compress? Or?
If you keep it at 16-bits, then it's no harder to compress. If you upsample to 24 bits without processing, it's no harder to compress. If you go to 24 bits
and do
any processing, then yes... As pdq said the extra bits are "filled up".
You are working with integer files, and when you go from 16 to 24 bits,
the additional bits are analogous to digits to the right of the decimal point. The extra bits are not used to make your file "louder", they are used to give you more "detail". (That's not
strictly true... 24 bit audio is also used to get more headroom.)
Your
data is integer form, but your scale factors are not... So, when you apply a mathematical operation to your 16-bit integer, you need more precision, or you can round-off.
For example, if you increase a sample value by 1dB, that's a scale factor of ~1.12202. With integer data that's going to require rounding. You will get more precision (less rounding error) with 24 bits. And, the higher precision means it's "harder" to compress.
So long as sox does the internal math at a higher bit depth and then dithers to get back to 16 bits then I would do it that way. As has been discussed over and over, 16-bits is adequate for final delivery. You'll probably notice that your de-emphasized tracks will compress better than their original sources in both lossy and lossless formats.
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and
in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.
In comparison, HDCD decoding (up to 19 point something bits, right?) seems not to add much -- naturally, since it is storeable in 16 bits (plus a description of the algorithm) in the first place.
And the HDCD results indicate that the performance of FLAC does not worsen disproportionally with the word length (unlike what others have reported with other codecs).
@ Greynol:
So long as sox does the internal math at a higher bit depth and then dithers to get back to 16 bits
Well does it? (No, I haven't read the ultimate documentation -- i.e. the source code.)
You'll probably notice that your de-emphasized tracks will compress better than their original sources in both lossy and lossless formats.
Actually, it has struck me that they usually come out a
slight bit worse.
My hypothesis was that the original digital pre-master being 16 bits, and so increasing the treble by 10 dB would maybe lead to excess peak (and some information having to be cut away), maybe not, so the information content of the pre-emphed and de-emphed 16 bits would be more or less the same, but maybe the dithering noise being so idiosyncratic that it takes more space to compress.
But that is at odds with my results as of now.
Well does it?
Don't know. Are you able to tell the difference in a double-blind test between 24 bit output and 16 bit output under normal listening conditions (ie. you don't crank the volume up to 11 during fade-outs)?
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.
Are you suggesting that the encoder try rescaling the data by 2^24 different divisors to see if one of them magically restores the low 8 bits to all zeros? This hardly seems worth the effort.
Besides, in this case we weren't even talking about scaling by a constant, but about applying an equalization. How many possible de-equalizations are we supposed to try in order to reduce the data to 16 bits?
Besides, in this case we weren't even talking about scaling by a constant, but about applying an equalization. How many possible de-equalizations are we supposed to try in order to reduce the data to 16 bits?
Hey, don't ask me, I am only one of those mathematicians who would say that passwordcracking is a trivial issue as there are only countably many combinations
But nevertheless, the process of deemph'ing to 24 bits seems
in practice to take disproportionally much more space than HDCD-decoding to ... at most > 19, which was my "gut feeling" benchmark for what I would expect. My HDCD files seem to encode to something fairly less than 25 % more than the original, but my 24 bit de-emphed files are some > 70 % more.
That's
way more than the increase in wordlength.
("Test corpus": 15 discs, rock/darkwave.)
@ Greynol:
Objection!
Not even hearsay argument
For the ones that want a VST solution to do some de-emphasis i have a suggestion. There must have been a challenge to deliver VST plugins last year at KVR. There is one neat FREE Equalizer plugin from Matthew Lindsay named NCL Phase EQ.
http://www.kvraudio.com/developer_challenge_2009.php (http://www.kvraudio.com/developer_challenge_2009.php)
If you disable all other filters but the HighShelf and set Freq: 5300, Q: 0.48, Gain: -9.4 you get a nice matching de-emphasis curve similar exact as sox. At least with the reference file i have.
I found no way to contact the author myself. He has several presets already into this plugin. Adding de-emphasis could be a gimmick. Maybe someone of you has an idea?
PS: switching to double-precision (2x) made it impossible for me to find a matching curve.
Well, not the most interesting thing but i wanted to mention that while i played with the NCL EQ i throwed in an iTunes 10 de-emphasis curve some recommend here and there.
It was obviously most away from the ideal curve of all the ones i tested. It must be dipping up to -1dB to much. There should the audibility start.
(http://img706.imageshack.us/img706/7442/deemphnclitso.th.gif) (http://img706.imageshack.us/i/deemphnclitso.gif/)
@ DVDdoug and pdq:
The argument on the number of bits is flawed as long as information is not added. Using a lossless compression, then in theory (not in practice, I know) only the information content of the file should matter. If the last 8 bits are filled with a pattern that is easy to predict, then it should add very little.
In this case, there is nothing really "added" information-wise except for a relatively simple algorithm, and in theory it should be possible to compress it down to the size of the original file plus a description of the algorithm.
That's nonsense. In "theory" that's what happens in nondestructive editing, where information about the order and parameters of each applied algorithm is kept. Never wondered why it is called "destructive" editing once such a log isn't kept alongside an original file? It is not only a practical limitation, but especially "
in theory", the problem space, of what you expect a lossless compressor to accomplish, is too large for
any thinkable search algorithm.
For the rest, just reread DVDdoug's post #36. He pretty much nailed it.
That's nonsense. In "theory" that's what happens in nondestructive editing, where information about the order and parameters of each applied algorithm is kept. Never wondered why it is called "destructive" editing once such a log isn't kept alongside an original file? It is not only a practical limitation, but especially "in theory", the problem space, of what you expect a lossless compressor to accomplish, is too large for any thinkable search algorithm.
For the rest, just reread DVDdoug's post #36. He pretty much nailed it.
I dont understand this. Of course, a lossless encoder wont do an explicit blind modelling of the destructive processing done to its input. I took it more as an argument about information or entropy.
A lossless encoder contains a prediction filter that tries to predict future samples, right? Inserting new, unpredictable information (noise) means that the encoder cannot predict information, and the residue is larger. However, doing static, linear filtering, this should only concolve up to a new (larger) optimal prediction filter, and depending on the length of the prediction done by flac, as well as the original source autocorrelation, could work?
"Entropy" and "information" is beginning to be vague concepts for me.....
-k
As said, it's already all in post #36.
Digital filtering involves rounding. Rounding of 16 bit values at 24 bit precision will fill the full 8 additional bits with new information, even if a given filter was very simple. That new information might not be significant, i.e. noise, but that's up to a lossy encoder to decide. A lossless encoder has to preserve it.
As said, it's already all in post #36.
Post#36 does nothing to explain this. It merely states the obvious, that the 8 lsb will typically be non-zero when doing non-trivial operations on a 16-bit signal wrapped into a 24-bit container.
Digital filtering involves rounding. Rounding of 16 bit values at 24 bit precision will fill the full 8 additional bits with new information, even if a given filter was very simple. That new information might not be significant, i.e. noise, but that's up to a lossy encoder to decide. A lossless encoder has to preserve it.
This does not make sense. A filter does not create information (look up Shannon).
If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.
There are surely practical constraints that can prevent this (buffer sizes, processing power, truely random dither etc), but no-one has elaborated on those.
-k
This does not make sense. A filter does not create information (look up Shannon).
That's true under the condition that we have machines operating on real numbers, which we don't. Instead we have samples of finite width, which when processed, can to lead to new samples of theoretically infinite width, which we represent by rounding.
If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.
Maybe in math operating on real numbers, but not
universally for an automaton processing samples of finite precision with finite storage space. The information might just be lost in some (if not most) cases. Yes, in other cases it might be recoverable. But claiming reconstructability "in principle" is definitely wrong.
Any re-quantization after a non-trivial operation destroys information by adding new information (quantization artifacts). By increasing the word length over the original length the information loss of re-quantization can be reduced. In the cited case the cost of that reduction of loss is 8 bits.
It is not the cost of filtering, but the cost of reducing re-quantization artifacts which take up the 8 lsb.
Now we are getting somewhere. So I should have stated "an ideal filter" in my quick-and-dirty "in principle". And someone should have stated above that they referred to the post (ideal) filter quantization.
Lets say the filtering is carried out ideally (double precision float or whatever), the input is 16 bits and the output is rounded to 24 bits. The proper explanation for why flac cannot compress the 24-bit signal as well as the 16-bit signal is not that the 8 lsb are non-zero, but that flac contains a linear predictor that cannot model a (non-linear) 24-bit truncation? This truncation/quantization could be modelled as a random process but it really is not (just listen to quantization error after 4-bit quanttization).
We dont know (or explanations have not been provided) whether the information is really lost, or if some imaginary perfect non-linear predictor in flac could solve the general case outlined.
-k
I think it's far simpler than you think.
Consider the 16-bit file as 24-bits with the 8 LSB being zeros.
You filter it, and the 8 LSB become none zero. Aside from clipping, dithering, rounding etc, the information content has changed very little, if at all.
Why is the bitrate with FLAC so different? Easy:
1. FLAC has an absolutely perfect (or near as damn it perfect) way of expressing the 8 LSB zeros efficiently.
2. FLAC has a very imperfect method of expressing the filtering efficiently. In fact, it makes no special attempt to, since in the general case there is no utility in doing so, and even in this specific case, the prediction (to give a perfect result, recovering 8 zeroed LSBs) would have to be far more accurate than anything else it ever achieves.
If you take the 24-bits with 8 zero LSBs, and add a DC offset of 000000000000000010101011 to it, thus breaking the "perfect" trick FLAC can use on those 8 zero LSBs, you'll probably find the compression differential between that and the filtered version narrows dramatically (unless FLAC is smart enough to remove it - I haven't tried). This is despite the DC offset carrying exactly 8 extra bits of information for the entire file over and above the 24--bits with 8 zero LSBs.
FLAC isn't a form of artificial intelligence trying to find the absolute smallest lossless representation of a given set of data, including reverse engineering any algorithms that may have "bloated" that data; it's just trying to do a reasonable job in a reasonable time.
Shameless plug: if you want to throw away bits that don't contain "real" information, try lossyWAV But then it won't be lossless.
Cheers,
David.
2. FLAC has a very imperfect method of expressing the filtering efficiently. In fact, it makes no special attempt to, since in the general case there is no utility in doing so, and even in this specific case, the prediction (to give a perfect result, recovering 8 zeroed LSBs) would have to be far more accurate than anything else it ever achieves.
Seems that you agree on my previous proposed explanation:
If the 8lsb can be "created" as a function of the last N input samples, and this function is held constant over a file, then a predictor looking for autocorrelation can in principle find that function (or the convolution of the original source spectrum and the filter). Once you have that function you can do predictions and transmitt only the prediction residue.
There are surely practical constraints that can prevent this (buffer sizes, processing power, truely random dither etc), but no-one has elaborated on those.
So perhaps the linear prediction (somehow the words Yule-Walker flash in the back of my head?) is inperfect. Perhaps it does only the 8 msb. Perhaps it has too short a buffer. Perhaps it does only a partial search through candidate coefficients.
If you take the 24-bits with 8 zero LSBs, and add a DC offset of 000000000000000010101011 to it, thus breaking the "perfect" trick FLAC can use on those 8 zero LSBs, you'll probably find the compression differential between that and the filtered version narrows dramatically (unless FLAC is smart enough to remove it - I haven't tried). This is despite the DC offset carrying exactly 8 extra bits of information for the entire file over and above the 24--bits with 8 zero LSBs.
FLAC isn't a form of artificial intelligence trying to find the absolute smallest lossless representation of a given set of data, including reverse engineering any algorithms that may have "bloated" that data; it's just trying to do a reasonable job in a reasonable time.
I was thinking about that. If other trivial inserted datasets into the 8 lsb also cause a large increase in filesize, then it seems to suggest that flac simply isnt really clever wrg the 8 lsb of a 24 bit stream. There may be very good reasons why it isnt.
Throughout this discussion I have had a crude mental model of lossless codecs where they try to predict future samples from a historical buffer, transmitting only the (slowly varying) model + the prediction residue (further compressed using entropy coding). If this is indeed a valuable reference for this kind of high-level discussions, it really boils down to the statistics of the input signal and the capabilities of the prediction, doesnt it?
-k
This does not make sense. A filter does not create information (look up Shannon).
That's true under the condition that we have machines operating on real numbers, which we don't. Instead we have samples of finite width, which when processed, can to lead to new samples of theoretically infinite width, which we represent by rounding.
I fail to see how this should lead to a
dramatically worse performance. If rounding appears as uniformly random, it is a noise component which cannot be compressed, fair enough -- but are you altering the 24-16=8 bits? No. (Are you ever rounding off more than 1 LSB? That's not a retorical question, I don't know the answer.) And the finite-wordlength constraint merely means that there are more functions fitting, modulo the rounding.
So in principle, it should be possible to compress to something fairly near original size (17/16 if roundoff and dither is in the LSB only, right?).
Now my original inquiry was due to the observation that files became
> 50% larger -- up to 70%, actually. That corresponds to (more than!) a fully 24 bit recording (27 bits in the 70 % case).
Now assume you have
(A) 24 bits of music (resp. 27), FLACed
vs
(B) the file in (A), which you crop down to 16 bits (discarding 1/3 of the information), adding a little dither, applying a filter (adding no information although representing it in a 24 bit file) and rounding off. Then you FLAC it.
Is it at all reasonable that file (B) should be as large as file (A)?
Yes, but the design is such that it works "best" with the vast majority of content out there - which means it might not do very well on a given special case. The predictor looks at the sample values as a whole (i.e. all 24-bits) - it's never looking at the 8LSBs specifically (or the 16 MSBs!) and seeing if they can be predicted from anything.
Plus I've read (don't know if it's true) that the design of FLAC isn't especially focussed on 24/96 audio. Certainly it's a practical implementation of lossless coding - it's not meant to reach for perfection in terms of compression ratio.
FWIW, IIRC, the Wavepack author said that audio which had been created from another source (e.g. by simple normalisation, or I suppose by filtering - I think the specific case he mentioned was normalisation without dither which meant certain sample values would be completely unused), where that other source would have required fewer bits to store, wasn't unheard of - but he didn't think it was worth building something to reverse engineer the transformation. Sometimes it would noticeably reduce the bitrate, but the encoding effort would slow things down too much. (with apologies to David if I've mis-remembered this! I searched but couldn't find the quote.)
Cheers,
David.
(Are you ever rounding off more than 1 LSB? That's not a retorical question, I don't know the answer.)
Rounding, in the strict sense of the word*, only affects the 1 LSB. But when you process at 16 bit values at 32 or 64 bit precision, you get true 32 and 64 bit results. Storing them in 24 bit will give 8 unique bits over 16 bit and not some trivially correlated pattern.
The thing is, you guys asked for a
very special case handling and why the predictor isn't able to detect the pattern automagically. It is just too special to scan for specifically. We also can't use an universal predictor covering all possible cases. A complete implementation would probably itself be unpredictable (halting problem) or at least have unpractical exponential costs. As 2Bdecided put it very nicely: "FLAC isn't a form of artificial intelligence"
That FLAC has considerably worse performance at 24 bit than 16 bit is true, but not necessarily related to its effectiveness in the "reconstruct n LSB as a function of the l-n MSB in the last x samples" special case.
*Instead of saying to 'round' a 32 bit value to 24 bit, as I may have done, the term 'word length reduction' would have been more appropriate.
I guess that whatever per-channel correlation is in a music file is generally mainly to be found in the 8msb (no matter what the bit-depth happens to be)?
-k
Why exactly 8?
Rounding, in the strict sense of the word*, only affects the 1 LSB.propriate.
go on, think that one through again!
(or round 8999.9 to the nearest integer and see how many of the numbers change).
You could just say "truncate". Apart from this special case, it's often just an academic difference with audio, since rounding = add 0.5 then truncate. A 0.5 LSB DC offset is usually irrelevant - except here I suppose.
Back to the original topic, if you want a lossless but efficient method, I'm sure it's possible to calculate how many bits you need to store optimal 16-bit + de-emph without creating a detectable difference. I bet 20 is enough, in which case, dump the last four bits. I'm not even going to say dither, though I suppose you could. Either way, it'll bring the FLAC bitrate down.
You could say that 20 isn't as good as 24. True. 24 isn't as good as 32. 32 isn't as good as 64. But you have to stop somewhere. 16 is already enough IMO, but if you're using lossless you might want some extra headroom, no matter how irrational or at least unimportant.
Cheers,
David.
Why exactly 8?
Could just as well say 1 or 15. The further into the noise/dither floor you are trying to predict, the harder it is, right?
-k
go on, think that one through again!
You're right. Rounding can affect all digits of a value's representation.
However, rounding never affects the actual, represented value by more than what the LSB represents.
Why exactly 8?
Could just as well say 1 or 15. The further into the noise/dither floor you are trying to predict, the harder it is, right?
That's absolutely a point (with a reservation below), if it is true. It is an easy test (which takes some time): gather a test corpus of 16 bit recordings (or, if you can find a representative test corpus of 24-bit recordings). Truncate down by 1 bit, 2 bits, ... down to some practicality bound (8 is a round figure? Maybe even 4.). Compress. Plot compressed-file size as function of bits. Does it look linear?
(One could do the same with dithering, but that should simply shift the effective # of bits, right? 16 bits dithered down to 15 would be roughly equivalent -- in information content -- to somewhere between 15 and 16 of the original 16? So that there would be expected one bit difference between 16-to-15-with-dithering 16-to-14-with-dithering?)
But then the reservation: If the file is originally 16 bits, and becomes 24 bits just by padding with zeroes and applying a filter, then the last 8 bits are not in the noise floor, are they?
FWIW, I checked this with multiple lossless encoders. No surprises in terms of relative performance -- relative filesizes about what you would expect in a 16-bit test. So all compress about equally (relative to their assumed quality) bad -- they all returned file sizes greater than the original 16-bit WAV.
I.e.: With the de-emphasis algorithm and a single flag indicating application of it, then even WAV -- without any compression whatsoever -- would outperform today's state-of-the-art lossless algorithms (on this fairly hardrock-ish test corpus, that is). That's the state of today's art
Procedure:
The 16-bit files were converted to 24 bits de-emphed by SoX. Each album converted to one single WAV file by foobar2000. Test corpus with 24-bit file sizes below, total 11 286 552 576 bytes in the file (11 286 517 144 according to foobar2k, strange since it is not divisible by 3 ...).
Then each wav file was compressed with a range of encoders (Monkey's and TAK by their GUI applications, ofr.exe by CLI, while Foobar handled FLAC (1.2.1) and WavPack -- the latter unbearably slow, taking hours.) In order of filesize:
7 761 846 272 bytes - flac -8
7 672 954 880 bytes - WavPack high, x5
7 629 369 344 bytes - Monkey's extrahigh
7 582 380 032 bytes - TAK -p4
7 533 199 360 bytes - ofr extranew
For comparison, the original 16 bit signal approximated by removing 1/3
11 286 552 576 bytes*2/3= 7 524 368 384 bytes (file size)
11 286 517 144 bytes*2/3= 7 524 344 763 (minus a third. Audio size)
Test corpus with 24-bit wav filesizes:
. 779 289 380 Backstreet Girls - Boogie Till You Puke
. 607 080 644 Black Sabbath - Black Sabbath [Castle orig
. 679 535 180 Black Sabbath - Black Sabbath, Vol. 4 [Castle orig
. 635 569 244 Black Sabbath - Master of Reality [Castle orig
. 704 629 844 Carnivore - Retaliation
1 099 159 028 Ebba Grön - Ebba Grön, 1978-1982
. 762 788 924 In Slaughter Natives - Enter Now the World
. 814 826 924 Leonard Bernstein - West Side Story
. 938 924 324 Lifelover - Konkurs
. 992 003 084 MZ.412 - Burning the Temple of God
. 933 198 380 MZ.412 - In Nomine Dei Nostri Satanas Luciferi Excelsi
. 803 773 700 Ordo Equilibrio - Reaping the Fallen...The First Harvest
. 864 977 444 Raison d'Etre - Prospectus I
. 670 761 044 Roger Waters - The Pros and Cons of Hitch Hiking
For reference:
7 761 846 272 bytes: flac -8 (24 bits)
4 043 931 648 bytes: Same, but set to 16 bits output (no dithering).
The first 16 bits take up 52.1% of the filesize. The last 8 bits (probably with dithering?): 47.9%.
I haven't found much info on emphasis. From what I understand, it's some kind of tag on the audio file that tells the CD amplifier to amplify this song by this much.
Close?
I haven't found much info on emphasis. From what I understand, it's some kind of tag on the audio file that tells the CD amplifier to amplify this song by this much.
Close?
No, it dictates application of a certain EQ curve which attenuates the treble.
Your description fits ReplayGain, which exists for file formats like FLAC and WavPack.
I haven't found much info on emphasis. From what I understand, it's some kind of tag on the audio file that tells the CD amplifier to amplify this song by this much.
Close?
No, it dictates application of a certain EQ curve which attenuates the treble.
Ok, still not understanding it. Any recommended reading?
Have you tried our wiki? It's generally the first place to look...
http://wiki.hydrogenaudio.org/index.php?title=Pre-emphasis (http://wiki.hydrogenaudio.org/index.php?title=Pre-emphasis)
I haven't found much info on emphasis. From what I understand, it's some kind of tag on the audio file that tells the CD amplifier to amplify this song by this much.
Close?
No, it dictates application of a certain EQ curve which attenuates the treble.
Ok, still not understanding it. Any recommended reading?
Pre-emphasis and de-emphasis has been around a long time and is used in FM radio, analog TV audio, analog tape and LPs. The theory is since the high frequency components are typically lower amplitude than the lows and mids, we can use a little of the unused real estate by boosting the highs on the transmit/record side and lower them at receive/playback. Any noise introduced after pre-emphasis will be attenuated along with the excessive highs. This restores the response and reduces the noise.
When CDs were introduced some thought pre-emphasis would be a good idea to reduce quantizing noise and while a few discs were made with pre-emphasis, most were not and for many years, none. On the CD the pre-emphasis is an analog high boost ahead of the A-D converter. This also sets a flag bit on the CD to activate the filter during playback if needed. During playback an analog filter after the DAC restores the response and reduces he quantizing errors. If you extract the digital audio from the disc during a rip session, you now have boosted highs but no analog filter to correct it. In theory the analog filter after the DAC is the best but in practice digitally processing the stream is certainly acceptable and possibly more accurate as the digital filter is not subject to 5% or even 1% component tolerances. They tell me Sox works well and I'm happy with the CoolEdit/Audition filter settings I used - all of 2 times.
G²
Have you tried our wiki? It's generally the first place to look...
http://wiki.hydrogenaudio.org/index.php?title=Pre-emphasis (http://wiki.hydrogenaudio.org/index.php?title=Pre-emphasis)
I looked through the main topics under technical but didn't find anything.
Thanks.
[misread]
I looked through the main topics under technical but didn't find anything.
It can be found under Signal Processing (http://wiki.hydrogenaudio.org/index.php?title=Category:Signal_Processing) which is under Technical, though I found it simply by typing "preemphasis" in the search field (or "pre-emphasis", it doesn't matter either way) .
Are they still making CDs with pre-emphasis [...]?
Yes, unfortunately. My most-recent CD purchase: http://www.discogs.com/Lifelover-Konkurs/release/1513652 (http://www.discogs.com/Lifelover-Konkurs/release/1513652) from 2008.
Argh. Cthulhu has risen from R'lyeh again, as reported here: http://forum.dbpoweramp.com/showthread.php...ll=1#post121273 (http://forum.dbpoweramp.com/showthread.php?13900-Pre-emphasized-CDs&p=121273&viewfull=1#post121273) .
The current 'newest pre-emphasis CD' to my knowledge is http://www.discogs.com/Marc-Almond-Michael...release/3066984 (http://www.discogs.com/Marc-Almond-Michael-Cashmore-Feasting-With-Panthers/release/3066984) , released June 2011.
Please excuse my posting in this old topic. I did not find any additional information on this topic, will try to ask here.
@ Porcus, would be interesting to know the outcomes of your investigation on how to properly de-emphasize CD image using Sox.
Would you recommend up-convert to 24-bits before de-emphasizing with Sox? Did any one was be able to find if Sox does the internal math at a higher bit depth and 16-bit to 24-bit up-convert would not be required?
Don't remember. I ended up deleting my de-emph'ed files and retrieving my CD rips, when there arrived a way to de-emphasize on-the-fly upon playback with foobar2000 (the foo_deemph component (https://hydrogenaud.io/index.php/topic,99394.0.html)).
(And I keep them as WavPack; I wanted to distinguish them by file format, should I accidentally lose the PRE_EMPHASIS tag - there I got an excuse for calling myself a WavPack user. They are not so distinguished out now as musicians started to put 32-bit floating-point PCM from their workstations directly out on the web (like one song on this EP (http://timeisdivine.com/soz/soz-014.php) and a one song off Soundcloud here (https://soundcloud.com/termo-records/silence-instead)) and I'd rather just keep them as I downloaded them - that means WavPack or OptimFrog, which in practice means WavPack. But I digress.)
@Porcus, thanks for clearing things up.
I will keep de-emphing with Sox, no 24-bit up-convert before, and burn them to CD, as my CD player Cambridge Azure 840C does not perform the de-emphasis procedure.