Q: Will all (at least all widely used) 'lossless' audio encoders abort with the appropriate error message if the user tries to feed unsupported audio? Or is there any which will discard channels, downsample, crop bits or just produce nonsense?
The reason why I ask, is that this forum fairly often gets the n00b question of whether 'lossless' is indeed lossless -- and get no more than a simple 'Yes' (or 'Yes, thread closed'). The devil in the detail is hardly ever mentioned, and most users will anyway never encounter that mismatch. But if the situation is 'in the few instances where the usual answer is inadequate, the encoder will warn the users rather than eating audio information', it would certainly be A Good Thing.
The devil in the detail is hardly ever mentioned, and most users will anyway never encounter that mismatch.
There just is no devil in the detail...
I don't know what you imagine digital audio is like. Encoders don't "eat" audio information and silently leave on the plate what they don't like. Every stream has a format and every encoder a set of formats it accepts. For your case, a developer would have to
intentionally convert the data before feeding it and
intentionally not inform the user about it. That is very unlikely and if such bad practice is found, you should rather think about using this developer's software than thinking about whether there needs to be a disclaimer attached to all claims of losslessness.
ders don't "eat" audio information and silently leave on the plate what they don't like. Every stream has a format and every encoder a set of formats it accepts. For your case, a developer would have to intentionally convert the data before feeding it and intentionally not inform the user about it. That is very unlikely ...
Unlikely in the case of developers who are members of HA, but can't you imagine a commercial company dedicated to making life easy for consumers who might silently modify inputs their app couldn't handle to give the consumer the best possible outcome and not bother them with scary techy details? I can imagine three who
might do that sort of thing (Apple, Sony and perhaps Microsoft). I have of course, not the slightest suspicion that any of them have done anything like that, but OP's question seems to be something it is worthwhile seeking reassurance on.
edit: omission
I agree that stuff like this is happening in the field of lossy audio and video compression. Sometimes such can indeed improve average customer experience. The same is not true for introducing loss into something labeled 'lossless'. I also don't know a case where something like this would have happened.
With foobar2000 1.1.1 and flac encoder 1.2.1b:
- Source file = wav 32-bit (floating-point)
- Output file format = flac
- Output bit depth = either "32-bit" or "Auto"
Due to the specifications of the flac format two identical
24-bit flac files are created, without an error message or a lead on the bit depth reduction. foobar's console reports: "Track converted successfully."
Conclusions:- Porcus posed a legitimate question.
- It's the duty of the user to check which specifications are supported by the encoder he uses.
FWIW, AFAIK, neither flac.exe nor foobar2000 warns the user when encoding floating-point lossless sources to FLAC. (FLAC does not support floating point so an implicit conversion to 24-bit fixed point is necessary.)
EDIT: LOL Robertina!
With foobar2000 1.1.1 and flac encoder 1.2.1b:- Source file = wav 32-bit (floating-point)
- Output file format = flac
- Output bit depth = either "32-bit" or "Auto"
Due to the specifications of the flac format two identical 24-bit flac files are created, without an error message or a lead on the bit depth reduction. foobar's console reports: "Track converted successfully."
Conclusions:- Porcus posed a legitimate question.
- It's the duty of the user to check which specifications are supported by the encoder he uses.
Since floating point numbers are by definition approximate, I don't think theres an expectation that processing will be lossless.
Since floating point numbers are by definition approximate, I don't think theres an expectation that processing will be lossless.
Yes.
This "floating-point" in round brackets only should indicate that I used both a 32-bit and a 32-bit floating point file for my test.
Since floating point numbers are by definition approximate, I don't think theres an expectation that processing will be lossless.
saratoga,
after my post above I began to brood over your statement and I would like to apply to you for help:
If I at first convert a 32-bit floating-point WAV to a WavPack file, then convert that wv-file back to a WAV file and compare the latter with my original wav file, foobar's Binary Comparator (http://www.foobar2000.org/components/view/foo_bitcompare) component says:
All tracks decoded fine, no differences found.
Does that result not mean that a lossless conversion of a floating-point file can be expected or is there an error in my logic?
foobar2000 v1.1.1, WavPack encoder v4.60.1
Does that result not mean that a lossless conversion of a floating-point file can be expected or is there an error in my logic?
Floating point is only approximate, but still very accurate (particularly if your source wavpack file is relatively low precision compared to the intermediate floating point). Depending on the CPU, precision and software it may or may not give you the same result.
Thank you for your clarification, saratoga.
----------------------------------------
Since flac files can have 24-bit maximal and hence all types of 32-bit source files (floating- and fixed-point) will lose bit depth on the conversion process, from my point of view it would be appropriate to complete at least the flac WIKI (http://wiki.hydrogenaudio.org/index.php?title=Flac) with an appropriate hint, as long as neither the encoder nor the software used as a graphical front-end show any warning.
I am glad that I didn't rely on that "lossless is lossless" statements as mentioned by Porcus when I began to convert my 32-bit audio files to a lossless format, but did my own tests where I realized this situation. So I think it was a good point, Porcus, to bring this up.
Since floating point numbers are by definition approximate, I don't think theres an expectation that processing will be lossless.
Disagree totally here. We are telling users that 'lossless' means ... well, actually,
lossless. By assumption, we are talking about users who don't know the meaning of the term 'lossless audio format'. Users who cannot be expected to have even the faintest idea of the concept 'floating-point number'.
Obviously, they cannot be expected to know anything about format internals, or encoder/decoder internals, and all they are told is,
yes goddammit, what part of 'lossless' did you fail to understand?, please accept as a universal truth (thread closed and STFU).
encoder/decoder internals
Then, arrgh, decoder bugs: http://www.hydrogenaudio.org/forums/index....showtopic=84101 (http://www.hydrogenaudio.org/forums/index.php?showtopic=84101)
I stand corrected by practice, it seems.
I still think that Foobar or the Flac encoder is far from behaving as it should, here, and that this rather belongs into a bug report than into the Flac Wiki. In programming there is no implicit Float->Int conversion. So an explicit conversion is happening here - this is where the information loss occurs (not in the encoder) - and the developer is not telling you about it! It's only slightly less worse in the 32 bit integer case, where the 8 LSB are probably just omitted without checking whether they are empty.
Float preservation is hard to accomplish. It's not impossible, though. Developers must just not rely on a specific hardware's float capabilities for the correctness of their program.
Then, arrgh, decoder bugs: http://www.hydrogenaudio.org/forums/index....showtopic=84101 (http://www.hydrogenaudio.org/forums/index.php?showtopic=84101)
This is seems to be a common range offset bug (start counting from 0 instead of 1 and vice versa). While this is certainly annoying and a sign of bad QA, the same could basically happen to any WAV implementation. It is not specifically related to the math of lossless compression.
I still think that Foobar or the Flac encoder is far from behaving as it should
I think this too, but why would it be wrong to mention in the FLAC Wiki that this file format can have 24-bit max, but accepts 32-bit sources as input files and that the encoder doesn't provide a warning in that cases? I admit that I don't understand that part of your post starting with "this is where the information loss occurs (not in the encoder) - and the developer is not telling you about it!"
If the information loss doesn't occur in the encoder: whose developer do you mean then with "and the developer is not telling you about it"? I am really confused about that but I am far away from having your technical knowledge, so sorry for bothering you with that.
Float preservation is hard to accomplish. It's not impossible, though. Developers must just not rely on a specific hardware's float capabilities for the correctness of their program.
So I would like to hear Bryant's (David's) comment on this. I am using several computers (= different hardware) and converting mainly 32-bit floating-point wav files to WavPack and on all these PCs there never has been found a difference between an original source file and a WavPack file that I converted back.
I think this too, but why would it be wrong to mention in the FLAC Wiki that this file format can have 24-bit max, but accepts 32-bit sources as input files and that the encoder doesn't provide a warning in that cases?
FLAC doesn't convert 32-bit input to 24-bit. Foobar2000 does this.
BTW, WavPack has -p switch: "practical float storage (also affects 32-bit integers, no longer technically lossless)".
I think this too, but why would it be wrong to mention in the FLAC Wiki that this file format can have 24-bit max, but accepts 32-bit sources as input files and that the encoder doesn't provide a warning in that cases?
You are right. That would be valuable information, until it is fixed (if ever).
I admit that I don't understand that part of your post starting with "this is where the information loss occurs (not in the encoder) - and the developer is not telling you about it!"
With "encoder" I mean the actual encoder within the code. There are additional components within a converter program, for example, file I/O, parsing, user interface, etc. All encoders can only interpret a limited set of input formats. If the format of your input doesn't match any of those, it has to be converted first. Sometimes a developer can rely on implicit conversion, for example when you feed 24-bit integer samples into an encoder that works with 64 bit integer values internally. They get just padded with zero. In the case of float and FLAC you have a completely incompatible input format, which has to be converted
explicitly before being fed to the encoder. The developer must have included such functionality, else it wouldn't silently "work", but doesn't tell you about it when it happens. Since there is information loss, this is bad practice.
If the information loss doesn't occur in the encoder: whose developer do you mean then with "and the developer is not telling you about it"?
Good question. To possibilities: 1. libFLAC's API accepts float samples without throwing an exception and applies the lossy conversion internally. 2. libFLAC does not accept float samples and Foobar converts before feeding the samples to libFLAC. I'll have a look it later.
Edit: lvqcl has already provided the answer.So I would like to hear Bryant's (David's) comment on this. I am using several computers (= different hardware) and converting mainly 32-bit floating-point wav files to WavPack and on all these PCs there never has been found a difference between an original source file and a WavPack file that I converted back.
In practice that's usually not an issue. A program can check for the precision of the available hardware floating point units and only use them, when they are sufficient. If not, alternative, higher precision routines are used, which are slower. Developers often can rely on libraries, which do these checks behind the scenes and can just be used transparently.
Floating point is only approximate
It's a digital file. In which way is a copy of that file "approximate"? What do you think the words "copy" and "lossless" mean in the context of binary data?
Cheers,
David.
Thank you for your full answer [a href="index.php?act=findpost&pid=737922"][{POST_SNAPBACK}][/a], googlebot.
FLAC doesn't convert 32-bit input to 24-bit. Foobar2000 does this.
Edit: lvqcl has already provided the answer.
Does the flac encoder show a hint when it is controlled via command line parameters (= without the possibly influence by an additional software like foobar) if somebody tries to feed files with more than 24-bit or floating-point files?
f32.wav: 32-bit floating point.
f32.wav: ERROR: unsupported format type 3
i32.wav: 32-bit integer
i32.wav: WARNING: legacy WAVE file has format type 1 but bits-per-sample=32
i32.wav: ERROR: unsupported bits-per-sample 32
i32w.wav: 32-bit integer, waveformatextensible
i32w.wav: ERROR: unsupported bits-per-sample 32
By the way, WavPack accepts 32-bit integer files, but foobar2000 converts 32-bit integers to 32-bit floats. So it isn't possible to losslessly pack 32-bit int WAVs with foobar2000.
By the way, WavPack accepts 32-bit integer files, but foobar2000 converts 32-bit integers to 32-bit floats.
You are right, lvqcl.
But if I compare these two files (the source 32-bit integer wav file with the back-converted and now 32-bit floating-point wav), foobar's Binary Comparator doesn't find any differences between them ("No differences in decoded data found").
That's probably because foobar2000 converts everything to 32-bit float during decoding. So Binary Comparator compares 32-bit float data with 32-bit float data.
I am not really happy with what I have learned in this thread so far...
The devil in the detail is hardly ever mentioned
It seems so, indeed.
32 bit float values can safely store 24 bit int samples. Could anyone enlighten me why we should care about 32 bit int samples (besides the generality that losses should be reported for processes labeled 'lossless')? No DAC in the world can produce an equivalent analog output.
Floating point is only approximate
It's a digital file. In which way is a copy of that file "approximate"?
The ones and zeros making up a floating point file do not necessarily have a consistent meaning across different CPUs and even operating systems, and the output of identical floating point operations on different systems is not required (or even expected) to be identical. The values represented in each floating point sample are therefore literally approximations of an (unknown) true value that is not recorded.
What do you think the words "copy" and "lossless" mean in the context of binary data?
I think lossless means you should be able to save a value and then load it latter in a machine independent manner with a guarantee that you will get an
identical value out latter. I'm skeptical that this is possible for a user mode program like flac short of running it in a virtual machine or reference floating point emulator. I think the simple process of converting between different machine representations of floating point values is likely to be lossy. Although someone more familiar with IEEE754 compliance may know better then I.
AFAIK the meaning is well defined. Just the precision of operations may differ from machine to machine. But this is only an issue when you want to exploit a specific machine's performance benefits regarding FP calculation. Then you need to know your minimum requirement. Limited hardware doesn't prevent FP math of arbitrary precision, though. It's all a question of effort.
I think lossless means you should be able to save a value and then load it latter in a machine independent manner with a guarantee that you will get an identical value out latter. I'm skeptical that this is possible for a user mode program like flac short of running it in a virtual machine or reference floating point emulator.
It is definitely possible for 7zip or winrar.
I think lossless means you should be able to save a value and then load it latter in a machine independent manner with a guarantee that you will get an identical value out latter. I'm skeptical that this is possible for a user mode program like flac short of running it in a virtual machine or reference floating point emulator.
It is definitely possible for 7zip or winrar.
I think you have not understood what you have quoted. The point is subtle, so I will try again. In order to zip a floating point value you would first have to save it to a binary (integer) value. To unzip it you would have to reverse this process. In order to recover the original floating point value you must make a number of assumptions about how the hardware and operating system will convert its initial internal representation of a floating point value to an integer, and then how the decompressing machine (which may not be based on the same hardware or even using the same algorithm) will reverse this process.
To put this another way: lets say I give you a double precision PCM stream stored in IEEE754 format. I don't tell you if the machine that created it was operating in double or extended precision mode, if it was intel or amd, or if it was even x86. I then give you the same file, except with the volume doubled (exponent incremented by 1) by the first machine. Can you get a CPU with ARM vfp unit and an Intel Pentium 2 to take the former file, load it into FP registers, double the volume, and then (losslessly) generate the latter? In theory you would think yes, since doubling the volume just increments the exponent without rounding, but that assumes the internal representation of all values and operations is the same. This is a serious question, I really have no idea what would happen on a few different CPUs . . .
Both compressing and decompressing can be performed using only integer operations (bit shifts, bitwise AND, etc) or using bitfields:
struct float32
{
int mantissa: 23;
int exponent: 8;
int sign: 1;
};
Certainly doing anything meaningful with floating point data requires a "relatively" (heh) inexact measure of equality. That is, there is usually some nonzero amount of relative error to every arithmetic operation you can do with a float. That's part and parcel of using such a format. While it is reasonable to ignore that point when comparing float WAVs, that effectively limits the number of lossless operations one can perform on those WAVs to bit shifts, whereas with fixed point WAVs that would instead be any add followed by an equal-valued subtract (and would explicitly exclude bitshifts extending through the MSB/LSB). IOW, the context here admits two notions of "equality" for float WAVs, not one.
The present discussion, as defined by the OP, is explicitly and solely concerned with maintaining 'absolute' lossless encoding of floating point WAVs, as defined by bit equality (and not by equality to within a relative error or epsilon). That this may not have anything to do with audible sound quality differences (and it likely doesn't!) is entirely besides the point and off topic, for the same reasons why we can have honest discussions about whether or not one sound card or another supports Kernel Streaming or otherwise can send a bit-perfect output signal, even while we might not actually claim that ought to improve audible sound quality.
I recall years ago reading (on this board) about non-audio information that WAVPACK would preserve, which FLAC would not. I can't remember if it was some PCM metadata that is only used in a recording studio, or what it was exactly.
It seemed more important to those who were archiving the cd as one file with a cue sheet.
FLAC has --keep-foreign-metadata switch:
If encoding, save WAVE or AIFF non-audio chunks in FLAC metadata. If decoding, restore any saved non-audio chunks from FLAC metadata when writing the decoded file.
That's not really on-topic but...
It was with wave files that contain metadata. Generally not (more likely never) applicable to those who embed CUE sheets in lossless images, unless you store a CUE sheet in your wave images (if so tell me how you do this!). flac does announce the error and has since added a switch that can be used to keep this information, IIRC.
FLAC has --keep-foreign-metadata switch:
If encoding, save WAVE or AIFF non-audio chunks in FLAC metadata. If decoding, restore any saved non-audio chunks from FLAC metadata when writing the decoded file.
Yeah, that's it. I think at one time that switch didn't exist, or maybe people just didn't know about it at the time. Either way, that is one example of some (non-audio) data that could be lost if the format doesn't support it, or if the user doesn't know to use a required command line switch.
Edit: verbage
or if the user doesn't know to use a required command line switch.
flac does indeed tell you to use the switch when this happens (also keeping on-topic with this off-topic part of the discussion; see the part that Porcus put in italics).
Even though I originally asked about encoders, I do think that decoding is highly on-topic as well -- if you do get differences when transcoding it does not really matter which of the steps is the culprit.
As for fb2k specifics: Recall that the wiki warns the user against the LA fb2k plugin: http://wiki.hydrogenaudio.org/index.php?title=Lossless_Audio (http://wiki.hydrogenaudio.org/index.php?title=Lossless_Audio) .
BTW, I don't mind spinning this thread off with metadata discussions, but I suppose that it is fairly well known that you cannot safely throw unicode chars into [enter even text file formats here]. (I do think that the 'this couldn't be audible!' argument belongs to the lossies department and not to this thread, though.)
I think you have not understood what you have quoted. The point is subtle, so I will try again. In order to zip a floating point value you would first have to save it to a binary (integer) value.
That is incorrect. A floating point value can be saved in binary notation without loss. Thus it can be zipped and unzipped without problem. I think you are confusing the floating point numbers with real numbers. Floating point numbers are approximations of real numbers, just in the same way that you can approximate the root of 2 by writing down as many digits as you can fit on a sheet of paper. The representation of floats is well standardized. The only thing differing between machines, as I have written already, is the precision of algorithmic
operations and at some special case handling. But this is really not a big deal. The results of two machines after a series of algorithmic operations may differ by some insignificant digits, if you run your operations on the bare bone hardware without additional checks. If consistency is required, most development platforms provide standardized operations which are identical across machines. This is rarely used though, since precision and performance is a higher priority goal than consistency in most scenarios where floating point math is used.
To unzip it you would have to reverse this process. In order to recover the original floating point value you must make a number of assumptions about how the hardware and operating system will convert its initial internal representation of a floating point value to an integer, and then how the decompressing machine (which may not be based on the same hardware or even using the same algorithm) will reverse this process.
Completely false, sorry.
To put this another way: lets say I give you a double precision PCM stream stored in IEEE754 format. I don't tell you if the machine that created it was operating in double or extended precision mode, if it was intel or amd, or if it was even x86.
It helps to know whether it was single or double precision. The rest doesn't matter.
For that matter, IIRC, since IEEE 754 demands normalization, which generally makes 7-15 bits of each number fairly coherent across an array, one ought to be able to unambiguously autodetect the endianness of a bytestream of doubles, no matter what numbers are actually encoded. (Unless both the exponent and mantissa truly are randomized, but even then I think you might have outs.)
The ones and zeros making up a floating point file do not necessarily have a consistent meaning across different CPUs and even operating systems, and the output of identical floating point operations on different systems is not required (or even expected) to be identical. The values represented in each floating point sample are therefore literally approximations of an (unknown) true value that is not recorded.
The sound you can hear is that of goal posts shifting. Let's take it step by step...
The ones and zeros making up a floating point file do not necessarily have a consistent meaning across different CPUs and even operating systems
False. There's no magic. Those ones and zeros mean exactly one number, and it's perfectly well defined what that number is.
the output of identical floating point operations on different systems is not required (or even expected) to be identical.
We're not talking about operations. FWIW I don't expect the result of identical fixed point operations in different audio editors to be identical - they should be dithering, and they won't be using the same PRN or even algorithm.
The values represented in each floating point sample are therefore literally approximations of an (unknown) true value that is not recorded.
How is this different from fixed-point audio samples?
googlebot wrote a more articulate response, but I think the above floors in the argument are even more basic and fundamental.
Cheers,
David.
I’d like to clarify a couple things about WavPack that were touched upon earlier.
WavPack is 100% lossless with both 32-bit integer and 32-bit floating-point audio files. For floating point this includes handling the special cases of denormals, NaNs, +/- infinities, etc. Basically, in all cases the binary data is preserved regardless of the content. In fact, one of my tests is to change the interpretation of the various data types and make sure everything still works (although, of course, the compression ratio suffers with incorrect interpretation).
As was alluded, floating-point operations (especially divide, I think) are not always guaranteed to produce the same result on all hardware (at least not by default) and this is one reason why WavPack does not use any floating-point math to compress floating-point data. Instead, the float values are converted to the largest 25-bit integers that can be represented without clipping (each block is scanned first) and compressed with the standard integer code, and then a second data stream is created to losslessly store any data that was truncated during the conversion to integers (this second stream goes in the correction file for hybrid lossless).
The -p option mentioned above discards this second stream, making the operation equivalent to 25-bit integers, although since each block is still independently coded with potentially shifted 25-bit “windows” into the floating-point data, it still retains most of the advantage of using floating-point representation. While for all practical purposes this is as good as lossless, it isn’t technically lossless, and is therefore reported as lossy (and is not the default behavior).
Regarding the original question, there actually is a situation in which WavPack will discard information and still claim lossless compression, although it’s a pretty esoteric case. If the wav file header indicates that the audio has fewer significant bits than the container (for example, 20-bit audio in 3 bytes) then only those significant bits are stored, even if the other bits are not zeros. This will show up as a decoding error only if md5 checking is enabled (because that sum is of the raw audio bytes). I don’t believe that this is an issue because all the audio specified in the wav header is losslessly stored, however it was a case that needed special attention for inclusion in WinZip because of the implicit assumptions of a file archiver.