Skip to main content

Topic: Any encoders which will 'accept but crop' too high resolution  (Read 11571 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • saratoga
  • [*][*][*][*][*]
Any encoders which will 'accept but crop' too high resolution
Reply #25
Floating point is only approximate
It's a digital file. In which way is a copy of that file "approximate"?


The ones and zeros making up a floating point file do not necessarily have a consistent meaning across different CPUs and even operating systems, and the output of identical floating point operations on different systems is not required (or even expected) to be identical.  The values represented in each floating point sample are therefore literally approximations of an (unknown) true value that is not recorded.

What do you think the words "copy" and "lossless" mean in the context of binary data?


I think lossless means you should be able to save a value and then load it latter in a machine independent manner with a guarantee that you will get an identical value out latter.  I'm skeptical that this is possible for a user mode program like flac short of running it in a virtual machine or reference floating point emulator.  I think the simple process of converting between different machine representations of floating point values is likely to be lossy.  Although someone more familiar with IEEE754 compliance may know better then I.

  • googlebot
  • [*][*][*][*][*]
Any encoders which will 'accept but crop' too high resolution
Reply #26
AFAIK the meaning is well defined. Just the precision of operations may differ from machine to machine. But this is only an issue when you want to exploit a specific machine's performance benefits regarding FP calculation. Then you need to know your minimum requirement. Limited hardware doesn't prevent FP math of arbitrary precision, though. It's all a question of effort.
  • Last Edit: 04 January, 2011, 02:14:16 PM by googlebot

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Any encoders which will 'accept but crop' too high resolution
Reply #27
I think lossless means you should be able to save a value and then load it latter in a machine independent manner with a guarantee that you will get an identical value out latter.  I'm skeptical that this is possible for a user mode program like flac short of running it in a virtual machine or reference floating point emulator.

It is definitely possible for 7zip or winrar.

  • saratoga
  • [*][*][*][*][*]
Any encoders which will 'accept but crop' too high resolution
Reply #28
I think lossless means you should be able to save a value and then load it latter in a machine independent manner with a guarantee that you will get an identical value out latter.  I'm skeptical that this is possible for a user mode program like flac short of running it in a virtual machine or reference floating point emulator.

It is definitely possible for 7zip or winrar.


I think you have not understood what you have quoted.  The point is subtle, so I will try again.  In order to zip a floating point value you would first have to save it to a binary (integer) value.  To unzip it you would have to reverse this process.  In order to recover the original floating point value you must make a number of assumptions about how the hardware and operating system will convert its initial internal representation of a floating point value to an integer, and then how the decompressing machine (which may not be based on the same hardware or even using the same algorithm) will reverse this process. 

To put this another way:  lets say I give you a double precision PCM stream stored in IEEE754 format.  I don't tell you if the machine that created it was operating in double or extended precision mode, if it was intel or amd, or if it was even x86.  I then give you the same file, except with the volume doubled (exponent incremented by 1) by the first machine.  Can you get a CPU with ARM vfp unit and an Intel Pentium 2 to take the former file, load it into FP registers, double the volume, and then (losslessly) generate the latter?  In theory you would think yes, since doubling the volume just increments the exponent without rounding, but that assumes the internal representation of all values and operations is the same.  This is a serious question, I really have no idea what would happen on a few different CPUs . . .

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Any encoders which will 'accept but crop' too high resolution
Reply #29
Both compressing and decompressing can be performed using only integer operations (bit shifts, bitwise AND, etc) or using bitfields:

Code: [Select]
struct float32
{
    int mantissa: 23;
    int exponent: 8;
    int sign: 1;
};
  • Last Edit: 04 January, 2011, 03:12:56 PM by lvqcl

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Any encoders which will 'accept but crop' too high resolution
Reply #30
Certainly doing anything meaningful with floating point data requires a "relatively" (heh) inexact measure of equality. That is, there is usually some nonzero amount of relative error to every arithmetic operation you can do with a float. That's part and parcel of using such a format. While it is reasonable to ignore that point when comparing float WAVs, that effectively limits the number of lossless operations one can perform on those WAVs to bit shifts, whereas with fixed point WAVs that would instead be any add followed by an equal-valued subtract (and would explicitly exclude bitshifts extending through the MSB/LSB). IOW, the context here admits two notions of "equality" for float WAVs, not one.

The present discussion, as defined by the OP, is explicitly and solely concerned with maintaining 'absolute' lossless encoding of floating point WAVs, as defined by bit equality (and not by equality to within a relative error or epsilon). That this may not have anything to do with audible sound quality differences (and it likely doesn't!) is entirely besides the point and off topic, for the same reasons why we can have honest discussions about whether or not one sound card or another supports Kernel Streaming or otherwise can send a bit-perfect output signal, even while we might not actually claim that ought to improve audible sound quality.

  • indybrett
  • [*][*][*][*][*]
  • Members (Donating)
Any encoders which will 'accept but crop' too high resolution
Reply #31
I recall years ago reading (on this board) about non-audio information that WAVPACK would preserve, which FLAC would not.  I can't remember if it was some PCM metadata that is only used in a recording studio, or what it was exactly.

It seemed more important to those who were archiving the cd as one file with a cue sheet.
  • Last Edit: 04 January, 2011, 04:23:45 PM by indybrett
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Any encoders which will 'accept but crop' too high resolution
Reply #32
FLAC has --keep-foreign-metadata switch:
Quote
If encoding, save WAVE or AIFF non-audio chunks in FLAC metadata.  If decoding, restore any saved non-audio chunks from FLAC metadata when writing the decoded file.

  • greynol
  • [*][*][*][*][*]
  • Global Moderator
Any encoders which will 'accept but crop' too high resolution
Reply #33
That's not really on-topic but...

It was with wave files that contain metadata.  Generally not (more likely never) applicable to those who embed CUE sheets in lossless images, unless you store a CUE sheet in your wave images (if so tell me how you do this!).  flac does announce the error and has since added a switch that can be used to keep this information, IIRC.
  • Last Edit: 04 January, 2011, 04:52:15 PM by greynol
13 February 2016: The world was blessed with the passing of a truly vile and wretched person.

Your eyes cannot hear.

  • indybrett
  • [*][*][*][*][*]
  • Members (Donating)
Any encoders which will 'accept but crop' too high resolution
Reply #34
FLAC has --keep-foreign-metadata switch:
Quote
If encoding, save WAVE or AIFF non-audio chunks in FLAC metadata.  If decoding, restore any saved non-audio chunks from FLAC metadata when writing the decoded file.



Yeah, that's it.  I think at one time that switch didn't exist, or maybe people just didn't know about it at the time.  Either way, that is one example of some (non-audio) data that could be lost if the format doesn't support it, or if the user doesn't know to use a required command line switch.

Edit:  verbage
  • Last Edit: 04 January, 2011, 04:39:01 PM by indybrett
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro

  • greynol
  • [*][*][*][*][*]
  • Global Moderator
Any encoders which will 'accept but crop' too high resolution
Reply #35
or if the user doesn't know to use a required command line switch.

flac does indeed tell you to use the switch when this happens (also keeping on-topic with this off-topic part of the discussion; see the part that Porcus put in italics).
  • Last Edit: 04 January, 2011, 05:06:47 PM by greynol
13 February 2016: The world was blessed with the passing of a truly vile and wretched person.

Your eyes cannot hear.

  • Porcus
  • [*][*][*][*][*]
Any encoders which will 'accept but crop' too high resolution
Reply #36
Even though I originally asked about encoders, I do think that decoding is highly on-topic as well -- if you do get differences when transcoding it does not really matter which of the steps is the culprit.

As for fb2k specifics: Recall that the wiki warns the user against the LA fb2k plugin: http://wiki.hydrogenaudio.org/index.php?title=Lossless_Audio .



BTW, I don't mind spinning this thread off with metadata discussions, but I suppose that it is fairly well known that you cannot safely throw unicode chars into [enter even text file formats here]. (I do think that the 'this couldn't be audible!' argument belongs to the lossies department and not to this thread, though.)

  • googlebot
  • [*][*][*][*][*]
Any encoders which will 'accept but crop' too high resolution
Reply #37
I think you have not understood what you have quoted.  The point is subtle, so I will try again.  In order to zip a floating point value you would first have to save it to a binary (integer) value.


That is incorrect. A floating point value can be saved in binary notation without loss. Thus it can be zipped and unzipped without problem. I think you are confusing the floating point numbers with real numbers. Floating point numbers are approximations of real numbers, just in the same way that you can approximate the root of 2 by writing down as many digits as you can fit on a sheet of paper. The representation of floats is well standardized. The only thing differing between machines, as I have written already, is the precision of algorithmic operations and at some special case handling. But this is really not a big deal. The results of two machines after a series of algorithmic operations may differ by some insignificant digits, if you run your operations on the bare bone hardware without additional checks. If consistency is required, most development platforms provide standardized operations which are identical across machines. This is rarely used though, since precision and performance is a higher priority goal than consistency in most scenarios where floating point math is used.

To unzip it you would have to reverse this process.  In order to recover the original floating point value you must make a number of assumptions about how the hardware and operating system will convert its initial internal representation of a floating point value to an integer, and then how the decompressing machine (which may not be based on the same hardware or even using the same algorithm) will reverse this process.


Completely false, sorry.

To put this another way:  lets say I give you a double precision PCM stream stored in IEEE754 format.  I don't tell you if the machine that created it was operating in double or extended precision mode, if it was intel or amd, or if it was even x86.


It helps to know whether it was single or double precision. The rest doesn't matter.
  • Last Edit: 04 January, 2011, 08:08:18 PM by googlebot

  • Axon
  • [*][*][*][*][*]
  • Members (Donating)
Any encoders which will 'accept but crop' too high resolution
Reply #38
For that matter, IIRC, since IEEE 754 demands normalization, which generally makes 7-15 bits of each number fairly coherent across an array, one ought to be able to unambiguously autodetect the endianness of a bytestream of doubles, no matter what numbers are actually encoded. (Unless both the exponent and mantissa truly are randomized, but even then I think you might have outs.)

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Any encoders which will 'accept but crop' too high resolution
Reply #39
The ones and zeros making up a floating point file do not necessarily have a consistent meaning across different CPUs and even operating systems, and the output of identical floating point operations on different systems is not required (or even expected) to be identical.  The values represented in each floating point sample are therefore literally approximations of an (unknown) true value that is not recorded.
The sound you can hear is that of goal posts shifting. Let's take it step by step...

Quote
The ones and zeros making up a floating point file do not necessarily have a consistent meaning across different CPUs and even operating systems
False. There's no magic. Those ones and zeros mean exactly one number, and it's perfectly well defined what that number is.

Quote
the output of identical floating point operations on different systems is not required (or even expected) to be identical.
We're not talking about operations. FWIW I don't expect the result of identical fixed point operations in different audio editors to be identical - they should be dithering, and they won't be using the same PRN or even algorithm.

Quote
The values represented in each floating point sample are therefore literally approximations of an (unknown) true value that is not recorded.
How is this different from fixed-point audio samples?

googlebot wrote a more articulate response, but I think the above floors in the argument are even more basic and fundamental.

Cheers,
David.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Any encoders which will 'accept but crop' too high resolution
Reply #40
I’d like to clarify a couple things about WavPack that were touched upon earlier.

WavPack is 100% lossless with both 32-bit integer and 32-bit floating-point audio files. For floating point this includes handling the special cases of denormals, NaNs, +/- infinities, etc. Basically, in all cases the binary data is preserved regardless of the content. In fact, one of my tests is to change the interpretation of the various data types and make sure everything still works (although, of course, the compression ratio suffers with incorrect interpretation).

As was alluded, floating-point operations (especially divide, I think) are not always guaranteed to produce the same result on all hardware (at least not by default) and this is one reason why WavPack does not use any floating-point math to compress floating-point data. Instead, the float values are converted to the largest 25-bit integers that can be represented without clipping (each block is scanned first) and compressed with the standard integer code, and then a second data stream is created to losslessly store any data that was truncated during the conversion to integers (this second stream goes in the correction file for hybrid lossless).

The -p option mentioned above discards this second stream, making the operation equivalent to 25-bit integers, although since each block is still independently coded with potentially shifted 25-bit “windows” into the floating-point data, it still retains most of the advantage of using floating-point representation. While for all practical purposes this is as good as lossless, it isn’t technically lossless, and is therefore reported as lossy (and is not the default behavior).

Regarding the original question, there actually is a situation in which WavPack will discard information and still claim lossless compression, although it’s a pretty esoteric case. If the wav file header indicates that the audio has fewer significant bits than the container (for example, 20-bit audio in 3 bytes) then only those significant bits are stored, even if the other bits are not zeros. This will show up as a decoding error only if md5 checking is enabled (because that sum is of the raw audio bytes). I don’t believe that this is an issue because all the audio specified in the wav header is losslessly stored, however it was a case that needed special attention for inclusion in WinZip because of the implicit assumptions of a file archiver.