HydrogenAudio

Hydrogenaudio Forum => General Audio => Topic started by: Porcus on 2017-10-22 20:05:08

Title: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2017-10-22 20:05:08
There are a few of these out on the 'net by now. I can only speculate that more artists use software that work with such files, and more often than before an artist will upload a file without having to worry whether to save bandwidth by going mp3.

But why? Of course there will always be someone who on purpose will upload the biggest files they can, but I know that not all of my files were intended to be uploaded in that format ...
Is it so that these DAWs do not even tell you that these are not your "usual" WAV files? (What does Audition do? I'm too lazy to download a trial version just for this.)

Just curious.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: jsdyson on 2017-10-22 20:18:26
In my little world of Linux/Unix and Windows command line, I use the utility sox (the command name is actually soxi), and that is how I normally determine what kind of data/sample rate in a wav/mp3/flac/etc file.   The utiiliies that I write also automatically display the data type/rate.   My own 'normal' data type is 32bit float (or 24bit signed int for flac), even though I normally produce 16bit to send to others.  So, the best bet that costs NO money is to use the command line soxi.   Also, I happen to know that Audacity (free GUI audio software) displays the data type also ( i use Audacity once in a while.)   A wonderful (almost lossless) tradeoff is to use 24bit signed flac, which ends up being not far different from a 16bit wav file in size.

John
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2017-10-22 20:29:17
A very horrible thing with float 32 file is they can be encoded in different "normalized" fashions. If the decoder misinterpret the format, the result will be horrible full scale square wave-ish like crap which may destroy your equipment. I experienced this nightmare several times, sometimes in foobar2000, sometimes in Audition.

Of course, in DAWs if people find the decoded waveform looks like crap, they won't hit the play button, but that's not the case with typical audio players.

https://forum.cockos.com/showthread.php?t=85641
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: jsdyson on 2017-10-23 01:37:40
A very horrible thing with float 32 file is they can be encoded in different "normalized" fashions. If the decoder misinterpret the format, the result will be horrible full scale square wave-ish like crap which may destroy your equipment. I experienced this nightmare several times, sometimes in foobar2000, sometimes in Audition.

Of course, in DAWs if people find the decoded waveform looks like crap, they won't hit the play button, but that's not the case with typical audio players.

https://forum.cockos.com/showthread.php?t=85641
oh my!!!   Most of the files that I have dealt with are +-1.0 floating point, but I do know that it is theoretically possible to sent out files with bigger numbers.  I always believed that +-1.0 floating point is the same as +-32767  for 16 bit signed -- and so be it...  So, you have seen different?   But of course, invalid files with >+-1.0 (assumed that they were invalid) should be clipped and/or flagged.   In between running my Unix pipelines, I do (incorrectly, I believe) SOMETIMES during my experiments,  take advantage of being able to deal with greater than normal floating point values.   For example, I might have an expander whose output might peak at +-2.828 (just as an example) due to exceptional material, but then future normalization stages will fix it.  On the other hand, 'sox' will complain about such files, and probably should process the file if the results are correctly normalized,  or MAYBE SHOULD BASED UPON COMMAND LINE FLAGS simply bounce the file somehow?.

Your point about nonstandard floating point values is definitely a valid issue, however.  (By the way, the term normalized/un-normalized has a specific meaning the the floating point realm -- when the term different 'normalization' was used -- I assumed being able to handle differing/odd ranges of values, right?)
So, to me, it seems like the FP format  should be standardized, or avoided for interchange.  I suspect that for most (all) PRACTICAL purposes, the signed 24 bit should be sufficient, but wonder if it also has such scaling issues?  (I use 24bit only so that I can use flac with reasonably high resolution.)

John Dyson
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2017-10-23 08:01:40
Thanks for the warning. And thanks to ReplayGain and fb2k's preamp option that turns down files w/o RG info ...

There could be a lot of different signals in a WAVE file (https://msdn.microsoft.com/en-us/library/windows/hardware/dn653308.aspx - date says 2007, but a version was around before y2k), but I don't think I have seen music published as anything but 16-bits integer, 24-bits integer, whatever-32-bits-that-WavPack-can-contain - and one single IMA ADPCM.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Nick.C on 2017-10-24 02:39:00
David made a change to WavPack to permit the storage of floating point audio using the same compression method as for integer audio (around the time I was developing TransPCM - which can output Float16 audio from several different inputs).

For Float16, I made use of the full range, i.e. ±65504 at the top end to ±5.96046E-08 as the minimum subnormal, giving a range of 1,098,975,582,421:1 (or approximately 40 bits) with up to 11 significant bits.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2021-01-25 11:29:07
A very horrible thing with float 32 file is they can be encoded in different "normalized" fashions. If the decoder misinterpret the format, the result will be horrible full scale square wave-ish like crap which may destroy your equipment. I experienced this nightmare several times, sometimes in foobar2000, sometimes in Audition.

Of course, in DAWs if people find the decoded waveform looks like crap, they won't hit the play button, but that's not the case with typical audio players.

https://forum.cockos.com/showthread.php?t=85641
OK I made a tool to automatically detect these kinds of files, and convert them to the standard formats as shown here. Of course, you need to have these kinds of files to try it.

X

The detection methodology is also somewhat relevant to this thread as well, as my program showed more accurate result than others (SoX, Bitter, Adobe Audition). For example, here is a 8-bit file, with 1dB volume reduction, then dithered to 16-bit.
https://hydrogenaud.io/index.php?topic=120395.0
I search for a tool that can check my 24bit downloads I have purchased over the years. Honestly its just a handful compared to my whole 16bit FLAC collection but I often wondered whether these 24bit files are actually true or just upconverted 16bit.
You could try a bit meter. A few are available as VST plugins, for example Bitter (https://www.stillwellaudio.com/plugins/bitter/).
X

My result:
X

Download link:
https://1drv.ms/u/s!AvzB71jO7t0-gYwfVGulbuFbVFZcmg?e=uN2p8Z

SHA256:
ee3a919fb267b9afec9494a0ae7f9b3ebd4fad97b8a03cc65245839ec2138d4f
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: magicgoose on 2021-01-25 20:50:35
equipment doesn't get destroyed from its intended use (playing sounds), this is some urban legends.
it can destroy ears if you accidentally play it too loud, though.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: j7n on 2021-01-26 01:05:56
Tweeters are almost never capable of outputing maximum power in high frequencies that clipping or integer overflow can create.

A 1 dB volume adjustment with dither can also be detected on a histogram of byte values where some are more likely to occur than others. All bytes of a sample are overlapped on the same chart, but the effect is still clear. This can be run on a fadeout portion to detect smaller discontinuities. But a mild bass boost is enough to make the histogram smooth again.

I was unable to run this program because it requires NET Framework. Would it describe a file with bass boost and phase shift in a way that is detectable? Or in other words, what else besides a volume adjustment can it detect?

Maybe you could output a pseudographic bar chart in the analysis output to get a histogram that is easier to parse by eye (more symbols: greater value).
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2021-01-26 05:23:50
I was unable to run this program because it requires NET Framework. Would it describe a file with bass boost and phase shift in a way that is detectable? Or in other words, what else besides a volume adjustment can it detect?
The ReadMe mentioned these things already if you cannot run it. To elaborate, BitSort is used to assist the development of oldsCool so that I don't need to convert everything to .wav files during analysis. It reads the data bytes in files in integer and float with different ranges and guess the format as the header information could be unreliable. So anything resembles a bit meter is just a side effect rather than the main function of the software. For example, BitSort will still detect 16.8 float content if the files are being saved in WavPack as it is lossless and supports both 32-bit integer and float.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2021-01-26 07:46:40
Anyway, here is how the 8-bit RMAA signal looks like with bass EQ, peak normalization (to avoid clipping) and dither. The highlighted parts are still correlated to the original file, not scattered around. Of course, it cannot be compared in this way if there is no reference file, and if you resample, it falls apart anyway.
X

The frequency response of the EQ looks like this:
X

I don't like the presentation of Bitter as the UI is too small and it is pretty hard to count the number of lines, especially when they move quickly.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-01 18:11:21
OK I made a tool to automatically detect these kinds of files, and convert them to the standard formats as shown here.

Hm. Exposing my ignorance:

* What am I doing wrong when bitsort.exe filename.wav returns "Not enough storage is available to process this command." and I have more than enough space on the drive?

* How do I read the output - in particular, what indicates that I have an Audition-type file where WavPack/OptimFROG should be invoked with the correction option?
I take it that 16/16 for 32-bit floating-point PCM I guess means that someone has taken in a 16-bit file and just exported as float? But what do the two "16" mean, really?

And then this:
"Can't guess the type" and a warning that "output clipped 185095 samples; decrease volume?"
These files - uploader calls them "NON playable 32 bit wav version" :-)
https://soundcloud.com/deceiverdubz/destrudo-free-download
https://soundcloud.com/deceiverdubz/hate-1


And ... maybe a bit off topic, but:
* when a 32-bit file uses only 16 or 24, there should be a way to decimate it "losslessly except for overall volume information" to integer ... but how?

Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Octocontrabass on 2022-03-02 02:03:59
* when a 32-bit file uses only 16 or 24, there should be a way to decimate it "losslessly except for overall volume information" to integer ... but how?
If it's normalized to ±1.0, multiply it by 32768 or 8388608 and convert to integer. If it's normalized to some other peak, divide by that number first.

You have to be a little bit more clever to figure out whether you can perform such decimation in the first place and how to do it correctly if so...
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-02 06:27:54
* What am I doing wrong when bitsort.exe filename.wav returns "Not enough storage is available to process this command." and I have more than enough space on the drive?
BitSort does not accept direct file input, it is meant to be used with external decoders.
Quote
* How do I read the output - in particular, what indicates that I have an Audition-type file where WavPack/OptimFROG should be invoked with the correction option?
I am not familiar with WavPack/OptimFROG commands, so can't help.
Quote
I take it that 16/16 for 32-bit floating-point PCM I guess means that someone has taken in a 16-bit file and just exported as float? But what do the two "16" mean, really?
16/16 is SoX's interpretation, not mine. My results are always in the generated txt files. My guess is SoX only sees the used highest bit and lowest bit, without checking anything in-between. My software on the other hand also looks for missing bits in-between. If the input file has a float header, BitSort and oldsCool will always display the results in float format, i.e. not counting bits, just show max/min sample values. For example, 16-bit files disguised as float may often show  1 / 32768 = 3.0517578125e-05 minimum sample value. The same reason that foobar2000's RG scan results often show 0.999969 peak value, which means 32767 / 32768 = 0.999969482421875.
Quote
And then this:
"Can't guess the type" and a warning that "output clipped 185095 samples; decrease volume?"
These files - uploader calls them "NON playable 32 bit wav version" :-)
https://soundcloud.com/deceiverdubz/destrudo-free-download
https://soundcloud.com/deceiverdubz/hate-1
Everything before "BitSort is reading n bytes..." are printed by SoX, not BitSort. As mentioned in the help file in the "Decoder differences" section, for .wav files, oldsCool is the right tool to use, not BitSort. From my observation these files used values up to +1.0, which translates to +32768 or +8388608 or +2147483648 in different integer bit-depth. The last digit is 8 but not 7 and therefore SoX reported clipping when BitSort is used with SoX. With oldsCool, these files should show this:
Code: [Select]
-------------------------------------------------------------------------------
E:\download\DESTRUDO MASTER V2.wav
00:02:20.8000000 = 12418560 samples / 2-ch @ 44100Hz
32-bit floating point
       Ch   Position    Value(+/-)               dBFS
Max    0    28          1                        0
Min    1    4709505     4.085826676991644e-10    -187.77440120645838
-------------------------------------------------------------------------------
E:\download\PAIN.wav
00:02:03.8571428 = 10924200 samples / 2-ch @ 44100Hz
32-bit floating point
       Ch   Position    Value(+/-)               dBFS
Max    1    18914       1                        0
Min    1    5446601     1.0007897621733264e-08   -159.99314292029698
-------------------------------------------------------------------------------
.WAV file(s) processed: 2
foobar2000 uses 32-bit float internal format too, so it can also show the correct values when used with BitSort. foobar2000 however is lossy for 32-bit integer files.

PS: For those who don't know, the updated versions of BitSort and oldsCool are posted here:
https://hydrogenaud.io/index.php?topic=97746.msg1004399#msg1004399
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-02 08:50:28
* How do I read the output - in particular, what indicates that I have an Audition-type file where WavPack/OptimFROG should be invoked with the correction option?
I am not familiar with WavPack/OptimFROG commands, so can't help.

Reason I am asking here is to detect an Audition-type file.

Typing "-a" I can do myself, but fwiw, here is the point:
Quote from: off.exe --help
  --correct_audition  correct Adobe Audition / CoolEdit conversion bug
        when converting from int to float, Audition converts 0 values
        to random noise with maximum relative amplitude < 5*10^-16;
        this option carefully corrects this bug setting them back to 0
        and significantly improves compression ratio in these files
Quote from: https://www.wavpack.com/wavpack_doc.html
The WAVEFORMATEXTENSIBLE structure is used (if present) to determine the format details. However, there are some programs that use their own non-standard format extensions. The most popular of these is Adobe's Audition (previously Syntrillium's CoolEdit) which created two new 32-bit floating point formats. An option has been added to WavPack (-a) to force the "adobe" interpretation of these floating point formats. If you are compressing integer files do NOT use this option.

Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-02 14:52:02
Here are the 16.8 and 24.0 float files saved with Audition 1.5 so that you can try them with WavPack. Audio data in these two files should be identical to DESTRUDO normal.wav when correctly encoded and decoded.

Bona fide 16.8 and 24.0 float formats were introduced in Syntrillium Cool Edit Pro/2000. 16.8 and 24.0 float files from different versions of Adobe Audition are supposed to show some differences in file headers, so don't take the attached files as the only reference. Installers of old Syntrillium software do not work with modern 64-bit Windows, I used VirtualBox when debugging BitSort and oldsCool but uninstalled now, so can't attach those files at this moment.

Another thing is don't attempt to use oldsCool with CMD, always use drag and drop or SendTo shortcut, otherwise will lose full Unicode path name support. BitSort with foobar2000 is fine with Unicode, as foobar2000 itself can deal with Unicode.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-02 16:51:05
Wait: so the answer to
"how to check whether a file is of this kind",
is to
"drop it onto oldsCool.exe and see if it converts"
?

Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-02 18:00:22
Look for the 16.8 and 24.0 suffixes when oldsCool is operating in read-only mode (by setting log.txt as read-only). Files with these suffixes will convert when oldsCool is running in normal mode. oldsCool will also change the WAVE_FORMAT_EXTENSIBLE tag (WAVEFORMATEXTENSIBLE is the structure) in non-multichannel (mono/stereo) files so that old Cool Edit Pro can open these files. So if oldsCool displays WAVE_FORMAT_EXTENSIBLE in mono/stereo files, the files will be converted as well, even if they are not 16.8 or 24.0 float files.
Code: [Select]
Press ENTER to operate in read-only mode, other keys to quit.
-------------------------------------------------------------------------------
E:\download\DESTRUDO\DESTRUDO 16-8.wav
00:00:10 = 882000 samples / 2-ch @ 44100Hz
32-bit floating point (16.8)
       Ch   Position    Value(+/-)               dBFS
Max    0    87          1                        0
Min    1    299505      4.085826676991644e-10    -187.77440120645838
-------------------------------------------------------------------------------
E:\download\DESTRUDO\DESTRUDO 24-0.wav
00:00:10 = 882000 samples / 2-ch @ 44100Hz
32-bit floating point (24.0)
       Ch   Position    Value(+/-)               dBFS
Max    0    87          1                        0
Min    1    299505      4.085826676991644e-10    -187.77440120645838
-------------------------------------------------------------------------------
E:\download\DESTRUDO\DESTRUDO normal.wav
00:00:10 = 882000 samples / 2-ch @ 44100Hz
32-bit floating point
       Ch   Position    Value(+/-)               dBFS
Max    0    87          1                        0
Min    1    299505      4.085826676991644e-10    -187.77440120645838
-------------------------------------------------------------------------------
.WAV file(s) processed: 3
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-03 10:43:35
OK. Assume the .wv files are correctly encoded, here is the CMD script to pipe wvunpack to BitSort. Put BitSort.exe, wvunpack.exe and BitSort-wv.cmd in the same location, and send one or more files to BitSort-wv.cmd. Inputs not ended with .wv will be filtered to prevent errors. I did not try OptimFROG but perhaps you can fiddle with the script and try to figure it out.

Make a shortcut of BitSort-wv.cmd in the SendTo folder and send files to the shortcut, otherwise it may not work.
X
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-05 13:41:16
The script above is pretty lame and buggy due to my poor CMD knowledge. After some studies I attached a better one. Files ended with .wv will be decoded by wvunpack, other extensions except .txt will be decoded by SoX, then pipe to BitSort. Piping to FFmpeg could be a good idea as well but the documentation is ridiculously long and hard to follow.

BitSort will still show error if for example, the .wv files are in DSD format. The WavPack manual mentioned -f[n] can be used to check file info without decoding but the results will be sent to stdout. Without modifying the source code of BitSort I don't know how can I use -f[n] in a CMD file so that BitSort can either skip the DSD files or decode to PCM for analysis.

Another thing is while MPC-HC is fine with WavPack DSD, it still produces Merzbow-style sound when playing 16.8 and 24.0 float WavPack files. foobar2000 on the other hand is fine, as well as Adobe Audition, but no surprise because the Audition plugins are downloaded from the official WavPack website.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-05 13:57:13
MPC-HC, is that forked off to being developed? (Does it use ffmpeg, or ... ? ffmpeg does some strange things every now and then.)
But now I have fought enough codecs for a while, so ... I'm just going to take note that I couldn't understand OptimFrog's handling, and shrug it off.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-05 14:14:21
I am using this one:
https://github.com/clsid2/mpc-hc/releases

Also, WavPack's manual linked to some test files:
http://www.rarewares.org/wavpack/test_suite.zip
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-07 11:05:59
Despite all the geeky talks on the OptimFROG website, the tools provided are pretty easy to use. Drag integer files (32-bit is supported) to ofr.exe and float files (normal, 16.8 and 24.0 are supported) to off.exe and they will encode, --correct_audition seems unnecessary, at least for the DESTRUDO snippet I posted, but could be useful in case of false positive/negative depending on how OptimFROG deals with these formats.

Decoding on the other hand is not too user friendly as off.exe only works with float and ofr.exe only works with integer, but the encoded files are all ended with .ofr. I have a feeling that the author has a plan to merge these formats and make all of them capable of real time decoding/seeking, but abandoned for years due to some unsolvable difficulties or simply because everyone is using flac now.

The attached script added OptimFROG support. Files ended with .ofr will be decoded by ofr.exe, .off files will be decoded by off.exe.

BTW, Microsoft is officially distributing a stereo 24/96 .wav file in Windows 10 without using WAVE_FORMAT_EXTENSIBLE, so kind of funny. File name is camerashutter.wav but the exact location can be different in different machines. The file has a marker too, and markers can be important for DAW users. flac by default discards these things.
https://www.virustotal.com/gui/file/4ecdcbd3018c03d813978ba453aaecc53ddd178a5b46e82d3c03e11c262403e8
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-08 10:16:23
--correct_audition seems unnecessary, at least for the DESTRUDO snippet I posted
Ah, OK. Explains my headscratching over how to make it matter. Thx.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-10 21:55:15
The attached file was generated to trigger the Audition bug, as well as putting --correct_audition into effect.
X
Audacity is showing what the waveform is supposed to be, a distortion free tone, while Audition is showing some funny stuff because it thinks this int32 file is 16.8 float, though oldsCool indicates the file is int32 with 17 empty bits. With --correct_audition, off.exe said it corrected something. However, after correction the file is no longer bit-perfect. Most importantly, it created a lot of distortion in the corrected file.

On the other hand, simply drag the file into ofr.exe, the encoded file will be bit-perfect and much smaller. Because a type 1 32-bit .wav file can also be a Cool Edit/Audition 16.8 float file, both ofr.exe and off.exe can accept these files.

I don't understand what the OptimFROG author wanted to achieve, but given the lossy nature of --correct_audition, this option is of little use. The worst thing is even without using --correct_audition, off.exe still nagged me about this "feature".

A sensible solution is to avoid using a buggy version of Audition to open such files, or use something else like Audacity, or ideally, a 64-bit float DAW (e.g. Reaper) to edit these kinds of int32 files. It sounds strange to me why a software bug should be corrected by changing the audio data of a healthy file, especially for a lossless codec.

Also, I did not know APE also supports int32 now. Apart from insane doesn't work, the Cool Edit/Audition filter provided in the installer also does not support reading and writing int32 APE files. However, since MAC.exe supports int32 and foobar2000 is lossy for int32, the CMD script is updated to support APE. I did not install the whole package and only extracted MAC.exe and the Cool Edit/Audition filter from the installer, so can't comment about other things.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-10 23:00:19
Frog behaviour beats me, could you maybe alert the developer?
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-11 05:09:25
I won't. The author obviously has great knowledge in algorithms and such. For example, 7-Zip, WavPack and Monkey have much poorer compression ratio than ofr.exe for this test file. The author may simply have a different way of thinking than ordinary users.

It is also possible that the version of Audition the author uses showed different result than mine, who knows.

APE developer(s)? completely evaded this issue by not supporting int32 in Cool Edit and Audition. WavPack offers manual override option. In case of false positive BitSort and oldsCool may say the file's max sample value is several hundred dB above full scale and such, so should be pretty easy to notice.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-11 18:29:23
A "slightly different" question. To identify whether a given 32-bit float signal can be fit into 24 bit (or less) integer with no other loss than a volume normalization - I could read a BitSort log and subtract min from max, but is there any quicker method (or other quicker tool)?
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-11 19:07:19
Knowing the min and max values does not guarantee a file can be losslessly converted to a fixed bit depth because the intermediate values are not shown. The most reliable method is to check every sample. It sounds like a slow operation but in practice still much faster than typically used operations like dither, resample and so on.

The readme file mentioned oldsCool can check if converting a 16.8 or 24.0 file to a normal file is lossy or not, this operation checks all samples, or until it finds the first lossy sample.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2022-03-11 19:09:55
Ah, right. Ruling out isn't ruling in, basic logic blunder. Thx.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-12 21:32:46
ALAC (requires refalac.exe) support added. Also changed the fallback SoX command to demonstrate the scale factor feature of stat. With -s 256 sample values will be displayed as 24-bit, and -s 65536 for 16-bit and so on. In SoX, max and min amplitude values are signed, while BitSort and oldsCool display max and min absolute values.

I am not confident to correctly explain the meaning of every item displayed by SoX, better check the SoX manual.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-03-17 15:01:42
https://hydrogenaud.io/index.php?topic=122263.msg1009279#msg1009279
Real world example that oldsCool shows the HDCD files posted by @koawmfot  only used 17 bits while SoX and Audition showed something else. It is recommended to save the files as flac or 7zip as they are smaller and have built-in error check so that we know the downloaded files are not corrupted.

SoX
Code: [Select]
Input File     : 'E:\download\SW Decode - ffmpeg.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:15.00 = 661500 samples = 1125 CDDA sectors
File Size      : 3.97M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

In:100%  00:00:15.00 [00:00:00.00] Out:662k  [ =====|=====-]        Clip:0
             Overall     Left      Right
DC offset   0.000009  0.000009 -0.000004
Min level  -0.603907 -0.495191 -0.603907
Max level   0.536741  0.536741  0.481397
Pk lev dB      -4.38     -5.40     -4.38
RMS lev dB    -21.43    -21.23    -21.64
RMS Pk dB     -16.77    -16.77    -18.34
RMS Tr dB     -29.31    -29.20    -29.31
Crest factor       -      6.19      7.29
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      24/24     23/24     24/24
Num samples     662k
Length s      15.000
Scale max   1.000000
Window s       0.050
Done.

Input File     : 'E:\download\SW Decode - CUETools.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:15.00 = 661500 samples = 1125 CDDA sectors
File Size      : 3.97M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

In:100%  00:00:15.00 [00:00:00.00] Out:662k  [ =====|=====-]        Clip:0
             Overall     Left      Right
DC offset   0.000009  0.000009 -0.000004
Min level  -0.603907 -0.495192 -0.603907
Max level   0.536739  0.536739  0.481396
Pk lev dB      -4.38     -5.40     -4.38
RMS lev dB    -21.43    -21.23    -21.64
RMS Pk dB     -16.77    -16.77    -18.34
RMS Tr dB     -29.31    -29.20    -29.31
Crest factor       -      6.19      7.29
Flat factor     0.00      0.00      0.00
Pk count           2         2         2
Bit-depth      20/20     19/20     20/20
Num samples     662k
Length s      15.000
Scale max   1.000000
Window s       0.050
Done.

oldsCool
Code: [Select]
-------------------------------------------------------------------------------
E:\download\SW Decode - ffmpeg.wav
00:00:15 = 1323000 samples / 2-ch @ 44100Hz
24-bit fixed point
Bit    Count            Percent
0      118          0.008919124
8      199           0.01504157
9      426           0.03219955
10     847           0.06402116
11     1730           0.1307634
12     3303           0.2496599
13     6834           0.5165533
14     13527           1.022449
15     27112           2.049282
16     54488           4.118518
17     107067          8.092744
18     205345          15.52116
19     346154          26.16432
20     382560           28.9161
21     161670          12.21995
22     11612          0.8777022
23     8           0.0006046863
-------------------------------------------------------------------------------
E:\download\SW Decode - CUETools.wav
00:00:15 = 1323000 samples / 2-ch @ 44100Hz
24-bit fixed point
Bit    Count            Percent
0      118          0.008919124
8      199           0.01504157
9      426           0.03219955
10     847           0.06402116
11     1730           0.1307634
12     3303           0.2496599
13     6834           0.5165533
14     13527           1.022449
15     27112           2.049282
16     54488           4.118518
17     107067          8.092744
18     205345          15.52116
19     346154          26.16432
20     382560           28.9161
21     161670          12.21995
22     11612          0.8777022
23     8           0.0006046863
-------------------------------------------------------------------------------
.WAV file(s) processed: 2

To identify whether a given 32-bit float signal can be fit into 24 bit (or less) integer with no other loss than a volume normalization
I am thinking about adding this check, as well as other things like more advanced wBitsPerSample analysis and such. I upgraded my PC and still don't have Visual Studio and other stuff installed so it may take some time.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2022-05-11 12:49:21
1.0.0.2

Download link:
https://1drv.ms/u/s!AvzB71jO7t0-gYwnoZKDu2Dr3TrAiQ?e=OnUhXp
SHA256:
8d82b8c79ec2ac85825fcc0ad03f6666ed19fdbfdc0f4beb7230c203cc465684

The CMD scripts attached in older posts are not compatible with BitSort 1.0.0.2, use the scripts included in the archive above. Other details are included in the updated readme file.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-03 21:34:19
1.0.0.3

Download link:
https://1drv.ms/u/s!AvzB71jO7t0-gYxSXo8wLrAYuWOuUw
SHA256:
daed1b4c095f2f7d284632c38b5b07f059898a909d2dac665c4765becd34c735

Also added foobar2000 x64 specific info in the readme file.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-04 18:26:22
After some tests I found that the logic of foobar2000 x64's "Highest BPS mode supported" is that choosing 32-bit float will also include 32-bit fixed, but not the other way around. Therefore BitSort users should always set "Highest BPS mode supported" to 32-bit float, there is no need to create two BitSort presets like what I mentioned in the readme file.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-08 09:36:24
1.0.0.4

Download link:
https://1drv.ms/u/s!AvzB71jO7t0-gYxXHoM-nm9xnQTgBQ
SHA256:
f5eb7a73b9a1399d7906dfbbd00b0d7b5bbfbf2dd54d75878dcfd7b9410988ed
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: ktf on 2023-05-08 12:28:45
Through the years there have been quite a few people asking for float support in FLAC. Implementing this gives quite a few challenges, so at some point I pondered whether it would be useful to have some utility (perhaps the flac command line program) able to losslessly convert float files that have 'the right characteristics' to 32-bit int. This would involve a reversible (and thus lossless) normalisation step.

Of course, this means the input cannot have NaN or Inf samples, and the dynamic range is limited to what can be contained by 32-bit int. This conversion would have to be done in two passes over the input, first to find the required normalisation, the dynamic range of the signal and whether the file contains non NaN or Inf samples.

So, this gets me back to the original question: what do 32-bit floating point PCM files in the wild look like? If about 99% of those files can be losslessly contained in 32-bit integer PCM, the suggestion above would make sense. If a much larger share than 1% of those files cannot be converted using this method, it is probably not a good idea to implement it, as it would cause a lot of disappointment I suppose.

Is this something BitSort can do, reporting how much bits a 32-bit float file would use when converted to a non-float representation?

edit: this approach has the additional benefit that the playback application does not need to know about floating point representations, and that (crude) normalisation has already taken place.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-08 13:06:03
Is this something BitSort can do, reporting how much bits a 32-bit float file would use when converted to a non-float representation?
BitSort internally converts 32/64-bit float input to 32-bit fixed. If lossless conversion is possible, it will further check if fewer bits can be used or not.

WavPack still has an advantage to preserve non-audio data integrity in some cases, even if the files are not float.

BTW, I just found that my HTML readme file has incorrect syntax by using the "<" symbol literally instead of "&lt;" though it seems that browsers can still render the page without issue.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: mycroft on 2023-05-08 17:11:40
Use astats filter from FFmpeg instead, astats bit depth calculation is 64bit.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-08 19:51:59
Just to clarify when BitSort/oldsCool receive audio samples, the stats are calculated in 64-bit float precision, and calculates 32-bit float and fixed point values in parallel. Otherwise it is not possible to detect integer overflow when converting to fixed point and denormal underflow when checking the Cool Edit/Audition float formats. Check the bundled ReadMe file for more details.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-09 06:24:05
So, this gets me back to the original question: what do 32-bit floating point PCM files in the wild look like? If about 99% of those files can be losslessly contained in 32-bit integer PCM, the suggestion above would make sense. If a much larger share than 1% of those files cannot be converted using this method, it is probably not a good idea to implement it, as it would cause a lot of disappointment I suppose.
It is more likely that lower than 1% of 32-bit float files in the wild can be losslessly converted to fixed point by simple bit-shifting. Most software DAWs and plugins internally use floating point processing. For non-destructive operations (e.g. drawing a volume/pan envelope, or using insert/aux effects), processed 32/64-bit float data will pass through the master bus and subsequently the file renderer. If the DAW user export to fixed point the result will look like the Sound Liaison example I mentioned in the ReadMe file when the rendering pipeline uses 32-bit float. For destructive operations, even if a DAW supports 64-bit float processing, it may store intermediate results in disks (e.g. undo data and bounced tracks) using 32-bit float, so the ideal export format will still be 32-bit float.

So it is the other way around: 32-bit fixed point files in the wild has a higher chance to achieve lossless results when being converted to 32-bit float.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-05-10 08:42:13
FWIW, the 32bit integer file that I used for comparison is the RMAA test signal.
That's no good. RMAA's int32 file is converted from the float32 signal, and if you carefully look at the "stereo crosstalk" spectrogram, all integer formats are clipped, the only unclipped one is the float signal. Luckily RMAA only uses that signal to measure crosstalk and ignores the clipping. The clipping can be easily identified even with a low precision analyzer like Spek. Notice the vertical lines in the integer file, those parts are clipped.
[attach type=image]22897[/attach]
Saving the 32-bit int RMAA signal without including the crosstalk signal, the result will round trip to 32-bit float.
X
Code: [Select]
-------------------------------------------------------------------------------
E:\download\Test signal (44 kHz 32-bit).wav
00:01:00.5132653 = 5337270 samples / 2-ch @ 44100 Hz
32-bit fixed point
Bit    Count            Percent
0      1646379         30.84684
1      424387          7.951387
2      6            0.000112417
3      6            0.000112417
4      7           0.0001311532
5      10          0.0001873617
6      13          0.0002435702
7      15          0.0002810426
8      21          0.0003934596
9      40          0.0007494468
10     61           0.001142906
11     96           0.001798672
12     167           0.00312894
13     264          0.004946349
14     495          0.009274404
15     953           0.01785557
16     1824          0.03417477
17     3541          0.06634478
18     7026           0.1316403
19     14094          0.2640676
20     29706          0.5565767
21     91781           1.719624
22     32315          0.6054593
23     18736          0.3510409
24     38079          0.7134546
25     71325           1.336357
26     147799          2.769187
27     278214          5.212665
28     410494          7.691085
29     612116          11.46871
30     1070134         20.05021
31     437166          8.190817
Round Trip: 32f
-------------------------------------------------------------------------------
oldsCool 1.0.0.4
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-08-26 10:09:29
Compression test. On float files I found in the wild. 13 of them. 5 from the same artist and another 2 from another artist, effectively down to 8. One of them is an 80 minutes podcast - many songs yes, but I guess the whole thing has been through the same (lossy?) processing first.
All found on Soundcloud except one from a netlabel. Description and most download links at the end - you need a Soundcloud account to download the WAVE I think.

Why even bother to test? Float is potentially a quite different beast compression-wise, than integer. And there are now actually four codecs that handle floating-point - five if you count the adaptive-mode ALS as a separate codec with the same name (it kinda is).
So, are we in for any surprises? Maybe? But first, a non-surprise:

TL;DR: Use WavPack, unless you have some very particular preferences.


Warning on Monkey's:
Very bad compression ratio - and worse the slower settings!  Yes, "Fast" compresses best and "Insane" worst - not only in average, but in most single tracks too.
If you are already using Monkey's over WAVE in some audio software plugin, sure it does improve slightly, and now you can use it on float too - so kinda it defends its existence, and in a setting where you likely would use "fast" anyway I guess?
No 3rd party support. (Monkey's float is too new, may or may not be coming.)

Warnings on OptimFROG (which, if you are willing to wait, gets you smallest files - frog doing frog things).
I got it to crash, several times.  It seems it was when compressing with wildcard, off *.wav, not on FOR looping.  Memory management?
Luckily I did not get it to produce corrupted files.
Also, OptimFROG isn't supported by ffmpeg or anything - but official tools are available for several OSes and include a fb2k component.

Warnings on MPEG-4 ALS (for the morbidly curious)
Nothing else can decode this.  ffmpeg and VLC have ALS decoding support you say? Oh no, not for floating-point. (And in any case, not for the -z codec.)
Although I did not make it misbehave on float, I have earlier on provoked the ALS reference encoder/decoder to produce files it cannot read (and nothing else can!).  It seems not even those who care about ALS at all (the dev team?), bother about -z.


Performance then? Most like expected, except Monkey's doing bad. And, slower Monkey's doing even worse (but only like 0.0 to 0.2 percentage points, I omit them from table).
Timings are very rough. Used timer64, but not all were re-run. YMMV, but the big picture is kinda ... well perhaps WavPack -h and -hh didn't appear that expensive in decompression speed compared to in ktf's tests, that is maybe a mild surprise?
Also the last file in the below list was added after timing, and is only included in the sizes and percents.

No manual check for any CoolEdit files, no option to correct for that applied.

unw.avg %sz-wghtWAVE = 3469880566enc.timedec.timecodec/setting/b]comment
86,0%85,3%29601218563610NTFS compact /C /EXE: LZXNew compression in Win10?
79,9%78,3%27177384024440Monkey fastfast = smallest and insane = largest, hands down - not only in averages, but in most files
60,3%61,3%21265540523727wavpack -f
58,6%58,9%204289607444541ALS default
58,6%58,7%20377502286339wavpack -hx
58,5%59,4%20609322746671OptimFROG fastest= --mode fast --optimize none
57,8%57,3%198862468124min18s467ALS -z1
57,8%57,3%198803518836min40s41wavpack -hhx6
57,7%57,8%200513382011287OptimFROG default
57,6%57,0%197787712827min13s1387ALS-z3
57,5%56,9%197323500867min34149ALS -7 -p
57,0%56,4%195760277829min37618OptimFROG heaviest= --mode ultranew-light --optimize high
Underlined: The two "fastest" OptimFROG do benefit from the ordering by unweighted - they are not equally good on the biggest file. Otherwise, weighted and unweighted order are the same.

Since I am obviously touting WavPack here, how about some more figures comparing more settings and with ffmpeg's implementation?
unwwghtencWavPack:
60,5%62,2%46ffmpeg @ default
60,3%61,3%37-f
59,4%60,3%46-g (default)
58,6%58,7%63-hx
58,0%57,9%646-hx4
58,0%58,1%6HOURS10minffmpeg-wavpack @ -compression_level 8
57,8%57,3%36min40s-hhx6
No big surprises here either I'd say, it is known that ffmpeg's compression level "8" is not ... efficient. I did include it to see if this stupid level of effort could push the limits. It couldn't, not even spending 10x the time of -hhx6 (it also seems to decode slightly slower, but who cares?). Unweighted size is bigger than reference wavpack.exe at -hx4.


Files, most can be downloaded with a Soundcloud account, with annotations; percents are "lower is better"; I did remove all metadata first.

CVLT CAST Doom 2016 (yep, a podcast on doom metal)
https://soundcloud.com/cvltnation/cvlt-cast-episode-one-doom-2016 , 79 minutes - several songs, but it is one podcast, I assume they have all been through the same processing.
WARNING of an extreme peak of 9.312567. Have not checked whether that is a CoolEdit format thing.
This varies a lot in compression ratio: OptimFROG got it to 55.2%, WavPack -hhx6 to 56.2%, WavPack up to 64.0 for ffmpeg-default - and then Monkey's at 77 and NTFS at 81.

Kosey: "Sensual Desire" (House)
https://soundcloud.com/kosey-1/kosey-sensual-desire-original-mix-free-download-wav-320kps-links-in-description
65.5 to 68.6 except Monkey's (80) and NTFS (91)

SNCHZ "Destrudo" and "Hate VIP", averaged together before applying unweighted averages
https://soundcloud.com/deceiverdubz/destrudo-free-download and https://soundcloud.com/deceiverdubz/hate-1
71.7 to 73.3 percent for everything but Monkey's (81) and NTFS (90)

The Opium Cartel (that features Tim Bowness of No-Man): "Silence Instead" (prog)
https://soundcloud.com/termo-records/silence-instead
60.3 to 63.3 except Monkey's (82.6) and NTFS (92)

Treha Sektori's "Music, Blood and Spirit", track 1 (dark ambient, 23 minutes) 
http://web.archive.org/web/20170206220402/http://timeisdivine.com/soz/download/Treha_Sekori-MBAS_OST-WAV-SOZ.zip
IIRC, it has been processed as integer (witnessed by signs of clipping) before converted to float.
53.7 to 58.3 except Monkey's (78) and NTFS (91)

Captain Flanger: "Groove On" (Electronic) - user has deleted themselves from Soundcloud.
16 bits exported to 32 - that happens from time to time.  That means it compresses to 30% to 33% except it fools Monkey's big time (80 percent) - the NTFS compression makes it to 60, that is better than Monkey's.
Also the only 48kHz file among these.

And then five tracks off a metal band that has taken their WAVE files off Soundcloud, sorry. The music, for those who care. (https://valemon.bandcamp.com/album/ocean-reveries-2017)
I averaged the five tracks together before taking unweighted averages.
62.5 to 64.9 except Monkey's (84) and NTFS (92)

Xiena Project: "I Can Feel The Rhythm" (dance)
https://soundcloud.com/xiena_project/wav-format-upload-test-can-feel-the-rhythm-192khz-32bit-high-resolution   
192kHz file found and added afterwards; all timings are without this one! (also I forgot to remove the RIFF chunk before compressing, but it was only 34 bytes - no huuge embedded image.)
56.8 to 58.6 except Monkey's (76) and NTFS (91)
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-08-26 16:20:41
The files in "MBPWavs" are 32-bit float as well.
https://github.com/JBFH-Dev/Keystroke-Datasets
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-09-05 15:14:18
Here is a 40 minutes file that exhibits something-above-my-paygrade about CoolEdit/Audition and how the codecs handle it:
https://soundcloud.com/okiamjordank/iamjordank-playlist-project-32bit-44100

810 MB WAVE.
1 GB OptimFROG with --mode normalnew, returns the warning:
WARNING: Adobe Audition bug analyzer found 96439823 corrupted zero samples.
         You should use --correct_audition to correct these corrupted zero
         samples back to 0 and significantly improve compression ratio.

656 MB OptimFROG with --correct_audition --mode normalnew, with the message
Adobe Audition bug: corrected 96439823 corrupted zero samples back to 0.
640 MB WavPack -hx
Then, what happens if I use WavPack's -a?
1 GB there too.

Geeh, this isn't easy for a novice to understand.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-05 15:49:15
If your downloaded copy is not corrupted, it should be a normal 32-bit fixed point file. Try ofr.exe, not off.exe

SHA256
644985b14ca018c74f560328e5d26a3660748b3cb0d4aa8c8303a2b02aa3b6af
iamjordank - playlist project - 32bit-44100 .wav

Code: [Select]
-------------------------------------------------------------------------------
E:\download\iamjordank - playlist project - 32bit-44100 .wav
104216 extra bytes at 850200700
00:40:09.8658049 = 212550164 samples / 2-ch @ 44100 Hz
32-bit fixed point
Bit    Count            Percent
0      214         0.0001006821
1      195         9.174305E-05
2      393         0.0001848975
3      769          0.000361797
4      1542        0.0007254758
5      3105         0.001460832
6      6020         0.002832273
7      11337          0.0053338
8      19245        0.009054333
9      28080           0.013211
10     33140         0.01559161
11     33947         0.01597129
12     41510         0.01952951
13     47350         0.02227709
14     52243         0.02457914
15     64629         0.03040647
16     91045         0.04283459
17     132761        0.06246102
18     189495        0.08915307
19     275697         0.1297092
20     425448         0.2001636
21     724422          0.340824
22     1332435        0.6268803
23     2487198          1.17017
24     4601880          2.16508
25     8450132         3.975594
26     15410568         7.25032
27     27242207        12.81684
28     44333960        20.85812
29     58329023        27.44248
30     42912026        20.18913
31     5268148         2.478543
-------------------------------------------------------------------------------
oldsCool 1.0.0.4
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bryant on 2023-09-05 17:23:07
Yeah, as @bennetng says, that file is not a 32-bit float file at all. It’s 32-bit integers, and that’s what the WAV header says it is. It was created by Adobe Media Encoder (whatever that is) according to the RIFF info at the end.

If you interpret it as float then the compression is going to be really bad (which is what -a is forcing WavPack to do). I’m not sure what Florin means by “correcting corrupted zeros”, but interpreting integer data as float gives wacky stuff.

I can also tell that this file was originally 32-bit float data because the new --optimize-32bit option in WavPack (the one suggested by @bennetng) improves compression by 12%.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-09-05 18:51:42
Assumption is the mother of all .f**c-up, yeah.

As you guys might have guessed, I just took off.exe to be the answer to whether it was float. Because feeding it integer will usually make it throw the error
Code: [Select]
Exception UNKNOWN in file unknown, line 0
description: sample type not yet implemented
Apparently it does not for 32-bit. Tried both extensible and old WAVE ... uh, at least I think I did.

This is not the nicest of bugs. Compressing bad but losslessly is well, at worst only an unfroggish thing to do. But this one asks the user to apply an operation that just destroys the audio.
I should have listened to it before I posted too - this is so destructive I won't even call it "lossy".

Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-07 11:50:29
As you guys might have guessed, I just took off.exe to be the answer to whether it was float.
FWIW off.exe always treat all input files as float, either in standard type 3, or in Cool Edit 16.8 and 24.0 formats, so it must not be used as a tool to identify if a particular input file is float or not. In the case of this 810MB file Audition 1.5 also knows it is not a float file and can be correctly opened without any issue, unlike the example I posted in Reply #24 (https://hydrogenaud.io/index.php/topic,114816.msg1009053.html#msg1009053) where Audition misinterprets as 16.8 float even when I have Audition's format detection option disabled.

WavPack's --optimize-32bit is very useful for identifying if the file is originated from float or not (or vice versa). oldsCool shows "Round Trip: 32f" only when the whole file can be losslessly converted, which means even when there is only one lossy sample it will not round trip. WavPack's --optimize-32bit does not have this limitation.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-09-08 11:02:12
As you guys might have guessed, I just took off.exe to be the answer to whether it was float.
FWIW off.exe always treat all input files as float,
Maybe (... likely ...) all 32 bit files, but certainly not 24 or 16, cf the error message I pasted above.

WavPack's --optimize-32bit is very useful for identifying if the file is originated from float or not (or vice versa)
Originated from, that is question for when input file is integer. ("32-bit integer"?). BTW, to other readers: that option is as of now only in a beta.
I have yet to see WavPack mis-identify input files.

(Not counting all the times I have been scared by wvunpack's "not compatible with this version of WavPack file!" error, which it throws when by mistake I have called wvunpack on something that is not a WavPack file at all.)
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-08 11:40:38
Maybe (... likely ...) all 32 bit files, but certainly not 24 or 16, cf the error message I pasted above.
Of course I meant 32-bit. Because even if someone managed to create a 16 or 24 bits float file it will not be supported by all other third party programs and hardware player. It requires a special decoder:
https://hydrogenaud.io/index.php/topic,90770.0.html

Quote
Originated from, that is question for when input file is integer. ("32-bit integer"?). BTW, to other readers: that option is as of now only in a beta.
I have yet to see WavPack mis-identify input files.
I meant if you see obvious size differences when using --optimize-32bit, then the file is likely converted from another 32-bit format. So this 810MB file is encoded in integer, but because --optimize-32bit can further reduce the file size significantly, then it means the file was originally processed in float, but exported to integer. So it is not about misinterpretation of file encoding formats. The file can be compressed to 537MB without using -h or -x when using --optimize-32bit.

[edit]
Tested other settings using --optimize-32bit and --threads=8
Code: [Select]
561,864,578 gx4.wv
561,532,860 gx6.wv
555,605,858 hx4.wv
555,328,064 hx6.wv
553,126,794 hhx4.wv
553,009,866 hhx6.wv
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-08 14:37:58
Of course I meant 32-bit. Because even if someone managed to create a 16 or 24 bits float file it will not be supported by all other third party programs and hardware player. It requires a special decoder:
https://hydrogenaud.io/index.php/topic,90770.0.html
No longer supported by the latest version of foobar2000, and I don't have anything else which can correctly read these files.
X
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-09-08 16:53:22
Now we're over into integer, but anyway ... still a weirdly-high-bitdepth format to find in the wild, so ...

Maybe (... likely ...) all 32 bit files, but certainly not 24 or 16, cf the error message I pasted above.
Of course I meant 32-bit. Because even if someone managed to create a 16 or 24 bits float file it will not be supported by all other third party programs and hardware player.
That's not "of course" ... if you don't check what you encode, you could for that matter have take 16 bit integer as 32-bit float.

Anyway, only float PCM supported by ffmpeg, are 32 and 64 (both endiannesses for both).

The file can be compressed to 537MB without using -h or -x when using --optimize-32bit.

[edit]
Tested other settings using --optimize-32bit and --threads=8
Code: [Select]
561,864,578 gx4.wv
561,532,860 gx6.wv
555,605,858 hx4.wv
555,328,064 hx6.wv
553,126,794 hhx4.wv
553,009,866 hhx6.wv

Quite impressive, when you look at the competition:
676 534 856 refalac
664 965 457 flac -8p (1.4.3, without keeping foreign metadata)
663 189 990 WavPack -mhx4 (the "m" for MD5 sum ... out of habit)
655 937 031 ALS -7 -p
654 811 611 OptimFrog  --preset max (that is ofr.exe, as appropriate!)

(Is there any audio editing program that works in float, saves in integer, and could save by way of a WavPack plugin?)
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-08 19:03:28
That's not "of course" ... if you don't check what you encode, you could for that matter have take 16 bit integer as 32-bit float.
It is because the differences of 16, 24 and 32-bit formats are not only a simple bit-depth field, it also affects Block Align (which sets the boundary of how many bytes represent a sample in one or more channels), Bytes Per Second and other stuffs. If you indeed have a file which is written by a buggy software that set the bit-depth as 16-bit but storing 32-bit float data then it will just become garbage or unreadable. Audition for example has an option to open as raw and users have to manually enter info like bit-depth, number of channels, sample rate, endianness and so on. It does not detect anything when using this mode. So users need to guess these things manually to reconstruct the header.

It is pretty easy to see how foobar2000 2.0 can seemingly capable of identifying the base format is 16-bit float, but as long as I play it and run it through the integrity checker it shows error. It is based on the assumption that any type 3 format must be float rather than looking into the audio data. When putting this 16-bit float file in oldsCool it also shows the error message "unsupported 16-bit type 3 format" because oldsCool does not have any code which can deal with 16-bit float, and I have no intent to further branch the code to check for example if it is actually type 1 or something else.

If a specific piece of software is advertised to have "forensic" grade of data analysis for recovery etc. then the approach would indeed be trying to check as many things as possible.

The header of .wav file is arranged in a way that it can be easily parsed sequentially instead of requiring the programmer constantly going back and forth. For example, it is perfectly legal for the format to have something like 56-bit integer and 40-bit float, but these formats are unsupported so it is completely meaningless to look any further into the audio data.

There are of course also ACM .wav format where you can stuff things like MP3 into a .wav file but the format tag will not be type 1 or 3. The format tag is placed in a very early position of a .wav file so that the programmer can quickly checks whether a format is supported or not.

As a practical example I reported a bug in someone's software due to misuse of format tag. When it is wrong the file is quickly rejected by everything I have.
https://www.audiosciencereview.com/forum/index.php?threads/beta-test-deltawave-null-comparison-software.6633/post-802776

Therefore the Cool Edit formats are special in a way that because they are documented as type 1, so the code needs to look further to identify these formats.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: soundping on 2023-09-09 00:23:02
Is it possible to use 32-bit float with flac?
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-09 04:41:22
The latest version of FLAC only supports up to 32-bit integer.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-09-09 08:13:59
And the FLAC format does not provide for it.

Big difference between 32-bit integer and 32-bit float, even if it took years until 32-bit integer was implemented as well:
* 32-bit integer was always part of the format: in the frame header, after the bits describing channel assignment, you got bits to describe bit depth, and after 110 for 24, there is 111 for 32. But the available encoders didn't support it until recently.
* But there is nothing for 32-bit float.

Use WavPack.
There's a ton of hypothetical LPCM WAVE files which you won't actually find, which WavPack supports and FLAC does not. Say, sample rate of 1 234 567 Hz.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-09 19:20:08
(Is there any audio editing program that works in float, saves in integer, and could save by way of a WavPack plugin?)
Reaper's highest supported processing bit-depth is 64-bit float, but can also be configured to use 32-bit float processing, and Reaper can export to 32-bit integer WavPack.

Reaper cares about WavPack metadata support too. schwa is a core Reaper developer.
https://hydrogenaud.io/index.php/topic,119263.0.html
https://hydrogenaud.io/index.php/topic,120280.0.html
https://forums.cockos.com/member.php?s=44e829c01da1fc124f5c9be4e37f787e&u=3752
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: Porcus on 2023-09-20 08:41:08
Googled up a few more files at soundcloud - and this time I checked file properties. Largest/longest first:

https://soundcloud.com/erikerixontechno/affenkafig-karnevalsrave-2018-bogen-2-koeln
110 minutes track. That's > 2 GB.

https://soundcloud.com/elektronikos
"Made with REAL synthesizers & drum machines, which for a while seems to have diminished."
102 min in 31 tracks, not counting one 16-bit track.

https://soundcloud.com/uzuknm7fvipv
This user (or bot?) has made a big number of playlists all with the same content (I think), just changing the name and the picture. Anyway, pick one playlist:
74 minutes of 32-bit float in 19 tracks. And about the same in 24-bit.

https://soundcloud.com/user-520842854
54 minutes, the second file is float.

A couple of single tracks:
https://soundcloud.com/tamore-3/the-not-gentle-techno-version-32bit-wave
https://soundcloud.com/outertone/crusadope-back-in-time-32bit-outertone-free-release


... and the THX "Deep note" can be found if sought - who knows how much processing that has been through until someone uploaded it in float.
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: soundping on 2023-09-20 13:13:09
Spread those bits around.

X
Title: Re: 32-bit floating-point (WAV) files "in the wild"
Post by: bennetng on 2023-09-20 19:08:13
https://soundcloud.com/elektronikos
"Made with REAL synthesizers & drum machines, which for a while seems to have diminished."
102 min in 31 tracks, not counting one 16-bit track.

https://soundcloud.com/user-520842854
54 minutes, the second file is float.
These two sound like good candidates for lossy codec listening tests.