Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Improve compression efficiency by treating 32-bit fixed point as float. (Read 9352 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #25 –
Quote from: bryant on
any application that is built with (or uses dynamically) a new libwavpack will transparently get the correct fully lossless decoding. Applications that use older versions of libwavpack (or FFmpeg) will decode such files as lossy.

"new" as in WavPack 5, or "new" as in "5.6.6" meaning it will break compatibility? Spoiler (click to show/hide)

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #26 –
Quote from: Porcus on
Quote from: bryant on
any application that is built with (or uses dynamically) a new libwavpack will transparently get the correct fully lossless decoding. Applications that use older versions of libwavpack (or FFmpeg) will decode such files as lossy.

"new" as in WavPack 5, or "new" as in "5.6.6" meaning it will break compatibility?
"new" as in "new new" (5.6.6). Still haven't decided how to approach this.

@bennetng I created a new Cool Edit / Audition filter with read-only support for this. The gain with float is only around 0.5% and the filter cannot write 32-bit integers anyway (although the improvement would obviously show up then).

However, I also turned on the new multithreading and the speedup is huge.  ;D

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #27 –
Quote from: bryant on
@bennetng I created a new Cool Edit / Audition filter with read-only support for this. The gain with float is only around 0.5% and the filter cannot write 32-bit integers anyway (although the improvement would obviously show up then).

However, I also turned on the new multithreading and the speedup is huge.  ;D
Works great, thank you very much.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #29 –
Here is a gigabyte 32-bit integer in the wild, with an impact of 12 percentage points - closing in on twenty percent: https://soundcloud.com/kyrokotei/in-the-distant-travels .
Remixing dreamy synths into some Alcest-ish post-black metal track. (Original at https://sadnessmusic.bandcamp.com/album/i-want-to-be-there and also put up later at higher price tag.)

1187200698 bytes AIFF at 192 kHz. --optimize-32bit saves around 144 MB both for -f and for -hhx6 --threads (the only multi-threaded I ran), that is about 12 percentage points. Ranging from 17.4 to 19.5 percent - not quite catching 20 :-o
It also will also outdo OptimFROG - them 144 are an order of magnitude above the megabytes that the frog can save over -hhx4 (after converting to WAVE, because the frog doesn't support AIFF.)

At 192 kHz, the file behaves in a way that would be "surprising", hadn't I already been "surprised" at hi-rez artefacts too many times already:
* -fx beats -hhx, -hx and -x (with and without --optimize-32bit). Although at -x4, order is restored.
* flac -0 beats all "single x" (without --optimize-32bit, of course). flac -0 uses fixed predictors only and dual mono. (I gave --keep-foreign-metadata to keep it comparable.) And from -7, FLAC mingles between the x4's.
* Not so surprising is that all apes and ALS are found between "worst x" and "best x4".

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #30 –
So 32-bit integer files are as suspicious as viruses. ::)

Did you (or other members) find any float file which can be improved by using --optimize-32bit? I posted a theoretical example on Reply #17 but I can't find such files in the wild, so Bryant may not include this optimization in release version.
https://github.com/dbry/WavPack/commit/67435bc8d61d73707cf883812c5addc9fa503332#commitcomment-129941853
Quote
I may drop the floating-point portion because the gain is so rare/minimal

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #31 –
I recall I tested --optimize-32bit on float "by mistake", I thought it was an integer-only thing and then a wildcard sent it off on more files than I intended. The files were different, so I realized then it does make a "difference" - but the size savings were small indeed. I tested one now for 0.002 percent size difference, and then (using fb2k bitcompare, uses an official WavPack that doesn't read those 5.6.6 corrections) I infer that --optimize-32bit was invoked in 79 percent of the samples.

So I am not sure this warrants any --optimize-32bit=<0 for none, 1 for integer, 2 for float, 3 for both, defaulting to 1> except in a version for testing. If we postulate that 32-bit integer is "kinda stupid anyway", then it might be a good idea to stick to handling the waste those create, keeping compatibility for everything else.

Spoiler (click to show/hide)

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #32 –
A few more tests on floats with and without --optimize-32bit. All downloaded from https://soundcloud.com/soundchaotic
11 .wav files, ran on default, -hx, -hx4, -hhx6 --threads, all four with and without --optimize-32bit

Actually, --optimize-32bit makes every file bigger.
Depending on how it was implemented, that may indicate (1) nope, not worth it - or (2) not implemented optimally yet. Maybe both. For example, I have no idea whether the optimize-32bit choice can be made per frame or has to be made per file - but looks like fat chance that even a a per-frame brute-force-select between "integer algorithm" and "float algorithm" could make big impact on this corpus.

File sizes for those who care (all with -m):
Code: [Select]
135 892 540 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-g--optimize-32bit.wv
135 891 042 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-g.wv
134 574 576 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hx--optimize-32bit.wv
134 573 078 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hx.wv
131 648 216 SoundChaotic - Brutoaler Supermarkt.-g--optimize-32bit.wv
131 646 774 SoundChaotic - Brutoaler Supermarkt.-g.wv
130 604 540 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-mvhx4--optimize-32bit.wv
130 603 042 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-mvhx4.wv
129 999 292 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hhx6--threads--optimize-32bit.wv
129 997 794 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hhx6--threads.wv
129 527 162 SoundChaotic - Brutoaler Supermarkt.-hx--optimize-32bit.wv
129 525 720 SoundChaotic - Brutoaler Supermarkt.-hx.wv
126 855 624 SoundChaotic - peppr�parat.-hx--optimize-32bit.wv
126 854 176 SoundChaotic - peppr�parat.-hx.wv
124 780 032 SoundChaotic - peppr�parat.-g--optimize-32bit.wv
124 778 584 SoundChaotic - peppr�parat.-g.wv
121 951 848 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-g--optimize-32bit.wv
121 950 572 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-g.wv
120 737 574 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hx--optimize-32bit.wv
120 736 298 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hx.wv
119 672 162 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-mvhx4--optimize-32bit.wv
119 670 886 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-mvhx4.wv
119 222 184 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hhx6--threads--optimize-32bit.wv
119 220 908 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hhx6--threads.wv
115 175 634 SoundChaotic - peppr�parat.-mvhx4--optimize-32bit.wv
115 174 186 SoundChaotic - peppr�parat.-mvhx4.wv
113 144 540 SoundChaotic - Brutoaler Supermarkt.-mvhx4--optimize-32bit.wv
113 143 098 SoundChaotic - Brutoaler Supermarkt.-mvhx4.wv
112 945 604 SoundChaotic - Brutoaler Supermarkt.-hhx6--threads--optimize-32bit.wv
112 944 162 SoundChaotic - Brutoaler Supermarkt.-hhx6--threads.wv
105 812 012 SoundChaotic - peppr�parat.-hhx6--threads--optimize-32bit.wv
105 810 564 SoundChaotic - peppr�parat.-hhx6--threads.wv
 77 824 390 wunderbare schließfächer(191).-g--optimize-32bit.wv
 77 823 644 wunderbare schließfächer(191).-g.wv
 77 705 398 wunderbare schließfächer(191).-hx--optimize-32bit.wv
 77 704 652 wunderbare schließfächer(191).-hx.wv
 77 587 660 wunderbare schließfächer(191).-mvhx4--optimize-32bit.wv
 77 586 914 wunderbare schließfächer(191).-mvhx4.wv
 77 529 236 wunderbare schließfächer(191).-hhx6--threads--optimize-32bit.wv
 77 528 490 wunderbare schließfächer(191).-hhx6--threads.wv
 66 905 162 SoundChaotic - Ave Maria (Bootleg).-g--optimize-32bit.wv
 66 904 490 SoundChaotic - Ave Maria (Bootleg).-g.wv
 66 418 906 SoundChaotic - Ave Maria (Bootleg).-hx--optimize-32bit.wv
 66 418 234 SoundChaotic - Ave Maria (Bootleg).-hx.wv
 66 022 726 SoundChaotic - Ave Maria (Bootleg).-mvhx4--optimize-32bit.wv
 66 022 054 SoundChaotic - Ave Maria (Bootleg).-mvhx4.wv
 65 832 444 SoundChaotic - Ave Maria (Bootleg).-hhx6--threads--optimize-32bit.wv
 65 831 772 SoundChaotic - Ave Maria (Bootleg).-hhx6--threads.wv
 65 590 442 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-g--optimize-32bit.wv
 65 589 054 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-g.wv
 64 921 404 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hx--optimize-32bit.wv
 64 920 678 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hx.wv
 63 017 692 french-flow-top (adilette 4.2).-g--optimize-32bit.wv
 63 017 088 french-flow-top (adilette 4.2).-g.wv
 62 395 966 french-flow-top (adilette 4.2).-hx--optimize-32bit.wv
 62 395 362 french-flow-top (adilette 4.2).-hx.wv
 62 135 468 french-flow-top (adilette 4.2).-mvhx4--optimize-32bit.wv
 62 134 864 french-flow-top (adilette 4.2).-mvhx4.wv
 62 012 880 french-flow-top (adilette 4.2).-hhx6--threads--optimize-32bit.wv
 62 012 276 french-flow-top (adilette 4.2).-hhx6--threads.wv
 60 941 328 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-mvhx4--optimize-32bit.wv
 60 940 602 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-mvhx4.wv
 60 326 594 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hhx6--threads--optimize-32bit.wv
 60 325 868 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hhx6--threads.wv
 51 611 052 SoundChaotic - Teddybären Core.-g--optimize-32bit.wv
 51 610 516 SoundChaotic - Teddybären Core.-g.wv
 51 239 586 SoundChaotic - Teddybären Core.-hx--optimize-32bit.wv
 51 239 050 SoundChaotic - Teddybären Core.-hx.wv
 50 932 904 SoundChaotic - Teddybären Core.-mvhx4--optimize-32bit.wv
 50 932 368 SoundChaotic - Teddybären Core.-mvhx4.wv
 50 842 498 french-flow-top (morgens bin ich immer müde 3).-g--optimize-32bit.wv
 50 841 946 french-flow-top (morgens bin ich immer müde 3).-g.wv
 50 821 604 SoundChaotic - Teddybären Core.-hhx6--threads--optimize-32bit.wv
 50 821 068 SoundChaotic - Teddybären Core.-hhx6--threads.wv
 50 392 592 french-flow-top (morgens bin ich immer müde 3).-hx--optimize-32bit.wv
 50 392 040 french-flow-top (morgens bin ich immer müde 3).-hx.wv
 50 005 536 french-flow-top (morgens bin ich immer müde 3).-mvhx4--optimize-32bit.wv
 50 004 984 french-flow-top (morgens bin ich immer müde 3).-mvhx4.wv
 49 779 740 french-flow-top (morgens bin ich immer müde 3).-hhx6--threads--optimize-32bit.wv
 49 779 188 french-flow-top (morgens bin ich immer müde 3).-hhx6--threads.wv
 38 282 888 Soundchaotic - Du Bist Liebe 3.-g--optimize-32bit.wv
 38 282 180 Soundchaotic - Du Bist Liebe 3.-g.wv
 37 735 506 Soundchaotic - Du Bist Liebe 3.-hx--optimize-32bit.wv
 37 735 148 Soundchaotic - Du Bist Liebe 3.-hx.wv
 36 653 190 Soundchaotic - Du Bist Liebe 3.-mvhx4--optimize-32bit.wv
 36 652 832 Soundchaotic - Du Bist Liebe 3.-mvhx4.wv
 36 377 552 Soundchaotic - Du Bist Liebe 3.-hhx6--threads--optimize-32bit.wv
 36 377 194 Soundchaotic - Du Bist Liebe 3.-hhx6--threads.wv

Quote from: Porcus on
If we postulate that 32-bit integer is "kinda stupid anyway", then it might be a good idea to stick to handling the waste those create, keeping compatibility for everything else.
Does WavPack's compatibility policy extend to 5.70 decoding whatever 5.66 can encode?  :))

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #33 –
The small amount of overhead could be due to the creation of an additional data stream. Notice in your test data that the overhead is proportional to file size, if it is done on per file basis the increment should be fixed.

I think it is like adding a flag in each frame and expecting optimizable incoming data, but when there is none the flag itself will take some space, and there could be some technical difficulties to remove the flag and reclaim the space after knowing the frame cannot be optimized.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #34 –
Quote
Quote from: Porcus on
If we postulate that 32-bit integer is "kinda stupid anyway", then it might be a good idea to stick to handling the waste those create, keeping compatibility for everything else.
Does WavPack's compatibility policy extend to 5.70 decoding whatever 5.66 can encode?  :))
Of course, since this version has been out for a while, and the “help” display mentions nothing about it being experimental, I will leave the decoder portion in so as not to obsolete any existing files, even if I decide to remove the encoder option.

Quote from: bennetng on
The small amount of overhead could be due to the creation of an additional data stream. Notice in your test data that the overhead is proportional to file size, if it is done on per file basis the increment should be fixed.

I think it is like adding a flag in each frame and expecting optimizable incoming data, but when there is none the flag itself will take some space, and there could be some technical difficulties to remove the flag and reclaim the space after knowing the frame cannot be optimized.
Yes, this is exactly right. The new format adds 2 bytes per frame even if it achieves nothing. It would be possible to back up and rewrite if I detect that there was no improvement, but that would slow everything down significantly, and there was actually another reason to not do that. If the new format was invoked, I wanted a frame to indicate it as soon as possible, otherwise it would increase the likelihood of a file being incorrectly identified by an old decoder as lossless that was actually lossy (if the first occurrence of the improved stream did not occur until well into the file). In any event, the loss is very tiny.

As for the float version, the improvement in float files that were derived from 32-bit integer files is about 0.5%, and these are probably not that common. Except maybe files directly converted from 32-bit ADCS? The situations where it makes a big difference is files with mantissas truncated to less than 24 bits, and in these cases the improvement can go over 10%. But I have never seen files like this in the wild, which is why I’m more inclined to not include this feature in the next release.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #35 –
Quote from: bryant on
As for the float version, the improvement in float files that were derived from 32-bit integer files is about 0.5%, and these are probably not that common. Except maybe files directly converted from 32-bit ADCS? The situations where it makes a big difference is files with mantissas truncated to less than 24 bits, and in these cases the improvement can go over 10%. But I have never seen files like this in the wild, which is why I’m more inclined to not include this feature in the next release.
Recently, "32-bit float ADCs" are somewhat popular among field recorders, but they are basically stacking multiple traditional fixed point ADCs together using a floating point DSP after digitization, and the floating point math will output float data which cannot be optimized. Search for patent US9654134 for one of these implementations, as well as the files below for some examples:
https://www.sounddevices.com/sample-32-bit-float-and-24-bit-fixed-wav-files/

Other vendors like Tascam and Zoom also sell floating point recorders. I don't have sample files to try out but I guess the recorded files cannot be optimized as well. So not including this optimization at this moment could be a correct decision.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #36 –
Warning: musing-aloud that possibly does nothing but exposing my ignorance.
But has the odd chance of being relevant when one introduces special treatment of 32-bit PCM in a compatibility-challenging manner:

That big third-party implementation does sometimes write 32-bit WavPack files when the source isn't 32-bit.  I think I have only seen it when source is 24-bit integer (then result is stored as 32-bit integer too ... I think?) - so in that case it is maybe precisely what is being addressed, to the extent that the wasted-bits treatment hasn't fixed it already?
Possibly relevant is how float signals aren't always treated gracefully by applications (WavPack nor Wave nor AIFC) - does that affect their conversion to/from .wv in a way that one might want to address?  If one still considers to give float an alternative treatment that some applications might treat in a lossy way, then ... should it offer to normalize volume in a way that proofs against bad behaviour (with in-between-frames correction)? 
(To be honest I don't really understand what --normalize-floats does, even.)

I have a hunch that it can be summarized into "problem originates soffmwhere else, should be solved there", but my excuse for stupid question is that 32-bit particularities are to be addressed anyway.
 

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #37 –
Quote from: Porcus on
That big third-party implementation does sometimes write 32-bit WavPack files when the source isn't 32-bit.  I think I have only seen it when source is 24-bit integer (then result is stored as 32-bit integer too ... I think?) - so in that case it is maybe precisely what is being addressed, to the extent that the wasted-bits treatment hasn't fixed it already?
Possibly relevant is how float signals aren't always treated gracefully by applications (WavPack nor Wave nor AIFC) - does that affect their conversion to/from .wv in a way that one might want to address?  If one still considers to give float an alternative treatment that some applications might treat in a lossy way, then ... should it offer to normalize volume in a way that proofs against bad behaviour (with in-between-frames correction)? 
(To be honest I don't really understand what --normalize-floats does, even.)
Well, what’s really being addressed here is not the 32-bit peculiarities, which really is just Cool Edit’s decision to exploit WAV file format redundancies to make their own format, but is improving the compression of some 32-bit data that wasn’t anticipated when I implemented the “wasted-bits” strategy. Fortunately, in the integer case, I do efficiently handle all cases of a fixed number of wasted bits, so the 24-bit stored as 32-bit works great, and shouldn't be any different even with --optimize-32bit. The case that wasn’t handled efficiently was a variable number of wasted bits which happens when your source was at some point floating point.

The --normalize-floats may not have been ideally named, but fortunately it’s harmless and won’t do anything unless it’s applicable, which is just with WavPack files created from those same silly Cool Edit WAVs. All float files from everywhere else are normalized to +/- 1.0 already, even though they may have values that exceed that (one of float’s advantages in fact).