Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Improve compression efficiency by treating 32-bit fixed point as float. (Read 5914 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #25 –
Quote from: bryant on
any application that is built with (or uses dynamically) a new libwavpack will transparently get the correct fully lossless decoding. Applications that use older versions of libwavpack (or FFmpeg) will decode such files as lossy.

"new" as in WavPack 5, or "new" as in "5.6.6" meaning it will break compatibility? Spoiler (click to show/hide)

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #26 –
Quote from: Porcus on
Quote from: bryant on
any application that is built with (or uses dynamically) a new libwavpack will transparently get the correct fully lossless decoding. Applications that use older versions of libwavpack (or FFmpeg) will decode such files as lossy.

"new" as in WavPack 5, or "new" as in "5.6.6" meaning it will break compatibility?
"new" as in "new new" (5.6.6). Still haven't decided how to approach this.

@bennetng I created a new Cool Edit / Audition filter with read-only support for this. The gain with float is only around 0.5% and the filter cannot write 32-bit integers anyway (although the improvement would obviously show up then).

However, I also turned on the new multithreading and the speedup is huge.  ;D

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #27 –
Quote from: bryant on
@bennetng I created a new Cool Edit / Audition filter with read-only support for this. The gain with float is only around 0.5% and the filter cannot write 32-bit integers anyway (although the improvement would obviously show up then).

However, I also turned on the new multithreading and the speedup is huge.  ;D
Works great, thank you very much.

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #29 –
Here is a gigabyte 32-bit integer in the wild, with an impact of 12 percentage points - closing in on twenty percent: https://soundcloud.com/kyrokotei/in-the-distant-travels .
Remixing dreamy synths into some Alcest-ish post-black metal track. (Original at https://sadnessmusic.bandcamp.com/album/i-want-to-be-there and also put up later at higher price tag.)

1187200698 bytes AIFF at 192 kHz. --optimize-32bit saves around 144 MB both for -f and for -hhx6 --threads (the only multi-threaded I ran), that is about 12 percentage points. Ranging from 17.4 to 19.5 percent - not quite catching 20 :-o
It also will also outdo OptimFROG - them 144 are an order of magnitude above the megabytes that the frog can save over -hhx4 (after converting to WAVE, because the frog doesn't support AIFF.)

At 192 kHz, the file behaves in a way that would be "surprising", hadn't I already been "surprised" at hi-rez artefacts too many times already:
* -fx beats -hhx, -hx and -x (with and without --optimize-32bit). Although at -x4, order is restored.
* flac -0 beats all "single x" (without --optimize-32bit, of course). flac -0 uses fixed predictors only and dual mono. (I gave --keep-foreign-metadata to keep it comparable.) And from -7, FLAC mingles between the x4's.
* Not so surprising is that all apes and ALS are found between "worst x" and "best x4".

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #30 –
So 32-bit integer files are as suspicious as viruses. ::)

Did you (or other members) find any float file which can be improved by using --optimize-32bit? I posted a theoretical example on Reply #17 but I can't find such files in the wild, so Bryant may not include this optimization in release version.
https://github.com/dbry/WavPack/commit/67435bc8d61d73707cf883812c5addc9fa503332#commitcomment-129941853
Quote
I may drop the floating-point portion because the gain is so rare/minimal

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #31 –
I recall I tested --optimize-32bit on float "by mistake", I thought it was an integer-only thing and then a wildcard sent it off on more files than I intended. The files were different, so I realized then it does make a "difference" - but the size savings were small indeed. I tested one now for 0.002 percent size difference, and then (using fb2k bitcompare, uses an official WavPack that doesn't read those 5.6.6 corrections) I infer that --optimize-32bit was invoked in 79 percent of the samples.

So I am not sure this warrants any --optimize-32bit=<0 for none, 1 for integer, 2 for float, 3 for both, defaulting to 1> except in a version for testing. If we postulate that 32-bit integer is "kinda stupid anyway", then it might be a good idea to stick to handling the waste those create, keeping compatibility for everything else.

Spoiler (click to show/hide)

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #32 –
A few more tests on floats with and without --optimize-32bit. All downloaded from https://soundcloud.com/soundchaotic
11 .wav files, ran on default, -hx, -hx4, -hhx6 --threads, all four with and without --optimize-32bit

Actually, --optimize-32bit makes every file bigger.
Depending on how it was implemented, that may indicate (1) nope, not worth it - or (2) not implemented optimally yet. Maybe both. For example, I have no idea whether the optimize-32bit choice can be made per frame or has to be made per file - but looks like fat chance that even a a per-frame brute-force-select between "integer algorithm" and "float algorithm" could make big impact on this corpus.

File sizes for those who care (all with -m):
Code: [Select]
135 892 540 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-g--optimize-32bit.wv
135 891 042 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-g.wv
134 574 576 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hx--optimize-32bit.wv
134 573 078 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hx.wv
131 648 216 SoundChaotic - Brutoaler Supermarkt.-g--optimize-32bit.wv
131 646 774 SoundChaotic - Brutoaler Supermarkt.-g.wv
130 604 540 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-mvhx4--optimize-32bit.wv
130 603 042 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-mvhx4.wv
129 999 292 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hhx6--threads--optimize-32bit.wv
129 997 794 Jasmine Thompson & Calum Scott - Love is just a Word (SoundChaotic Hard-Kick-Bootleg).-hhx6--threads.wv
129 527 162 SoundChaotic - Brutoaler Supermarkt.-hx--optimize-32bit.wv
129 525 720 SoundChaotic - Brutoaler Supermarkt.-hx.wv
126 855 624 SoundChaotic - peppr�parat.-hx--optimize-32bit.wv
126 854 176 SoundChaotic - peppr�parat.-hx.wv
124 780 032 SoundChaotic - peppr�parat.-g--optimize-32bit.wv
124 778 584 SoundChaotic - peppr�parat.-g.wv
121 951 848 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-g--optimize-32bit.wv
121 950 572 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-g.wv
120 737 574 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hx--optimize-32bit.wv
120 736 298 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hx.wv
119 672 162 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-mvhx4--optimize-32bit.wv
119 670 886 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-mvhx4.wv
119 222 184 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hhx6--threads--optimize-32bit.wv
119 220 908 Monsieur Periné feat. Vicente García - Nuestra Canción (feat. Vicente García)(SoundChaotic Hard-Kick-Vocal-Edit).-hhx6--threads.wv
115 175 634 SoundChaotic - peppr�parat.-mvhx4--optimize-32bit.wv
115 174 186 SoundChaotic - peppr�parat.-mvhx4.wv
113 144 540 SoundChaotic - Brutoaler Supermarkt.-mvhx4--optimize-32bit.wv
113 143 098 SoundChaotic - Brutoaler Supermarkt.-mvhx4.wv
112 945 604 SoundChaotic - Brutoaler Supermarkt.-hhx6--threads--optimize-32bit.wv
112 944 162 SoundChaotic - Brutoaler Supermarkt.-hhx6--threads.wv
105 812 012 SoundChaotic - peppr�parat.-hhx6--threads--optimize-32bit.wv
105 810 564 SoundChaotic - peppr�parat.-hhx6--threads.wv
 77 824 390 wunderbare schließfächer(191).-g--optimize-32bit.wv
 77 823 644 wunderbare schließfächer(191).-g.wv
 77 705 398 wunderbare schließfächer(191).-hx--optimize-32bit.wv
 77 704 652 wunderbare schließfächer(191).-hx.wv
 77 587 660 wunderbare schließfächer(191).-mvhx4--optimize-32bit.wv
 77 586 914 wunderbare schließfächer(191).-mvhx4.wv
 77 529 236 wunderbare schließfächer(191).-hhx6--threads--optimize-32bit.wv
 77 528 490 wunderbare schließfächer(191).-hhx6--threads.wv
 66 905 162 SoundChaotic - Ave Maria (Bootleg).-g--optimize-32bit.wv
 66 904 490 SoundChaotic - Ave Maria (Bootleg).-g.wv
 66 418 906 SoundChaotic - Ave Maria (Bootleg).-hx--optimize-32bit.wv
 66 418 234 SoundChaotic - Ave Maria (Bootleg).-hx.wv
 66 022 726 SoundChaotic - Ave Maria (Bootleg).-mvhx4--optimize-32bit.wv
 66 022 054 SoundChaotic - Ave Maria (Bootleg).-mvhx4.wv
 65 832 444 SoundChaotic - Ave Maria (Bootleg).-hhx6--threads--optimize-32bit.wv
 65 831 772 SoundChaotic - Ave Maria (Bootleg).-hhx6--threads.wv
 65 590 442 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-g--optimize-32bit.wv
 65 589 054 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-g.wv
 64 921 404 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hx--optimize-32bit.wv
 64 920 678 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hx.wv
 63 017 692 french-flow-top (adilette 4.2).-g--optimize-32bit.wv
 63 017 088 french-flow-top (adilette 4.2).-g.wv
 62 395 966 french-flow-top (adilette 4.2).-hx--optimize-32bit.wv
 62 395 362 french-flow-top (adilette 4.2).-hx.wv
 62 135 468 french-flow-top (adilette 4.2).-mvhx4--optimize-32bit.wv
 62 134 864 french-flow-top (adilette 4.2).-mvhx4.wv
 62 012 880 french-flow-top (adilette 4.2).-hhx6--threads--optimize-32bit.wv
 62 012 276 french-flow-top (adilette 4.2).-hhx6--threads.wv
 60 941 328 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-mvhx4--optimize-32bit.wv
 60 940 602 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-mvhx4.wv
 60 326 594 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hhx6--threads--optimize-32bit.wv
 60 325 868 Koen Groeneveld - Tonight The Music Seems So Loud (SoundChaotic Early Frantic Style).-hhx6--threads.wv
 51 611 052 SoundChaotic - Teddybären Core.-g--optimize-32bit.wv
 51 610 516 SoundChaotic - Teddybären Core.-g.wv
 51 239 586 SoundChaotic - Teddybären Core.-hx--optimize-32bit.wv
 51 239 050 SoundChaotic - Teddybären Core.-hx.wv
 50 932 904 SoundChaotic - Teddybären Core.-mvhx4--optimize-32bit.wv
 50 932 368 SoundChaotic - Teddybären Core.-mvhx4.wv
 50 842 498 french-flow-top (morgens bin ich immer müde 3).-g--optimize-32bit.wv
 50 841 946 french-flow-top (morgens bin ich immer müde 3).-g.wv
 50 821 604 SoundChaotic - Teddybären Core.-hhx6--threads--optimize-32bit.wv
 50 821 068 SoundChaotic - Teddybären Core.-hhx6--threads.wv
 50 392 592 french-flow-top (morgens bin ich immer müde 3).-hx--optimize-32bit.wv
 50 392 040 french-flow-top (morgens bin ich immer müde 3).-hx.wv
 50 005 536 french-flow-top (morgens bin ich immer müde 3).-mvhx4--optimize-32bit.wv
 50 004 984 french-flow-top (morgens bin ich immer müde 3).-mvhx4.wv
 49 779 740 french-flow-top (morgens bin ich immer müde 3).-hhx6--threads--optimize-32bit.wv
 49 779 188 french-flow-top (morgens bin ich immer müde 3).-hhx6--threads.wv
 38 282 888 Soundchaotic - Du Bist Liebe 3.-g--optimize-32bit.wv
 38 282 180 Soundchaotic - Du Bist Liebe 3.-g.wv
 37 735 506 Soundchaotic - Du Bist Liebe 3.-hx--optimize-32bit.wv
 37 735 148 Soundchaotic - Du Bist Liebe 3.-hx.wv
 36 653 190 Soundchaotic - Du Bist Liebe 3.-mvhx4--optimize-32bit.wv
 36 652 832 Soundchaotic - Du Bist Liebe 3.-mvhx4.wv
 36 377 552 Soundchaotic - Du Bist Liebe 3.-hhx6--threads--optimize-32bit.wv
 36 377 194 Soundchaotic - Du Bist Liebe 3.-hhx6--threads.wv

Quote from: Porcus on
If we postulate that 32-bit integer is "kinda stupid anyway", then it might be a good idea to stick to handling the waste those create, keeping compatibility for everything else.
Does WavPack's compatibility policy extend to 5.70 decoding whatever 5.66 can encode?  :))

Re: Improve compression efficiency by treating 32-bit fixed point as float.

Reply #33 –
The small amount of overhead could be due to the creation of an additional data stream. Notice in your test data that the overhead is proportional to file size, if it is done on per file basis the increment should be fixed.

I think it is like adding a flag in each frame and expecting optimizable incoming data, but when there is none the flag itself will take some space, and there could be some technical difficulties to remove the flag and reclaim the space after knowing the frame cannot be optimized.