HydrogenAudio

Lossless Audio Compression => WavPack => Topic started by: shadowking on 2021-01-16 12:38:38

Title: -h as the new default mode
Post by: shadowking on 2021-01-16 12:38:38
Perception is everything to many. Ones that look at percent compressed vs flac vs ones that look
for the 'fastest' decoding even though both are meanigless in modern scenarios.

Still, It will stick that both have same compression but flac is 'fast' and wavpack is 'slow' due to
looking at some 'studies' and graphs.

If -h became the new default, Wavpack compression on paper would look much better than flac. The speed
would still be very fast both ways. This could help popularity. It would stick in peoples mind that wavpack
compress better than flac and most other codecs - except TAK & APE.  Tak is closed source. APE -c2000  is fast but slow to decode.
APE -c1000 compression is similar to wavpack -h but still slower in decoding . WV has better license then APE.

I think win win for -h as default.  -hh can be new -h, normal could be 'fast' and fast could become 'very fast'

Title: Re: -h as the new default mode
Post by: bryant on 2021-01-20 03:38:22
Hi shadowking, and thanks for the suggestion! I actually have thought about this and I do use the high mode the most, although sometimes I use -x6 as the default when I’m not in a hurry.

I think though that there are other reasons for people not using WavPack unrelated to this, like limited hardware support and confusion about all its additional features. If people are looking for a codec just to rip their CDs to I suggest FLAC.

If someone needs something unique that WavPack has, like 32-bit audio (integer or float), or DSD, or hybrid/lossy, or dozens of channels, etc, then obviously I recommend it.

And at this point I think it would only add confusion to either rename the modes I have, or force high mode on by default and create a new option for “normal” (which is probably the way I’d go).

BTW, do you really still use 4.80?  :)
Title: Re: -h as the new default mode
Post by: shadowking on 2021-01-20 08:20:10
Hi David,

Last night I had another thought, This time much simpler. The idea is to hide the complicated options
and emphasise 'presets' for the most common lossless / hybrid options. I took ideas from zip, lzop, flac, mac cli tools

Proposal;

lossless
-0   = -f
-1   =  -x
-2   = -hx (recommended)


Hybrid
-hybrid0 = -b256hx3c
-hybrid1 = -b320hx3c
-hybrid2 = -b384hx4c

--
Yes still on 4.8. I really should try latest or just delete my dumb signature :)
Title: Re: -h as the new default mode
Post by: bryant on 2021-01-23 22:52:10
Yes, that's actually kind of a good idea that I had not thought of. In fact, FFmpeg works that way (although I don't agree with what they've chosen, for example you can only get the high -x modes with -hh).

Of course, it doesn't address the original issue unless we force one of them as default, but it's something to think about.

As for the 4.8 issue, you know you're missing out on the new quick verify feature by using that, right?  :)
Title: Re: -h as the new default mode
Post by: DARcode on 2021-01-27 23:02:27
Nice ideas by shadowking, course as long as the "complicated options" remain available (I guess that's what he means by hiding them).

P.S.
Stil on 4.80.0?! How can you live without wvtag ;) ? Come on, man :)) !
Title: Re: -h as the new default mode
Post by: DARcode on 2021-11-10 23:48:48
Has this gone anywhere since please? Thanks.
Title: Re: -h as the new default mode
Post by: Porcus on 2021-11-12 00:08:27
What one could do, is to elevate the "-x" to be the preset. -x0 to -x6.
Then take the fast/normal/high/very high mode to be a modifier. Akin to how TAK has -p as the preset and then with modifiers, except that I would propose -x0 as default if -h is default. "-x" could still mean "-x1".

If one is to go for -h as new default mode, one could have a letter for current "normal". "g" works. Between f(ast) and h(igh), there is g(ood).  (Heck, if others can call some preset "best", you can call that mode "good".) Or you can consider f and ff for the faster than high. Of f and ff, so that -x0ff means current --fast. And you can keep "--fast: synonymous to -x0ff".

-x1hh ... or, to highlight the warning against it for portables, require it to be given separately.


All this won't change much but the mindset; and it should, if you embrace -h (and still warn against -hh), for then the x is what the user would have to select unless diving into the detail.

"purely numerical" presets like FLAC have an issue with WavPack: as long as one has already decided that -hx shall be accepted as -h -x, then it would confuse a user if -3 -x is not the same as -x3.



Now whether going for -h is a good or a bad idea is another issue; it is a double-edged sword in that it puts the default to even slower decoding - but on the other hand, those obsessed with that might look elsewhere.

But I would think twice before letting new -hx be default, as it takes twice the time to encode as current default does. That is quite some change. -h is already smaller than any FLAC.  And with ktf's freshly posted flac beta, it is about the speed of -7.




Oh, and I think the manual does something very wrong when speaking of (very) high "quality" for -h and -hh. Very high compression effort?
Title: Re: -h as the new default mode
Post by: Porcus on 2021-11-12 10:20:45
... but if one wants to pick presets from a selection of reasonable settings, then ... I put my computer to a couple of overnight tasks with around 3 GB CDDA and 3 GB 96/24 (well a few 88.2/24), same corpus as here (https://hydrogenaud.io/index.php?topic=120158.msg1003738#msg1003738). Warning: Rock to metal, not much classical. But if one is to pick presets, I think the considerations below are a significant point (and then others can fill in with more testing).

The below table is interesting. The -x3 setting is frankly quite useless in this test. -x1 is so excellent on this hirez corpus that it is tempting to make it mandatory, despite whatever I said in the previous post about a new default taking so much more time.

Explanation: CDDA to the left, hi-rez to the right. Size and seconds are self-explanatory; times are taken from Igor Pavlov's timer64.exe ("process time", I think it ignores time to access files on the SDD.)
The -Δt/Δsize means the cost in seconds per megabyte saved compared to the setting immediately above it. Reasonably, this should be progressive within the mode (which means everything else equal): you would have to pay more seconds for another megabyte reduction if the file is already smaller.
-x3 violates that (except in fast mode): if you are willing to pay for going -x2 to -x3, you get better per second going -x4. Indeed, for the higher-rez material, -x3 is both bigger and slower than -x2, indicated by "BIGSLOW". (That means that a number immediately below "BIGSLOW" is low because it is in comparison to something bad.)
The other "bigger" indicated, are when mode is switched and file size goes up. Then the negatives are when higher mode yields both faster encoding and smaller file.

I also question the usefulness of -x5. It is already so slow that you would leave it in the background for long rather than watching it. If you are first going -x5, why not leave it overnight or for the week-end for -x6?

To get a reference to FLAC: ktf's newly posted beta, which improves quite a bit, beats WavPack on hirez unless you go up to -h -x4. (This edited after a big mistake.)
But for CDDA: -7 compresses to both time&size between normal and -x1, and -8 to both time&size between -x1 and -x2.  -x2 is likely smaller than any FLAC ever - for this CDDA corpus.

settingbytes CDDAsec-Δt/Δsize-Δt/Δsizesecbytes hirez
-f221075761468582354239318
-f -x1220207276210041832323061806
-f -x221986096181319161062321563994
-f -x3219752275020467561662320481204
-f -x42196193122461193653562317536024
-f -x521959122765713931064202316936660
-f -x6219582075065793871324912316926664
[n]217250079482−25−8612265273710
[n] -x1215924661213341962236071090
[n] -x2215664888619423351382234880022
[n] -x32155729900342161BIGSLOW2422238618428
[n] -x421488522401220128209642201779442
[n] -x52148336890171796314514402198486522
[n] -x62146702532332998620829672191150966
-h2143903450104−1152bigger812242607186
-h -x121381720661761321312217805534
-h -x22136565630279641162082217136902
-h -x321364302685351889BIGSLOW4002219912878
-h -x4213128279819642781516722137653818
-h -x52130429436267683539822492136206612
-h -x621300691323647269562632852134552628
-hh2133201720134biggerbigger1002233102796
-hh -x121287733982342321722203717666
-hh -x221277511103921551252912202768210
-hh -x321275147688061752BIGSLOW5852203496882
-hh -x4212463163628337032324772121330764
-hh -x5212374466045661954135838762120301594
-hh -x62123586950625510708292154832119751366
Title: Re: -h as the new default mode
Post by: Porcus on 2021-11-12 13:39:10
-x3 is both bigger and slower than -x2, indicated by "BIGSLOW". (That means that a number immediately below "BIGSLOW" is low because it is in comparison to something bad.)

Oh, but for hirez the numbers indicate something to scratch one's head over. Disregarding -f:

-x2 to -x4 (normal mode): Shaving 1 MB costs 25 seconds. That is cheaper than going -x1 to -x2 (35 sec per MB s(h)aved).
-x2 to -x4 (high mode): Shaving 1 MB costs 18 seconds. That is much cheaper than -x1 to -x2 (116 sec per MB).
-x2 to -x4 (hh mode): Shaving 1 MB costs 27 seconds. That is much cheaper than -x1 to -x2 (127 sec per MB).

So a reasonable interpretation (well the official documentation pretty much says the same) is that going for high x is a speculative investment of time; one that could yield good return if and when lower x (including no x) might not be well optimized for the particular signal in question.
Now this single test (no classical music, yeah ...) indicates that x2 might not be well-optimized for high-frequency content; x3 might not be well optimized at all?


Then also to think over: getting a MB for half a minute is pretty good bang for buck if you are in the market for high compression - but that number is because WavPack performs quite bad on these files without -x4: ktf's new FLAC beta at -8 saves two percent (45 megabytes) over -hhx and at faster encoding time.


This would provoke the question: should presets be "same for all signals" or should the encoder read sample rate and shift settings accordingly? But no, not in this case: a default setting cannot run fifteen times as long on a GB of 96/24, compared to a GB of CDDA.
(But @bryant : Is there maybe something you should look at about selecting block size from sample rate? Also I noticed from timer64 output that WavPack spends only a fraction of the memory that flac.exe does, but that might be due to flac.exe trying to buffer before writing to limit the number of i/o operations and the file fragmenting that would follow?)


-h is already smaller than any FLAC. 

That was CDDA.
Title: Re: -h as the new default mode
Post by: evan123 on 2022-07-15 00:57:38
If you would consider changing defaults...  -x (-x4?) might also be worth considering, given encoding is a one-time task, and decoding performance is not affected and compression is comparable to -h alone.
Title: Re: -h as the new default mode
Post by: shadowking on 2022-07-15 16:03:25
If you would consider changing defaults...  -x (-x4?) might also be worth considering, given encoding is a one-time task, and decoding performance is not affected and compression is comparable to -h alone.

Your right, -x4 does a lot of work without being very slow.
-hx4 takes it even further while still retaining OK speed.
The effect isn't as pronounced with -hhx4.  The -hh doesn't benefit as much.
Plain -h and  -hh work great for red book CD audio. pretty fast.
I guess looking at porcus chart, The normal mode is well balanced and very fast.
normal + x1 is also reasonable for compression vs speed.
Title: Re: -h as the new default mode
Post by: Porcus on 2022-07-15 16:57:38
David can answer for himself, but as far as I understand, WavPack has found a niche as a .wav substitute in CoolEdit and Audition (since it restores the .wav file bit-exactly with file structure and metadata).
That is not an "encode once" situation - you will wait for an encode every time you save. To slow down the default will be a nasty surprise to those users, and for all that I know they might make up a significant amount of the WavPack userbase.

Also, the new 5.5.0 lets you set your own options to the executable for drag and drop (yeah, touting my own idea here): https://hydrogenaud.io/index.php/topic,122626.msg1012159.html#msg1012159
Combine that with WavPack's nice recompression feature: it is now useful even to those who aren't happy about the command line. Copy David's suggested filename, and you have a WavPack exe that encodes to -hx with MD5 and verification - and drop all your default-saved .wv files onto it. Put a "4" or a "6" after the "x" for overnight jobs or when you are going on holiday.


As for the -x4 size s(h)avings:
ktf's comparison (most recent edition) confirms what the table above suggested, namely that -x4 makes a difference on high resolution material (https://hydrogenaud.io/index.php/topic,122355.0.html) (costly yes, but those compression figures for an asymmetric fast-decoding format would have looked awesome hadn't TAK moved the bar up). But also it is very expensive for a percent at CDDA (https://hydrogenaud.io/index.php/topic,122508.0.html).  You will find the multi-channel material around the middle of the two (https://hydrogenaud.io/index.php/topic,122317.0.html).
Even if the figures on bonkers-resolution files are impressive, it looks like a very bad idea to change the default to target the scarcer material.


I'd say, teach WavPack users the joys of recompression.
Title: Re: -h as the new default mode
Post by: bennetng on 2022-07-15 18:45:01
Even for DAWs using the standard file formats, it would also be handy to use a fast WavPack setting for the freeze files. Reaper for example can freeze to WavPack. Modern and advanced DAWs are often non-destructive. For example, in a complex orchestral project which uses dozens of Gigabytes of samples and hundreds of polyphony:
https://youtu.be/skAxKtDXnpU

Let's say you have the piano partially finished or want to deal with it later, you can let the DAW render that part to audio temporary to save CPU/RAM. Since the MIDI data for the piano is still there, at some point after you modified the strings section and find that the piano arrangement needed to be modified, you can unfreeze the piano, which means delete the rendered piano audio and let the plugin generating the piano note in real time again when you play the MIDI keyboard or edit the notes using mouse. Depends on the DAW and plugin it is even possible to selectively freeze a musical phrase instead of the whole track.
https://youtu.be/FUOLnz2cvB4

Also, with WavPack's float support, there is no need to worry about the frozen parts are clipped or not, as the rendering may also take some time and floating point will ensure there is no need to retry due to clipping. Some instruments, especially the electronic synthesizers, can have some dramatic and time-varying filters to induce clipping if rendered in fixed point.
Title: Re: -h as the new default mode
Post by: bryant on 2022-07-17 19:41:14
The two fastest encoding modes of WavPack are the non-extra “fast” mode and then the non-extra “normal” mode (the current default). Looking at ktf’s comparisons (for example, here (https://hydrogenaud.io/index.php/topic,122508.0.html)) it’s obvious that the first compression gain (from “fast” to “normal”) is pretty much the biggest jump. Except for hi-res, going all the way to -hhx6 doesn’t duplicate that improvement, so I think that makes the current default the “sweet spot” for encoding performance. And, by coincidence, it’s also the closest match to FLAC -5 (which is also, even more obviously, its “sweet spot” in ktf’s graphs).

My personal default setting is -hx because it gives another nice compression bump, but is over twice as slow encoding as the default, and also decodes slower. If someone else likes that they can rename their executable to include it (as Porcus suggests) or can even modify the source and build a custom executable (now that -g and -x0 are available to cancel those selections). But to require the user to specify additional (new!) options to get to the encoding sweet spot doesn't seem right.
Title: Re: -h as the new default mode
Post by: Aleron Ives on 2022-07-17 20:47:45
My solution is to just write a simple script with the CLI options I want, and then do all encoding by invoking the script, rather than running the program directly. These days computers are so fast I don't see much benefit in using anything but the most aggressive settings, as you only have to spend time encoding the file once, but the space savings are permanent. I can wait a little while longer. :)
Title: Re: -h as the new default mode
Post by: Porcus on 2022-07-17 23:36:18
My solution is to just write a simple script with the CLI options I want, and then do all encoding by invoking the script, rather than running the program directly. These days computers are so fast I don't see much benefit in using anything but the most aggressive settings, as you only have to spend time encoding the file once, but the space savings are permanent. I can wait a little while longer. :)

If anyone has a .bat that would do the following, then post it.
For each file (possibly traversing directory, possibly allowing drag and drop?)
- wvunpack -ss to find the mode, and reject file if it isn't a valid WavPack file or if it is high/extra high, but if it is fast or normal (or none given, isn't that what ffmpeg does?):
- recompress using -hx4 [or suitable setting]

As long as user is disciplined enough to stick to default or -f or ffmpeg (ffmpeg's default is like -fj0!) for initial encode, then this .bat can be run periodically.
Title: Re: -h as the new default mode
Post by: Porcus on 2022-07-18 00:08:48
t -g and -x0 are available
... though not in the release notes.
Title: Re: -h as the new default mode
Post by: Aleron Ives on 2022-07-18 03:32:35
If anyone has a .bat that would do the following, then post it.
Although this is possible, it's much simpler to just blindly recompress all .wv files using your new preferred settings, as you only need to run the script once. After that, you can just be sure to use those settings when adding new material to your collection, so you can be sure everything was compressed with the same settings.
Title: Re: -h as the new default mode
Post by: bennetng on 2022-07-18 11:12:32
bat/cmd can have strange quirks when dealing with foreign unicode characters, for example if Windows' default codepage is Chinese but you are dealing with files with Japanese file name. Also some reserved characters like brackets and such can screw up things. One needs to be a guru of the bat/cmd language to debug these issues.

The changing exe filename approach if done correctly (both developers and users) can somewhat simplify things without using a GUI frontend.

.ini files alongside with the .exe file could be another approach as well.
Title: Re: -h as the new default mode
Post by: Aleron Ives on 2022-07-18 20:00:25
Yeah, I try to use only ASCII characters in filenames and reserve non-ASCII characters for tags so that all programs will be able to parse the filenames correctly.
Title: Re: -h as the new default mode
Post by: shadowking on 2023-08-12 16:18:07
I thought again recently about this.  I now think nearly everything is fine as it is.
The normal mode is fine as default. The -h should be 'high mode' (vs quality)  just like -f is fast mode.
That sounds good in regards to compression (default is in the middle).
In the --help section it can also be mentioned as having quality related effect.
-c should -not- create the .wvc , Instead wvc should be created automatically whenever -b is
invoked.  This way all output is lossless and those objecting to using Wavpack on grounds that they might
end up with lossy only can step back  and dont have to run to flac etc..
Title: Re: -h as the new default mode
Post by: shadowking on 2023-08-12 16:23:49
Maybe  'complex' options can be moved to --help.  typing just wavpack can look very KISS  ;

 WAVPACK  Hybrid Lossless Audio Compressor  

 Usage:   WAVPACK [-options] infile[.wav]|infile.wv|- [outfile[.wv]|outpath|-]
             (infile may contain wildcards: ?,*)

 Options:
          -f  = fast mode (fast, but some compromise in compression ratio)
          -h  = high mode (better compression ratio, but slower)
          -v  = verify output file integrity after write (no pipes)
          -x  = extra encode processing (no decoding speed penalty)
          --help = complete help
Title: Re: -h as the new default mode
Post by: shadowking on 2023-08-12 16:31:32
Or without -x and -h appearing first ;


 WAVPACK  Hybrid Lossless Audio Compressor 

 Usage:   WAVPACK [-options] infile[.wav]|infile.wv|- [outfile[.wv]|outpath|-]
             (infile may contain wildcards: ?,*)

 Options:
          -h = high mode (better compression ratio, but slower)
          -f  = fast mode (fast, but some compromise in compression ratio)
          -v  = verify output file integrity after write (no pipes)
          --help = complete help
Title: Re: -h as the new default mode
Post by: Porcus on 2023-08-12 18:38:43
-c should -not- create the .wvc , Instead wvc should be created automatically whenever -b is
invoked.  This way all output is lossless and those objecting to using Wavpack on grounds that they might
end up with lossy only can step back  and dont have to run to flac etc..
The 5.6.4 beta takes a different approach where lossless-users don't have to touch the "-b". From the short help:
Code: [Select]
          -c<n> = shortcut for '-b<n> -c' to specify hybrid lossless mode
                  with a single option, n = 2.0 to 23.9 bits/sample, or
                                        n = 24-9600 kbits/second (kbps)

This way you don't have to "invoke lossy and then invoke correction" at the risk of forgetting the latter. You can just skip -b entirely.
Also it doesn't break old behaviour for anyone who used it correctly.

Maybe  'complex' options can be moved to --help.  typing just wavpack can look very KISS  ;
You keep getting more of a point here as the short help grows longer: by 5.6.4 now it exceeds the default cmd window's 25 lines. Surely there is a setting to alter those "25". But now the "Usage" falls out of the screen.
If it is to be shortened down, who knows what is to go. (By the way, I do not like the phrase "quality" for lossless ... though the text should be valid for lossy too.)

Anyway, if formats stay in the short help, here is a tweak to get slightly information without altering layout - it becomes one character wider in the top-right column, but note the AIFF column width matches the one immediately above. Also the Apple formats end up on one line, just as the DSD formats do.
Code: [Select]
​ Formats: .wav (default, bwf/rf64 okay)  .w64 (Sony Wave64)
          .aif/.aiff/.aifc (Apple AIFF)  .caf (Core Audio Format)
          .dff (Philips DSDIFF)          .dsf (Sony DSD stream)
          .wv (transcode from existing WavPack file, with tags)
Title: Re: -h as the new default mode
Post by: shadowking on 2023-08-13 11:13:24
 WAVPACK  Hybrid Lossless Audio Compressor  Win64 Version 5.6.0
 Copyright (c) 1998 - 2022 David Bryant.  All Rights Reserved.

 Usage:   WAVPACK [-options] infile[.wav]|infile.ext|- [outfile[.wv]|outpath|-]
          WAVPACK --drop [-options] infile[.wav]|infile.ext [...]
             (infile may contain wildcards: ?,*)


 Formats: .wav (default, bwf/rf64 okay)  .aif (Apple AIFF)
          .w64 (Sony Wave64)             .caf (Core Audio Format)
          .dff (Philips DSDIFF)          .dsf (Sony DSD stream)
          .wv (transcode from existing WavPack file, with tags)

 Options:
          -h  = high mode (better compression ratio, but slower)
          -f  = fast mode (fast, but some compromise in compression ratio)
          -m  = compute & store MD5 signature of raw audio data
          --pause = pause before exiting (if console window disappears)
          -v  = verify output file integrity after write (no pipes)
          -x  = extra encode processing (no decoding speed penalty)
          --help = complete help

 Web:     Visit www.wavpack.com for latest version and info
Title: Re: -h as the new default mode
Post by: shadowking on 2023-08-13 11:34:52
 -bn                        enable hybrid compression
                              n = 2.0 to 23.9 bits/sample, or
                              n = 24-9600 kbits/second (kbps) (Implies -c)

 -c                           hybrid lossless mode (used with -b to create
                              correction file (.wvc) in hybrid mode)


 --no-c                    Force hybrid lossy mode (use with -b to disable creation of
                              correction file (.wvc) in hybrid mode)

-------

Alternative:

-c<n> = shortcut for '-b<n> -c' to specify hybrid lossless mode
                  with a single option, n = 2.0 to 23.9 bits/sample, or
                                        n = 24-9600 kbits/second (kbps)
Title: Re: -h as the new default mode
Post by: Kraeved on 2024-02-18 04:16:06
Indeed, WavPack is so flexible that it becomes hard to use.

Whenever this codec pops up on the radar, I have to re-read the manual and look through forum threads to wrap my head around the optimal settings. And, to be honest, every time I feel like I've missed something. Because there are too many xxx and hhh without an intuitive relationship between the encoding/decoding speed and the size. Also because there are poorly documented options in terms of their benefit such as -b (from 24 to, err, 9600?), -g (normal mode?), --cross-decorr (good/bad?), --merge-blocks (do/dont?), -m (why not by default, why md5 instead of xxh3-64 (https://archive.is/5yaFl) or blake3 (https://peergos.org/posts/blake3)), -n (is that for my eyes or used internally as well?), -s (what is the default and what's the impact?), --use-dns (huh?), etc.

Aggrrhh!

For comparison: flac works optimally without settings, and as for lossy codecs I need to remember one flag only: qaac -V109 or even no setting at all because default -V91 is good enough, oggenc -q8, lame -V0, mpcenc --insane, etc.

For example, now I'd like to compress a few albums in hybrid mode, keeping transparent-sounding lossies on HDD and uploading correction files to the cloud. Believe it or not, I saw a suitable command not in the manual, but in the signature of @shadowking, namely wavpack -b320hhcs.5. However, I'm still lost, because he changed that command multiple times in favor of increasing the bitrate and adding xxx, let alone -s remains a mystery.
Title: Re: -h as the new default mode
Post by: Porcus on 2024-02-18 10:02:24
Indeed, WavPack is so flexible that it becomes hard to use.
I'm guilty of many misconceptions about the settings and impacts myself.
But as long as you are not interested in the hybrid, then you can rename the executable to get settings, and drag and drop. Like this: https://hydrogenaud.io/index.php/topic,122626.msg1012159.html#msg1012159


Whenever this codec pops up on the radar, I have to re-read the manual and look through forum threads to wrap my head around the optimal settings. And, to be honest, every time I feel like I've missed something. Because there are too many xxx and hhh without an intuitive relationship between the encoding/decoding speed and the size.
Quick to remember:
Impact on both encoding and decoding speed: alphabetical, -f/-g/-h/-hh
x-tra processing during encoding does not affect decoding speed (well except if it finds something smart). That's the -x

I'm guilty of proposing the "-g", which made it easier to run FOR loops for testing ...

As goes -m:
* Why MD5? If you want a hash of the uncompressed PCM, there is no use in implementing anything but MD5, since that is so well established. FLAC has it by default, so it can be used to identify files - even across codecs (as long as sources are little-endian signed ...). Yes with today's knowledge we can tell it was a bad choice, but we are stuck with it. WavPack calls it a fingerprint, rather than a verification tool.
* Why off by default? The historical explanation is likely that WavPack is older ... The reason to keep it off as default, is likely how WavPack is used in audio editor plug-ins, and so -hxm would slow down every save. Besides, version 5 of the file format introduces a (much quicker) checksums for encoded blocks, so you don't need it to verify.

I think we would have to look back at the history here. FWhen FLAC was released, one was very cautious about ensuring that the losslessness was verifiable - also to the extent that you could ensure that some random decoder would output the right thing. Hence MD5 on by default, even if it would slow down things. FLAC was the only open alternative (well depending on how you viewed the Shorten license), and so the only codec that would reasonably expect to get 3rd party decoders.
For trust in the losslessness, you got a hash of the unencoded PCM stored into the format, to be compared to a hash of the decoded PCM. Of course you would then take a checksum algorithm that was implemented everywhere. True, cryptographical vulnerabilities were already known, but still. Also, the early FLAC encoding was slow, so the slowdown from computing MD5 didn't raise eyebrows. A decoder in a low-powered hardware player could always refrain from computing it.

Indeed, at the time, you would see filename.shn files come with filename.md5. Like described here: https://userpages.umbc.edu/~hamilton/shnfaq.html . 

Now within a few years after FLAC was released, the status of MD5 was:
* FLAC had it on by default
* WavPack, OptimFROG and TAK offered it optionally - and no other checksum
* Monkey's being Monkey's would instead hash the encode, and produce a different checksum - but would use the MD5 algorithm too
* And then you got some un-checksummed formats that should die out if they haven't already. WMAL, ALAC - and TTA, which claims to have a checksum but which decodes to static without checking (https://hydrogenaud.io/index.php/topic,122094).

Bottom line is, yes you can argue against MD5 - not that it matters much with today's computing power - but you need to understand the history here. There was a need to assure the general public that yes this works. Still it took long to convince audiophools about losslessness - like, when speaker manufacturer B&W had a music store, they would offer AIFF because ... well just because.


For comparison: flac works optimally without settings
The default is well tuned indeed. But the reference flac encoder gives you direct access to a whole lot of settings that could confuse you if you care. Like, flac --lax -8pb8192 -r7 -l14 -A "subdivide_tukey(4);flattop;gauss(3e-2)" , there you go.
Title: Re: -h as the new default mode
Post by: shadowking on 2024-02-18 10:47:06
For lossless things are more obvious , For lossy HQ IMO what would help are preset mappings for the
most common rates as --hybrid  (default =1 etc)

--hybrid 1 = -b256hx3c
--hybrid 2 = -b320hx4c

Notice my suggestions are similar and also different to the manual. I have gone for a more defensive route while taking into account playback
hardware of the last 10 years.
Title: Re: -h as the new default mode
Post by: Kraeved on 2024-02-19 12:57:06
@Porcus, thank you for the historical background, so well-written that it should be moved to the official wiki.

@shadowking, I don't think that -h is attractive enough to become a default. FLAC-5 wins in both encoding and decoding speed by 3-4 times, taking up only ~100-200 kilobytes more than WavPack -h, at least on my end.

Code: [Select]
$ (for %# in (*.*) do @echo %~z# = %#) | sort
14408062 = vangelis-b4xh.wv
14410736 = vangelis-b4x4h.wv
14462110 = vangelis-b4x4.wv
14489676 = vangelis-b4x4f.wv
14494230 = vangelis-b4x.wv
14548740 = vangelis-b4xf.wv
15940004 = vangelis-b384x3h.wv
15945376 = vangelis-b384xf.wv
15967604 = vangelis-b384x4f.wv
15968562 = vangelis-b384x3f.wv
15990734 = vangelis-b384x4.wv
15996612 = vangelis-b384x3.wv
16004762 = vangelis-b384x.wv
32090104 = vangelis-xh.wv
32227531 = vangelis.flac
32408458 = vangelis-x.wv
32777412 = vangelis-xf.wv
56064668 = vangelis.wav

Since I do not understand where the number 384 came from, but I guess that 256 and 320 will not suit me due to the official remark (https://www.wavpack.com/wavpack_doc.html#:~:text=roughly%20equivalent%20to-,LAME%20MP3,-encoding%20somewhere%20between) that lossy codecs do better there, I have settled so far on -b4x. How do I intend to remember this setting? Mnemonics: -x = eXcellent (lossless), -b4x = bee-four-ex/before excellent (lossy).
Title: Re: -h as the new default mode
Post by: shadowking on 2024-02-19 14:13:10
I would use -x4 over -x .My rational is that if i wanted a speed compromise I'd rather do it  at 256 /  b2.9.
When using 352 / b4 I favour a more defensive approach. 

The remark you mention is that WV won't compete with traditional (128-192) rates . It won't operate less than 196k in 44khz/16 and quality won't be good . I found that the lowest rate 196 / -b2 works well for spoken word cd's etc.  For others I use 256 or more.
Title: Re: -h as the new default mode
Post by: shadowking on 2024-02-19 14:35:12
In my limited testing WV -h often outperforms Flac -8 for cd-audio. Sometimes WV -x4 too.
The decoding well IMO not an issue for typical & even heavier use unless maybe 24/7 transcoding or benchmarking .
And why flac -5,  My impression was that -0 or -1 decoded faster .  I would use -0 or -1 theres plenty of HDD for small to medium collections.
Then I tried WAV+cue and its even faster and theres HDD space for that too.
Title: Re: -h as the new default mode
Post by: shadowking on 2024-02-20 10:50:15
Here:  https://hydrogenaud.io/index.php/topic,125248.0.html

Exactly as I remember that WV normal is superb often beating flac. Would be intereseting to
enable -x1 by default since some tests for non-cd show huge impact.
Title: Re: -h as the new default mode
Post by: Porcus on 2024-02-20 14:06:09
-x is good on higher sampling rates, at least on the corpus I used in Reply 7 of this thread (https://hydrogenaud.io/index.php/topic,120454.msg1004848.html#msg1004848), that it should be recommended.
But I do see the point of not changing anything. It increases encoding time by like 50 percent, and for those who use WavPack plugins in audio processing, that is like "time to save a file", and you don't want to wait that much longer than uncompressed?

Even if David himself uses -hx IIRC. WavPack evidently has a release philosphy that if it ain't broken, don't break disturb it. (When was the last time anyone saw a WavPack 3 file?)
So that makes a case for keeping the defaults and rather tell users to rename the executable into one that applies the options.
For those who "need" drag and drop, it is a good thing to have that done once, without deal with command lines.


Now for how WavPack compares to FLAC, then reference flac.exe has had a couple of improvements - like calculating the predictor in double precision. But for what they are worth, look at ktf's compression studies: http://audiograaf.nl/downloads.html

* High resolution:
The most recent one has flac 1.4, and calculating the predictor with double precision makes a big impact on high resolution. Although WavPack does quite well on the overall graphs, the reason it beats flac so much is one generated synthetic file, the Pokemon, and that is likely to WavPack's treatment of wasted bits that captures "signals that shouldn't happen": https://hydrogenaud.io/index.php/topic,121770.msg1024715.html#msg1024715
In ktf's corpus, there is one such file. In a representative hi-rez corpus ... who knows?!
And so a competition between [TAK -p1 or FLAC] and WavPack -hx4 on 24-bit audio, would likely boil down to how many such weirdnesses there are around.

* CDDA:
Seems that WavPack -hx outcompresses flac -8 still.
ktf's revisions 5 and 6 test -hx4 (that hadn't been tested earlier), but not any -x1. But -h performs around flac.exe -6 to -7 on CDDA:
AMD CPU with slightly older flac.exe: http://audiograaf.nl/losslesstest/revision%205/Average%20of%20all%20CDDA%20sources.pdf
Intel CPU, with slightly newer flac.exe: http://audiograaf.nl/losslesstest/revision%206/Average%20of%20CDDA%20sources.pdf
In his revision 4, -hx over -h would make more impact than flac -8 over "-6 to -7". The WavPack curves are -f, default, -h, -hx, -hh, -hhx: http://audiograaf.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%204.pdf
Now why does FLAC improve so much from revision 4, relative to WavPack? Part of it is a much broader test corpus. It seems to benefit TAK when compared to Monkey's, and FLAC when compared to WavPack. That seems to be a bigger impact than the improvement of FLAC itself.  (I should add, there is absolutely no reason to accuse ktf of cherry-picking corpus to benefit FLAC - indeed, revision 5 was when we knew FLAC could be improved further, and yet it was run with the old one.  And the double-precision improvement on FLAC was decided upon before knowing the performance on that particular corpus too.)


Edit: Oh, CDDA sizes on my usual 38 CDs - a not-so-broad corpus compared to ktf's.
Here, WavPack -hx beats flac (1.4.3) by 0.375 percent (not percentage points) even when beefing up flac with -8pr7.   But the "-p" can make flac sneak past on classical music.
All 38, smallest first:
11913862544   wv -mhx
11958733454   flac -8pr7 --no-padding
11967171287   flac -8 --no-padding
11972043112   wv -mh
12032408675   flac --no-padding (that's default -5)
12039392684   wv -mx
12119277966   wv -m

The 14 CDs of classical music, also in size order:
 3982936563   flac -8pr7 --no-padding
 3983762402   wv -mhx
 3985858502   flac -8 --no-padding
 4013068344   wv -mh
 4021994386   wv -mx
 4057553192   wv -m

More edits:
* The WavPack 5.6.6 beta posted in the forums was used, and I see that there's -m there too - makes 56 bytes extra per CD I think?
* flac 1.4.3 32-bit from Xiph. Note, -r7 is unlikely to make much impact on classical music. Also note, keeping .flac with no padding isn't that smart ...
Title: Re: -h as the new default mode
Post by: Kraeved on 2024-02-24 19:10:29
When I compare wavpack -xh with any flac compression flag, including -8, the difference in decoding speed catches my eye: 85x vs 300-400x. How should this difference be understood? Little or no attention is paid to this detail in official manuals, although this probably has to do, at least, with resources consumption and seek speed, so in some cases this may outweigh the argument about the improving compression by one or two hundred kilobytes.
Title: Re: -h as the new default mode
Post by: Porcus on 2024-02-24 22:06:49
Decoding speed yeah. Does to a large extent measure CPU load, but note that compiler optimizations / instruction sets matter, and might sometimes just mean that parts of the process will go faster and hotter.

FLAC was constructed for ultra-light decoding. Look here: https://hydrogenaud.io/index.php/topic,82125.0.html for numbers on Rockbox devices, which would - to save battery - clock down and not go faster than necessary. FLAC is so "simple" at decoding that you can decode some samples by hand with pen and paper, and indeed if you look at the IETF draft, the appendix carries out decoding of minimal .flac files in detail. Latest at https://github.com/ietf-wg-cellar/flac-specification/releases/tag/draft-ietf-cellar-flac-14

A bit of history with all reservations for accuracy; bear in mind that WavPack the algorithm is older than both FLAC and Monkey's. (Indeed it inspired the creation of the latter.)
Monkey's and TTA are "symmetric", and so was WavPack (I think?) and OptimFROG in old days: you would decode using the encoding algorithm in reverse, taking the same time. Heavier compression? You had to fire more effort at it, in both directions.
I guess we were many who had the gut feeling that if hard drive space was the limiting factor, then you would need that. So Monkey's users could save $$s on hard drives and that codec had a heyday some twenty years ago. But when Monkey's introduced the "Insane" level, there were portable players that could play in principle play it, but not in real time. Meanwhile, FLAC would play at longer battery life than MP3.
Then came TAK. Asymmetric, somewhat heavier on CPU than FLAC, but compressed like Monkey's High (nearly by the tests performed then). TAK changed our perception of what was possible with an asymmetric codec.

Josh Coalson, the creator of FLAC, had a comparison of the codecs available then. This is the oldest web.archive.org version that has decoding times - when FROG had just become OptimFROG: http://web.archive.org/web/20021017022735/http://flac.sourceforge.net:80/comparison.html
It should be noted that WavPack's fast mode was faster at version 3.97, both in encoding and decoding, than early FLAC. You can see that -5 didn't encode particularly fast at the time, but FLAC was to improve. Note, FLAC and Shorten (and in part some RKAU mode) did exhibit this big asymmetry between encoding time and decoding time.
Fast forward to 2007: http://web.archive.org/web/20070823070902/http://flac.sourceforge.net/comparison.html

Now, what has happened since 2002? Except a few new ones?
* FLAC has become more efficient. Most recently also in the fastest encodings. But FLAC works as follows: try many possibilities and pick the best. None of them are heavy at decoding. And by trial and error one has found ways where FLAC can try fewer without having to do so much brute-forcing. You can put the reference encoder at work trying out > 1000 different parameter combinations (per frame) if you seriously want to see how low bang for the buck you can get.
* Among the "symmetric" codecs, both WavPack (the -x) and OptimFROG has gotten "extra encoding processing" which does some of the same: spend more effort on encoding without hurting decoding. But Monkey's has frozen its bitstream: given the signal and the mode (say "high"), the encoded bitstream is determined. Encode a CD with the newest Monkey's - it will be bit-identical to a 3.99 encode. TTA also has a bitstream determined uniquely by the audio - the only thing that could improve would be code optimization.
* We know more about compilers; build matters! https://hydrogenaud.io/index.php/topic,123025.msg1029768.html#msg1029768 .
* And CPU matters. When ktf recently switched his comparison studies from AMD to Intel, WavPack performance numbers suffered compared to FLAC. But meanwhile there was a test on Celeron, where all for sudden the fastest TAK mode would decode faster than FLAC.
* High resolution audio has started messing up what we "knew" from typical CD audio. Especially since a lot of "high resolution" has very little content in the top octaves or the lowest bits.
Title: Re: -h as the new default mode
Post by: shadowking on 2024-02-25 10:15:20
flac -8 doesn't give me anything over -7 but is way slower .. Anyone else find that ?
Title: Re: -h as the new default mode
Post by: Porcus on 2024-02-25 12:02:19
Like, the difference between the rightmost two blue triangles in the upper diagram here? Quite a flat end of the graph.
http://audiograaf.nl/losslesstest/revision%206/Average%20of%20CDDA%20sources.pdf

The reason that -7 compresses better than -6 is that it allows for longer prediction. -7 and -8 are the same on that parameter, and that's why they decompress equally fast; see the virtually overlapping triangles in the lower chart.