Experimental WavPack 5.7.2 (hybrid enhancements)

2024-05-24 17:27:57

A month ago @guruboolez let me know about an issue he had discovered with WavPack’s hybrid mode. When experimenting with very low bitrates and high sampling rates (i.e., 2 bits / sample with 96 kHz or 192 kHz audio) the quantization noise would sometimes jump way up in level and become audible. Using a higher bitrate or forcing dynamic noise shaping (--use-dns) would fix it.

This reminded me of a situation I had seen several months earlier, also with the 2 bps setting. In this case it was 24-bit file with just a 1001 Hz sine tone, and the -x6 setting would generate over 10 dB more quantization noise than -x5.

Lastly, there was the sample pointed out by @Kraeved and @shadowking where WavPack’s dynamic noise shaping was definitely not doing the right thing (which I described at depth in this post).

I started investigating all this and quickly descended into the rabbit hole! The first thing I discovered is there is a feedback issue with the hybrid mode at low bitrates. In other words, the quantization noise makes the prediction worse, which in turn increases the quantization noise, which makes the prediction worse, etc. There’s no real fix for it (at least I haven’t come up with one), but it is possible to mitigate it. For example, positive noise shaping makes it worse, which is why guru saw the problem with high sample rates that force first-order shaping in an attempt to shift the quantization noise into the inaudible range. One of my fixes is to limit this forcing for low bitrates.

To fix the killer sample issue, I came up with a smarter noise-shaping algorithm that actually filters the audio into high and low bands (split at fs / 6) and then compares the two averaged levels to determine the optimum shaping. This is more accurate than the old method, and so I have tuned it to be more aggressive, including stronger negative shaping (which helps with the feedback issue also). It greatly improves the killer sample (to my old ears) and might improve other samples (with any luck).

The new shaping algorithm is very quick to react, and so sometimes generates very short frames in the hybrid lossless mode. These short frames exposed some sub-optimal code and even some outright bugs in the “extra” modes, which is code that hasn’t had any meaningful updates in almost a decade. Fortunately the issues mostly concern these very short special-case blocks, but I have attempted to fix everything I found.
 
Bottom line is that I have created a very experimental version 5.7.2 that incorporates lots of changes to address both guru’s discovery and possibly do better noise-shaping in the hybrid/lossy mode. Here’s a quick, partial summary of the changes:

• fixed bug: use the correct residuals for parameter estimation in extra levels 4 through 6
• fixed bug: very rare edge-case where we would skip writing the correction bitstream
• new dynamic noise shaping (DNS) algorithm using frequency splitting filters with averaging
• only force positive noise shaping for sample rates >= 88,200 Hz and bitrates >= 4.5 bits / sample
• in the “extra” modes only take header overhead into account in pure lossless mode
• allow noise shaping to go further negative with bitrates under 3.0 bits / sample
• improve decorrelation and entropy parameter estimation for very short frames

I would be interested in any feedback on this, but please do not use this (at least yet) for any important archiving. And, as always, be sure to use the -v mode to verify any lossless compression operation. Note that this still uses the same simple noise-shaping curves as before (only the selection is new), so everything is completely compatible with prior decoders.

BTW, the lossless hybrid mode is very slow because the new DNS code is not fully optimized. I don’t think it will ever be as fast as the old code (which was super simple), but it will get better than this. Also, because of the shorter frames, the overall compression ratio has dropped roughly 1% (at least for now).

Experimental branch on Github


 

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #1 – 2024-05-26 19:33:35

Great new! Thanks for constantly trying to improve WavPack.
How could I exactly help? Testing lossy quality at lower bitrate with 44/16 files? On known problem samples? By measuring bitrate?

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #2 – 2024-05-27 12:56:32

Cool !  I'll run some tests later in next few days.

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #3 – 2024-06-03 13:48:18

Okay , Very brief testing and so far its looking good.

codectest16 sample - WV 5.72

-b384hx6
wv5.72 abx 8/10 (close)  ,s0 10/10 , s.5 9/10,  s.75 7/10

-b4x4
 much improved not hard to abx but not annoying.  -b4.5hx6 becomes hard.

new-dns >>> old-dns >> s0 > s.5 =s.75

summary
-newdns much better than olddns
-better than s0
-better than s.5,
-on par with s.75
-wv5.72 dns preferred over old-dns and -s.x on this sample.

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #4 – 2024-06-04 10:26:02

Busy weeks for me but I have an hour this morning.
Here's some feedback with the organ samples I sent you some weeks ago.

Few notes for other readers:

• it's a sample coming from an SACD. The DSD64 was converted to PCM using Poikosoft EZ CD Audio Converter in ULTRA mode with an output at 88200 Hz. Stereo track.
• WavPack 5.70 had very strong noise artifacts (I don't know how to call them: noise pumping maybe)
• The issue occurs with fast mode only, with a bit-per-sample between 2 and 2.4 only, and with x3, x4, x5 and x6 additional settings only. There are no known issue with different parameters. And the artifact is also absent with lower sampling rate (noise is nonetheless perceptible at 44100 Hz).
• The issue occuring with WavPack 5.70 could be fixed with --use-dns or maybe other noise shaping settings
• for people not familiar with wavpack: the -b setting is proportional to the sampling rate. -b2 is approximately 200 kbps for a CD encoding and for this 88.2 KHz sample bitrate is expected to be around 400 kbps

WavPack 5.70 -b2fx6 (403 kbps)
I didn't ABX it, it's seems very very obvious (a kind of Fatboy Slim artifact, for very old readers of this board). I joined the encoded file at the bottom.

WavPack 5.72 experimental -b2fx6 (399 kbps) vs REFERENCE
ABX = 7/8: slightly noisy on some organ notes. For my ears it's subtle but I'm not trained to detect hybrid formats artifact.

WavPack 5.70 -b2fx6 --use-dns vs WavPack 5.72 experimental -b2fx6
ABX: I failed to differentiate both encodings.

Conclusion: problem solved.

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #5 – 2024-06-09 23:32:01

Sorry for the delay in responding here; I have been traveling on holiday. I really appreciate you guys performing these tests and reporting your results!

So far this is very encouraging. I want to do some more testing myself, and I need to optimize the new DNS to not do so much redundant work (which is why it is so slow), but I am thinking that this really needs to be pulled into the mainline code and replace the existing DNS.

Thanks again!

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #6 – 2024-06-15 11:47:55

I also found the time to test it against this sample:
https://hydrogenaud.io/index.php/topic,120193.0.html

For this test I was using v5.7.0 @384k -hh vs experimental v5.7.2 @ 384 -hh
No need to ABX. I got both 10/10 but new experimental version was much better.
There was only very small amount of noise at second drum hit.
-b384hx4 didn't solve it. Still 10/10.

Than I tried -b450hh (v5.7.2). It was very hard. Super tiny noise. VERY hard to hear. I went through few cycles to even determine which is which @100% volume). For me, this is solved.
(-b450hh in v5.7.0 was obvious and easy to spot 10/10).


ABX test (original WAV vs v5.7.2 @450hh)

foo_abx 2.0.6c report
foobar2000 v1.6.8
2024-06-15 12:20:21

File A: Jari.Aalto-Electronica.wav
SHA1: 86786351d337f5065444a841bf07f1b2319869c1
File B: Jari.Aalto-Electronica-v5.7.2-450hh.wv
SHA1: 22ebe7310420cb0d039e0a0d35289c22e9566c5e

Output:
Default : Primary Sound Driver
Crossfading: NO

12:20:21 : Test started.
12:21:11 : 01/01
12:21:46 : 01/02
12:22:09 : 02/03
12:22:35 : 03/04
12:22:45 : 04/05
12:22:57 : 05/06
12:23:28 : 06/07
12:23:45 : 07/08
12:24:03 : 08/09
12:24:17 : 09/10
12:24:17 : Test finished.

 ----------
Total: 9/10
p-value: 0.0107 (1.07%)

 -- signature --
4457299b744426a983736511c14262292de683eb
 

5.7.2 hugely improved the quality of this sample.
I hope this helps.
Great work @bryant