Experimental WavPack 5.7.2 (hybrid enhancements)

2024-05-24 17:27:57

A month ago @guruboolez let me know about an issue he had discovered with WavPack’s hybrid mode. When experimenting with very low bitrates and high sampling rates (i.e., 2 bits / sample with 96 kHz or 192 kHz audio) the quantization noise would sometimes jump way up in level and become audible. Using a higher bitrate or forcing dynamic noise shaping (--use-dns) would fix it.

This reminded me of a situation I had seen several months earlier, also with the 2 bps setting. In this case it was 24-bit file with just a 1001 Hz sine tone, and the -x6 setting would generate over 10 dB more quantization noise than -x5.

Lastly, there was the sample pointed out by @Kraeved and @shadowking where WavPack’s dynamic noise shaping was definitely not doing the right thing (which I described at depth in this post).

I started investigating all this and quickly descended into the rabbit hole! The first thing I discovered is there is a feedback issue with the hybrid mode at low bitrates. In other words, the quantization noise makes the prediction worse, which in turn increases the quantization noise, which makes the prediction worse, etc. There’s no real fix for it (at least I haven’t come up with one), but it is possible to mitigate it. For example, positive noise shaping makes it worse, which is why guru saw the problem with high sample rates that force first-order shaping in an attempt to shift the quantization noise into the inaudible range. One of my fixes is to limit this forcing for low bitrates.

To fix the killer sample issue, I came up with a smarter noise-shaping algorithm that actually filters the audio into high and low bands (split at fs / 6) and then compares the two averaged levels to determine the optimum shaping. This is more accurate than the old method, and so I have tuned it to be more aggressive, including stronger negative shaping (which helps with the feedback issue also). It greatly improves the killer sample (to my old ears) and might improve other samples (with any luck).

The new shaping algorithm is very quick to react, and so sometimes generates very short frames in the hybrid lossless mode. These short frames exposed some sub-optimal code and even some outright bugs in the “extra” modes, which is code that hasn’t had any meaningful updates in almost a decade. Fortunately the issues mostly concern these very short special-case blocks, but I have attempted to fix everything I found.
 
Bottom line is that I have created a very experimental version 5.7.2 that incorporates lots of changes to address both guru’s discovery and possibly do better noise-shaping in the hybrid/lossy mode. Here’s a quick, partial summary of the changes:

• fixed bug: use the correct residuals for parameter estimation in extra levels 4 through 6
• fixed bug: very rare edge-case where we would skip writing the correction bitstream
• new dynamic noise shaping (DNS) algorithm using frequency splitting filters with averaging
• only force positive noise shaping for sample rates >= 88,200 Hz and bitrates >= 4.5 bits / sample
• in the “extra” modes only take header overhead into account in pure lossless mode
• allow noise shaping to go further negative with bitrates under 3.0 bits / sample
• improve decorrelation and entropy parameter estimation for very short frames

I would be interested in any feedback on this, but please do not use this (at least yet) for any important archiving. And, as always, be sure to use the -v mode to verify any lossless compression operation. Note that this still uses the same simple noise-shaping curves as before (only the selection is new), so everything is completely compatible with prior decoders.

BTW, the lossless hybrid mode is very slow because the new DNS code is not fully optimized. I don’t think it will ever be as fast as the old code (which was super simple), but it will get better than this. Also, because of the shorter frames, the overall compression ratio has dropped roughly 1% (at least for now).

Experimental branch on Github


 

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #1 – 2024-05-26 19:33:35

Great new! Thanks for constantly trying to improve WavPack.
How could I exactly help? Testing lossy quality at lower bitrate with 44/16 files? On known problem samples? By measuring bitrate?

Re: Experimental WavPack 5.7.2 (hybrid enhancements)

Reply #2 – 2024-05-27 12:56:32

Cool !  I'll run some tests later in next few days.