Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: "Tested": codecs for the effect of stereo decorrelation (mid/side) (Read 14497 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #25 –
Quote from: bryant on
Thanks again Porcus for your meticulous testing! Like you say, not the most useful files for choosing a compressor, but interesting to get down to the nuts and bolts!
Full consent!

Quote from: bryant on
One of the most common types I found was where the PCM values had obviously been multiplied by some value greater than 1 and just truncated (not dithered) resulting in regularly spaced missing values. I was able to squeeze these out (losslessly, of course) and get significantly better compression, in some cases then blowing WavPack past all the competitors.
I tried  the same for Tak when i noticed that Optimfrog had implemented such a  feature. Optimfrog also seems to take advantage of non uniform quantization (12 bit DAT e.g.). My test corpus was quite small but i too got the impression that most affected files were comparatively old recordings. I am not even sure if there was at least one example in the music i bought in the last 15 Years.

Quote from: bryant on
In the end I decided that this was not worthwhile. For one thing I feared that perhaps my rather old CD collection was not representative and modern mastering tools would not create audio like this any more. And of course this was going to be slow and create enormously complex code. It was an interesting exercise and it would have been fun to release a WavPack that significantly bettered existing encoders on specific files, but in the end thought better of it.
Same conclusion here. And i have to admit i wasn't smart enough to find a detection algorithm that would have been fast enough by Tak's standards. Furthermore Optimfrog's savings still were twice as large for not few of my test files.  To do better i would have to modify far more of the (quite well-tuned) existing codec than i wanted to.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #26 –
Quote from: TBeck on
I tried  the same for Tak when i noticed that Optimfrog had implemented such a  feature. Optimfrog also seems to take advantage of non uniform quantization (12 bit DAT e.g.).

You guys and Florin - who has been fiddling around with an asymmetric frog extension too! - should stick your heads together to see what you can unify.
Take one letter from each of WavPack, TAK and OptimFROG, in that order, and hope that it doesn't make World of Warcraft crash. If that is a problem, bring on board the ALS for WT A(ctual) F.  :D

(The serious part of this is that it is unrealistic to extend FLAC, as it is probably found in too many non-upgradeable embedded devices - but it could bring optimizations to WavPack, Matroska to TAK and playback support to the frog.)

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #27 –
Oh, and you must leave it to @TBeck to announce it. Just make sure he doesn't miss the date by less than an hour.


... and to new HA readers, here is a classic in the history of lossless audio: 
Once upon a time, on April 1st, someone posted about a codec compressing to sizes smaller than Monkey's High and at speeds like this.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #28 –
Would it be possible for any lossless codec to get anywhere close to RAR with stuff like this? It's ZX Spectrum beeper PWM.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #29 –
I suspected that these files have repeating patterns, and yes: when I cut one of them - the biggest, the DarkFusion - into one second segments and compressed each separately with xz -9 -e, file sizes increased by forty percent.

Here is the point: If you process 162 files with a general purpose processor that looks for inter-file patterns, it will detect that file 3 and file 151 are identical, and virtually compress away one of them.
An audio format meant to be streamable doesn't do that. After two and a half minutes you get an instruction that says "repeat segment seconds 3-4". Yeah sure I haven't stored that.
An audio format not meant to be streamable ... go ahead if you want to, and you will be able to outfox some other codecs on some samples yes, but ... is it worth it?

(WavPack beats all the audio codecs on that track though - but you have to use reference WavPack, not ffmpeg.)

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #30 –
Really have you proof for your "claims"? FFmpeg wavpack encoder is marginally different from wavpack reference ones.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #31 –
Quote from: mycroft on
Really have you proof for your "claims"?
Files appear too big to attach, so please go to https://www.sendspace.com/filegroup/JEjBiar2HYKPs2fq9zFAhQ to find encodes of the DarkFusion.wav audio:
* a 7.47 MB file produced using reference WavPack.
* the 8.74 MB file produced by recompressing the former by ffmpeg 5.0 using the awfully slow -compression_level 8.

Now open the WavPack file and see what mode it is. Hint: it is not "high". In case you will bother to even look at it - last time I took the effort to upload to you any evidence in form of a file, you just ignored it and never came back to the topic at all.


Quote from: mycroft on
FFmpeg wavpack encoder is marginally different from wavpack reference ones.
ffmpeg WavPack produces version 4 files.
Above I used the newest reference WavPack.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #32 –
Yes. I just tried to download those files, and sendspace is on scheduled maintenance. Why I ever bothered to visit this thread to get proofs for "bold" claims.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #33 –
Are you calling fake just because Sendspace temporarily is on maintenance? Do we need to teach you how to download WavPack and ffmpeg and try for yourself?

In "honor" of this bullshit of yours, I re-ran it - this time through WavPack 4.80 with and without --optimize-mono, which explains the whole thing:

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #34 –
These files are technically stereo but the content is mono - both channels should be the same.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #35 –
Again I get attacked by brave user of this forums, next time use -optimize_mono switch if you will ever be bothered again.

Hope your smart team is doing well.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #36 –
Why I ever bothered to write decoder for TAK? Users are typically ungrateful as always. Wanted to write RKA decoder too, but looking how are users very ungrateful I will pick something else to do and will never ever be bothered with evil users again.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #37 –
Quote from: mycroft on
Hope your smart team is doing well.
Mine? "team"? I'm just my arrogant swiney self, not affiliated with any codec developer(s). Bark up a different tree.

My issue with you here at the forum is not what code you make elsewhere (if you implemented the TAK decoder, then I am grateful even if I don't use TAK - and it was a better choice than RKAU yes). But your behaviour here isn't ... constructive.

Like, going full denial just because sendspace has a temporary downtime (that still lets you see the files!)?

And had you a solution to the TTA error handling crap or had you not? Hard to tell, and with that attitude, people will just shrug it off as useless.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #38 –
@bryant ... this about wasted bits:
Quote from: bryant on
Interesting fact: WavPack’s version also handles cases where the LSBs are 1s instead of 0s and the case where they’re either 1s or 0s, but still all identical for each sample. I have no idea how often those cases come up, or if other compressors bother, but it was so trivial to add that after the 0’s case that I could not resist.

I wonder whether that is what is going on in this track? https://relapsesampler.bandcamp.com/track/elysium-of-dripping-death
You can avoid the $1 price tag by downloading the entire compilation: https://relapsesampler.bandcamp.com/album/relapse-sampler-2015

But if so, it seems that --pre-quantize zeroes out the bits? Which more or less kills WavPack's advantage. Filesizes follow - "pre-quantized" FLAC files are re-encodes of the same WavPack-generated signals.

399515646   WAVE
270847726   ALAC (ffmpeg)
264226120   ALAC (refalac)
263107430   Monkey's FAST
254236383   TTA
250527682   Monkey's NORMAL
250377278   Monkey's HIGH
249947571   ALS default
248810066   Monkey's INSANE
247930508   TAK -p0
246826374   Monkey's EXTRA HIGH
241284186   OptimFROG –-preset 0 and -–preset 1
234564608   OptimFROG --preset 2 (default)
234100276   FLAC ... -8pe I think
233799103   OptimFROG –-preset 3 beats 0125476
233695667   FLAC -8pel32 -r8 -A subdivide_tukey(4) (That.Was.Slow.)
233066234   OptimFROG –-preset max
232621364   TAK -p4m
221559078   23-bit pre-quantized WavPack at -hhx4
219641625   23-bit pre-quantized FLAC at -8p
197709053   ALS -7 -p
190900370   21-bit pre-quantized WavPack at -hhx4
189654475   21-bit pre-quantized FLAC at -8p
153522746   WavPack -f.  On the original file yes. NOTE: to the byte same filesize as:
153522746   17-bit pre-quantized WavPack at -f
141018808   WavPack -g (default): to the byte same filesize as:
141018808   17-bit pre-quantized WavPack at -g (default)
131267306   WavPack -hx6
129890175   17-bit pre-quantized FLAC at -8p
129650300   17-bit pre-quantized WavPack at -hhx4
129650300   WavPack -hhx4.  Same filesize as 17-bit!
129464830   WavPack -hhx6

Shouldn't exist ... and so WavPack slays everything else by 1/3 (... ALAC by 1/2 ...). Quite a showcase.
That is, until you try to squeeze more out of it in a lossy manner. Now you "have to" make --pre-quantize smarter, don't you? ;D If you allow it to zero out the LSB, you should allow it to add one too ... ahem, unless the sample is full scale all ones. Oops.

Edit: I wonder, is there any "known software" that does any polarity inversion like sending every sample S --> -(S+1)?

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #39 –
Hey Porcus,

So what's your conclusive finding after running these multitude of tests on lossless compressors?

As the dev said, FLAC nails it for 16-bit CDDA audio, but isn't so helpful with 24 bit high res stuff. Which lossless compressor can give comparable compression levels like FLAC on 16 bit, but for 24 bit 96khz?

Users with MacBook and other computers with soldered storage, would share my pain of keeping the 256 gb drive healthy ;-;

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #40 –
For a computer with 256 GB, the solution is of course an external drive ;-)  Besides you want backup anyway. You aren't keeping the only copy of your music collection on a portable unit ... I hope?

Concerning compression performance, both CPU power and storage have gotten cheaper - but if you can barely fit your entire collection on your 256 GB then sure, go ahead find a format that compresses better. But then on a MacBook? The most impressive overall performing codec is arguably TAK, and that is a closed-source Windows encoder ... although ffmpeg can play it. Compatibility is an issue too - still!
What do you want to accomplish, really?

Now, I am not testing "overall" codec performance - for that, look to ktf's tests: https://hydrogenaud.io/index.php/topic,122508.msg1024541 for CD audio, and more at audiograaf.nl/losslesstest/
What I have been doing is to curiously try to put codecs (/encoders/decoders) through some peculiarities to explain why or whether some do this better than others. Some of this is performance related, but remember that for performance you want the grand total, not the odd counterexamples that don't matter much for more than a very few files. Here was a special kind of signal that WavPack can detect and compress easily, but if they were many enough to be a big part of the average, they would show up in the totals in ktf's tests.
Although a wide corpus like his doesn't get the picture for your collection if you have more of one genre than of others ... and most of us do.

Myself I am using FLAC. And then WavPack for some more odd signals. Those FLAC cannot contain, but also (long story) Isome that FLAC can actually handle.
But codec choice won't solve the "big problem" about 24-bit signals: the last few bits are typically noise. Not good for anything but taking up space, and pretty much incompressible.
WavPack, FLAC and TAK (and OptimFROG) handle well one particular aspect of certain 24-bit files: those which aren't really "24 bits", but where the last few bits are padded with zeroes. (WavPack also handles the very few cases where those last few bits are all 1 rather than all 0.) And then the files with higher sample rates are quite diverse - is it noise up there in the audible range, is it ... nothing? It is possible to squeeze out a bit more of FLAC using heavier settings than -8, but don't expect miracles (except on very few samples, some impressive but no miracles in the big picture). Again, see ktf's tests. You will see that for WavPack on high resolution signals, the -x4 switch (or in high or very high mode, -hx4 or -hhx4) can squeeze out quite a bit. (There are even heavier switches, -x5 or -x6, but they don't offer that much bang for the buck, they are very slow.).

I don't use appleware, so I don't know what is easier to handle there. Of course Apple wants you to use ALAC, which isn't a good codec.


Blah blah unsorted ramblings, sorry ... but what are your needs? Most lossless needs are perfectly well covered by FLAC. If you want to go for WavPack ... make sure your player can handle it.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #41 –
Quote from: darkalex on
As the dev said, FLAC nails it for 16-bit CDDA audio, but isn't so helpful with 24 bit high res stuff.
Huh? Which dev said that where?
Music: sounds arranged such that they construct feelings.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #42 –
Quote from: ktf on
Quote from: darkalex on
As the dev said, FLAC nails it for 16-bit CDDA audio, but isn't so helpful with 24 bit high res stuff.
Huh? Which dev said that where?

This dev said that right here:
Quote from: ktf on
This would also explain why FLACs benefit is only present for 16-bit (CDDA) material and not for 24-bit signals.

.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #43 –
You're taking these results (and my quote) completely out of context. What is tested in this topic is one specific aspect of lossless audio compression, trying to find intricacies. The title says it: 'the effect of stereo decorrelation'. This is in no way meant as a thorough comparison.

FLAC up until version 1.3.4 did reasonable with 24-bit material, but was not up to par with the other codecs. FLAC 1.4.0 and later do very well with 24-bit material. I am confusing things. FLAC up until version 1.3.4 did reasonable with high samplerate material with no actual content above 20kHz, but was not up to par with the other codecs. FLAC 1.4.0 and later do very well with this. FLAC (any version after 1.2.1) does fine with 24-bit material in general.
Music: sounds arranged such that they construct feelings.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #44 –
I think that the FLAC "irls-2021-09-21" build I used in this test, incorporates the change that made 1.4 improve on higher sampling rates, namely doing some number-crunching with higher precision. Also I used a stock 1.3 build that does not.
The high resolution files used in this particular test are not only 24 bits, but also higher sampling rates.

And just to confirm ktf's explanation of not only what I did, but also the purpose and limitations thereof: this is not an assessment of overall performance. The purpose is to isolate the impact of specific features (in the codecs/encoders or in the signals), and the relevance for "overall performance" are "unknown". Potential relevance:
* Understanding why. Geeky curiosity.
* Improvements. However, that would often be constrained by format specifications. For example, it seems that only WavPack can handle "wasted bits all being 1", but WavPack cannot do say Left+Side stereo decorrelation.
* ... which means that well sometimes one just has to accept that we are twenty years too late to implement this improvement, and likely one will just have to let <this competitor> have this advantage.
* Understanding that performance over a broad test corpus - which is what developers may aim at - doesn't necessarily match performance in more special signals. Possibly that might save devs from "but my collection doesn't measure that way!" questions - that was for the "obvious" thing that different music compress different - but for one step up in understanding, make that "performance differences" twice in the first sentence.
.
.
.
* And even if the outcome is a boring null result: write it up and post it when I have done the test job, because "nothing to see here, move on" also is information.
Besides, a clever question might spawn ... oh there might be more than completely nothing going on. https://hydrogenaud.io/index.php/topic,122056.msg1007356.html#msg1007356

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #45 –
Quote from: Porcus on
@bryant ... this about wasted bits:
Quote from: bryant on
Interesting fact: WavPack’s version also handles cases where the LSBs are 1s instead of 0s and the case where they’re either 1s or 0s, but still all identical for each sample. I have no idea how often those cases come up, or if other compressors bother, but it was so trivial to add that after the 0’s case that I could not resist.

I wonder whether that is what is going on in this track? https://relapsesampler.bandcamp.com/track/elysium-of-dripping-death
You can avoid the $1 price tag by downloading the entire compilation: https://relapsesampler.bandcamp.com/album/relapse-sampler-2015

But if so, it seems that --pre-quantize zeroes out the bits? Which more or less kills WavPack's advantage. Filesizes follow - "pre-quantized" FLAC files are re-encodes of the same WavPack-generated signals.
I downloaded both the sampler and the individual track and only got 16/44.1 versions. Not sure if that's because I don't "belong" or they changed them in the meantime.

But regardless, your theory as to why WavPack would slay this file seems like the only possibility. It might be that '1's are used for padding intentionally to make it harder to detect that the files were just upsampled? Or there was a logical complement in the chain somewhere (which is a valid operation on signed audio values, unlike negation which, as you point out, can overflow). In any event, it's not a good look.

And it makes me happy I included that check. It seems counter-intuitive that a compressor would do well on a set of data, but do poorly on its complement.  :D

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #46 –
Ooops. Will PM you the file in question.

That file was part of a different compilation announced for the label's 30th anniversary right before Xmas 2020.
It is now deleted from Bandcamp, which explains why I got a different version when I google'd up the track to get the link. Instead I got a previously published version.

Which makes me wonder what kind of creative "remastering" some wise person therein did for this compilation (and whether that was the reason it is gone?). But of course that is not necessarily what happened ... I did some detective work on the other downloads - and on CD samplers from the label. Among those 241 tracks
* 34 have same MD5s as earlier compilations. Fine.
* 7 are "remastered" to 48kHz/16, 7 to 48kHz/24 and 7 to 44.1kHz/24. I am not sure if that means remastering, or maybe that they picked a file that had not yet been downmized to CDDA later.
* 24 are "remastered" to 88.2kHz/24 or 96/24. Or again, it could be that they are retrieved from the DAW and that the previously released ones were finalized.
Among those 24, three have this feature that WavPack beats FLAC by say forty percent. A few more are in the -10 percent league, I guess things like that just happens when using x4 - especially when one of them is 20 percent smaller than the -hx encode.
* And a handful I also have on early 2000s CD promo compilations, they are all bit-different here although also in 44.1/16.

Anyway, nearly all the tracks can be listened to in a slightly different Spotify playlist.

Re: "Tested": codecs for the effect of stereo decorrelation (mid/side)

Reply #47 –
Thanks for the files!

So, for the ones that WavPack slays everything without even trying, those indeed have the redundant LSBs. I checked only one in detail, but it was converted from 16-bit with the new byte either all ones or all zeros (0xff) in bursts (not randomly sample-to-sample). So even checking for all ones in the LSB wouldn't catch this, but my code checks for some number of LSBs being identical, and in this file it's the lower 7 bits (0-6) always match the next bit (7), so my code essentially truncates 7 bits. Have no idea how that could happen except intentionally, but for what intention?

The others where the WavPack extra modes improve compression by 10% or so, those are upsampled from 44.1 with varying filters. In one it was a slow rolloff starting at 17 kHz, while in another it was a steep rolloff at 20 kHz. Funny that the upsampled ones might even have less HF content than the 16/44!