Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Need advices for listening tests (Wavpack lossy) (Read 9297 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Need advices for listening tests (Wavpack lossy)

Reply #25 –
The presets look good. 2 for speed vs 2 for quality .  -hx4 is a gem IMO - quality/compression is prioritized while still 'fast' .  -hhx6 is interesting the benefits are more sublte than -gx to -hx4 but certainly there in some cases.

Would be good to see how this new DNS goes.

The high anchor needs to be a 'monster' IMO :)   -b384hhx5,  -b400hhx6, -b4.5hhx5

Re: Need advices for listening tests (Wavpack lossy)

Reply #26 –
Shadowking, I was waiting for your input before diving headfirst into this test. Your opinions are authoritative on all matters concerning Wavpack lossy.

Today, I'm undecided between two setups: tests with 3 presets + anchor, and tests with 4 presets + anchor. The advantage of a 3-preset test is that it reduces the mental load. I believe the time required to test each sample will double with a 4th candidate (since everything has to be listened to multiple times to be sure of the hierarchy). The risk of error is higher with 4 files to test, and the statistical noise will increase consequently.

If I go with the original idea of 4 files, my initial proposition seems quite balanced (-f / -gx / -hx4 / -hhx6). If I drop one candidate, Porcus's proposal seems to rebalance the set quite well (-f / -hx / -hhx6). Indeed, out of the 28 levels offered by Wavpack, -hx is situated right in the middle. So, I would have the two extremes and an intermediate preset. In my opinion, I won't find much margin separating them, especially beyond the 200 kbps mark. What do you think about it?

For the HIGH ANCHOR, I'm giving up on including one. Except for rare cases, it will be perceptually identical to the original source. I'd rather use two MID-LOW ANCHORS (ADPCM & 8 bit audio) instead.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #27 –
I would add LossyWAV processed Wavpack since It TrueVBR?.
OptimFrog PC | TAK -p4m(Phone)

Re: Need advices for listening tests (Wavpack lossy)

Reply #28 –
Quote from: Blue900 on
I would add LossyWAV processed Wavpack since It TrueVBR?.
A lossyWAV vs Wavpack lossy comparison would be highly interesting (I would even add DualStream in the arena). But as said in the first post, this test will be dedicated to Wavpack performances.
Once done, and if I still have some envy, why not :)

But thanks for the suggestion :)
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #29 –
WARNING for babbling about optics without having tried them - and it is a BIG reservation whether CDDA lossless sizes are anything to go by. Of course bits saved can be used to attempt to improve a lossy, but this test is not about lossless.

Anyway, looking up file sizes in the guruspreadsheet, they span about five percent (that's percent, not percentage points) from biggest -f (2.47 percent over default -g) to smallest -hhx6 (2.65 percent smaller than default -g).
The midpoint is between -g and -gx. But -f is the oddball. Nearly a quarter of the range is the single step -f to -fx. That's much more in your collection than in my tests.

Anyway:
Going -hx4 and -hhx6 won't differ that much size-wise. Like around 0.4 percent.
The impact going -g to -gx is about twice that. Also the impact going -h to -hx. -gx to -h is not quite as much as the double.

So - again with these BIG reservations: Just select one among -hx4 and -hhx6. And then a third and no fourth?
It is easy to defend to select -f and -hhx6 and -g (because they are the extremes and the default).
Alternatively, two "more reasonable near-extremes" (-fx and -hx4) and then one "reasonable midway" like your fave among -g -gx -h -hx. It's not only about size but also about speed.

Re: Need advices for listening tests (Wavpack lossy)

Reply #30 –
Again, thanks for your advices, @Porcus !

I tried the empirical road. I took a sample which is not intended to be included in this test. The sample was suggested 6 years ago by @j7n. Jean-Michel Jarre, Equinoxe Part 3. I have a remastered edition, 24/48 I resampled to 16/44. Sample is joined at the bottom of this message (15 seconds).

I decided to check if I can differenciate several settings, and how big is the difference. Are tested here with Wavpack 5.80:
  • -b2f
  • -b2fx6
  • -b2g
  • -b2gx6
  • -b2h
  • -b2hx6
  • -b2hh
  • -b2hhx6

The user j7n (which doesn't seem active anymore) was quite right when he said the music suffers at around 200 kbps. Just keep in mind I deliberately choose a part which clearly sounds noisy with -fast mode. The part begins at 0'55''.


Result (ABC/HR log is uploaded at the bottom):
  • -b2f = 1.7
  • -b2fx6 = 2.2
  • -b2g = 2.4
  • -b2gx6 = 3.5
  • -b2h = 2.6
  • -b2hx6 = 3.5
  • -b2hh = 3.5
  • -b2hhx6 = 3.8

I did my best to rank them but honestly it's much easier to notice a difference between REFERENCE and any ENCODED file than ranking all ENCODED files. At least i didn't make big mistakes here.
It seems, judging by one single sample and at the lowest bitrate possible, that default (g) and high (h) are very close, and that gx6 and hx6 are even closer each others. The gap between -gx6 and -hhx6 is also rather small on this discriminant sample.

Conclusion: I'm even more tempted to keep the fastest and the slowest mode, and put in the middle one intermediary setting. But which one: -g / -gx / -h / -hx?
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #31 –
I did the same test with another sample, which is really ordinary this time (beginning of an orchestral movement).

Differences are now much lower and it was only easy to spot with one encoding (no surprise: it was the -f switch). For the others I need to be more attentive and I kept my attention on a short part.

Result (ABC/HR log is uploaded at the bottom):
  • -b2f = 3.9
  • -b2fx6 = 4.4
  • -b2g = 4.3
  • -b2gx6 = 4.5
  • -b2h = 4.5
  • -b2hx6 = 4.8
  • -b2hh = 4.7
  • -b2hhx6 = 4.9

The fastest switch is definitely the easiest to spot at ~200 kbps. But sound wasn't bad at all. I first ranked it at 4.5 and I progressively lowered it to have more headroom for ranking the others. The slowest preset is once again the best. And there are no big differences with all presets between fx6 and hx6 with this ordinary sample. "It took me a long time to rank them securely, and I had to do it multiple times. I'm even surprised that the ranking is consistent!
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #32 –
Based on the two previous posts:
• sample 1: lowest = 1,7 // highest = 3,8 // Mean= 2,9. Closest to the average score = -b2h = 2,6 (=> b2hx should even be closer)
• sample 2: lowest = 3,9 // highest = 4,9 // Mean= 4,5. Closest to the average score = -b2h = 4,5

Now I'm trying to take encoding speed into consideration. Based on Porcus' lossless speed table:

-h= 1,53x slower than -f
-hx= 2,58x slower than -f
-hx4= 29x slower than -f
-hhx6= 91x slower than -f

I tried on my side with one file (39 minutes 16/44 WAV file on my Laptop powered by AMD Ryzen 7 5800H 3.20 GHz):
-b2f= Total encoding time: 0:07.547, 309.40x realtime
-b2h= Total encoding time: 0:10.563, 221.06x realtime
-b2hx= Total encoding time: 0:25.656, 91.01x realtime
-b2hx4= Total encoding time: 2:24.265, 16.18x realtime
-b2hhx6= Total encoding time: 7:11.125, 5.41x realtime

For comparison, same file:
- MP3 HELIX -V150 HF2: Total encoding time: 0:06.156, 379.31x realtime
- MPC -q6 (extreme): Total encoding time: 0:20.047, 116.47x realtime
- Opus -b200= Total encoding time: 0:21.000, 111.19x realtime
- LAME 3.100 -V0: Total encoding time: 0:23.094, 101.11x realtime
- LossyWAV 1.43g -q-10.0 + FLAC: Total encoding time: 0:56.734, 41.15x realtime
- USAC Exchale 9 (200 kbps): Total encoding time: 2:40.515, 14.54x realtime


-h is less than twice slower than -f but -hx is a bit more three time slower than -f. But -f is insanely fast, and speed is not far from HELIX MP3 high bitrate VBR.
So -h is much faster than all other HQ transform codecs, and -hx speed is similar as LAME MP3, Opus, MPC...

So I'm tempted to keep the three following settings during my test:

-f: fastest mode, lowest quality. The penalty in quality is expected to be strong at lowest bitrate and with highly tonal samples but I suspect -f quality to become quickly transparent on regular music with higher bitrate
-hhx6: Highest theoretical quality but slowest mode. Very huge penalty on encoding speed, but modern processors can handle this unefficient mode. For reference, the recommended LAME setting on Hydrogenaudio.org (3.90.3 --alt-preset standard) was even twice slower (~x3 real time) than this on my desktop AMD Duron back in 2001. But on batch encoding on my laptop CPU (multithread) speed is close to x40 with -hhx6.
-hx1: competitive speed with transform encoders. Quality seems to be somewhere in the middle between -f and -hhx6. The setting benefits from enhancement of both -high mode and of the first level of -x bonus—which makes it attractive compared to -h or default -g.

Does it look good to you?
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #33 –
what about -x4 ?  Similar to -hx1 but its normal mode and can use -x4 .  Its 'fast' but not as much as -hx1

In any case the list looks good.

Re: Need advices for listening tests (Wavpack lossy)

Reply #34 –
Sorting the lines a little bit, this looks jolly good:

-b2f= Total encoding time: 0:07.547, 309.40x realtime
- MP3 HELIX -V150 HF2: Total encoding time: 0:06.156, 379.31x realtime

-b2hx= Total encoding time: 0:25.656, 91.01x realtime
- MPC -q6 (extreme): Total encoding time: 0:20.047, 116.47x realtime
- Opus -b200= Total encoding time: 0:21.000, 111.19x realtime
- LAME 3.100 -V0: Total encoding time: 0:23.094, 101.11x realtime

And then there is kind of a match between:
-b2hx4= Total encoding time: 2:24.265, 16.18x realtime
and
- USAC Exchale 9 (200 kbps): Total encoding time: 2:40.515, 14.54x realtime

... yet I do think it still is reasonable to take "fastest, slowest and middle ground". And this is a case for -hx: Bit rate and computational effort are about the ballpark of common well-developed lossies.


That said, going "-hx" also makes a case for including -g as it is the default, and doesn't use the -x method at all.
Oops. A fourth one.
(-f doesn't use -x, but we wouldn't be surprised if -f fares significantly worse than the others.)

Re: Need advices for listening tests (Wavpack lossy)

Reply #35 –
Quote from: shadowking on
what about -x4 ?  Similar to -hx1 but its normal mode and can use -x4 .  Its 'fast' but not as much as -hx1
Encoding speed test with x4: Total encoding time: 1:37.969, 23.83x realtime. It's now far behind most modern encoders, and quality is probably similar to -x1 on average stuff (maybe slightly better, maybe slightly worse).


Quote from: Porcus on
That said, going "-hx" also makes a case for including -g as it is the default, and doesn't use the -x method at all.
Oops. A fourth one.
You're right. But after thinking about it for many hours, I come to the conclusion that there will be always something missing unless I test all 28 available scales. So I decided to test WV at lowest / middle / highest.
If the test was only about critical samples, I believe I would add a fourth and maybe a fifth candidate: ranking should be much easier with killer samples (based on the experience with the Equinoxe sample). But with many regular samples I don't think I'll find headroom enough for ranking two intermediate settings.

The tested settings will be:
-f
-hx1
-hhx6

(and maybe -g and maybe -hx4/hx6 but only for killer samples).


Many thanks for your help and suggestions :)
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #36 –
I made a last pre-test before the plunge, and I must admit that -f (naked, without -x) is maybe not the best bang for the buck even for regular music. At least at 200 kbps. Considering -g: it's the default setting, it's almost as fast on the encoding side, and I suspect it to reduce the gap that exists between -f and -hx. The gap between -hx and -hhx6 seems much narrower.
Though it's not my preferred approach, considering the implementation of the default setting (-g) as suggested twice by Porcus seems worthwhile. And here's my fourth candidate again :/

With this new setting I have one candidate for each preset (fast/default/high/higher). Two of them being very fast, one being near as fast as modern HQ lossy encoders, and a last one for someone trying to reach the best possible quality and doesn't care about electricity cost.

Speed test (-g) as before: Total encoding time: 0:08.906, 262.19x realtime on a single thread.
With eight threads (8x the same file): Total encoding time: 0:13.031, 1433.54x realtime
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #37 –
See you when the results are ready, now in four hours (rather than three)   :D

Edit:
But seriously - lossy compression needs to do both the compression work and the "lossy" work, and even if the seconds saved on -f may be the same (or not), it is not a situation where the "fast" is so much needed. But ... the differences between -f and everything else could be interesting to see for those of us who don't have to do all the work.

Re: Need advices for listening tests (Wavpack lossy)

Reply #38 –
A few quick notes:

First, I would not place too much weight on what the so called “default” is, considering that it was cast over 20 years ago. There have been discussions about changing it over the years, and if I were starting today I would consider -hx as a reasonable starting point considering the advance in processing power.

If -f is falling down in some cases, maybe -fx would be better for the “fastest” setting? I guess that the extra processing might make a big difference with some problem samples (particularly because it can choose to skip M/S encoding) and it retains the fastest decoding possible (in fact, it might even decode faster in some cases). Maybe this would help you stick to just 3 settings?

Also toward the goal of reducing the sheer number of tests to try, maybe start with just 2 bps, 3 bps, and 4 bps and then only try the intermediate values if more information is needed (they should probably fall cleanly in-between).

I have to say I’m very impressed with how well 200 kbps lossy WavPack is doing, despite my suggestion that it’s not recommended. I’m sure that problem samples will trip it up in the near future, but it’s still encouraging. I wonder if the new DNS is responsible, or whether is was always this good.

Re: Need advices for listening tests (Wavpack lossy)

Reply #39 –
@bryant @Porcus > I started the test before your last answers. Anyway, thank you again for your suggestions :)
What I finally choose for this test is:
200 KBPS (-b2) is my starting point.
- WV 5.80 -b2f  /  -b2  /  -b2hx  /  -b2hhx6
- Anchor: ADPCM-QX 3 bit (no other options than w3). 2 bit ADPCM sounds too different.
- Anchor: 8bit dithered PCM with noise shaping.

I listened to both anchors before launching this test and their performance are erratic. As sometimes one fail to sound bad I expect the other to do the job.

Quote
maybe start with just 2 bps, 3 bps, and 4 bps and then only try the intermediate values if more information is needed
That's a good advice! Thanks David :)
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #40 –
I managed to finish two sets of samples: IgorC and a small classical set. I can share these intermediate results here, and then create a separate topic when results are complete.

IgorC's 192kbps list






Classical Music






Few comments:
  • The classical music group samples do not present any particular difficulties. They are taken from records used for my large bitrate table, and my selection criterion was to choose a passage that I appreciated musically, without looking for possible encoding difficulties. The objective of this choice, made in 2020, was precisely to be able to evaluate future encodings on trivial passages. These samples logically should not allow differentiation of very high bitrate encodings, but they are very useful for obtaining a rating that approaches the judgment of a listener confronted with a normal listening experience. A second group will follow based this time on a Billboard hit parade.
  • The samples selected by IgorC are mostly known for the difficulties they pose to certain encoders. Castanets, Fatboy, Enola, Eig, Bachpsichord, Trumpet, Applauds, Velvet are classics that have been used here for 20 years or more. As an anecdote, I noticed afterwards that 3 samples were replaced by IgorC to conduct the 192 kbps test. I didn't understand in a first time why he had chosen spoken samples (Female Voice, French Ad). After checking again, I noticed these changes in his selection. Consequently, I tested the three alternative samples proposed by IgorC. I will only present the results of the 12 samples he had selected for the 192 kbps test.

I must admit, I'm quite proud of myself for making very few mistakes while ranking all these files. When the results are uncertain, I prefer not to insist, and if two files sound identical, I choose to rank them equally rather than strain my ears to detect subtle differences and increase the risk of errors.

Even with known problem samples, Wavpack lossy perform quite well at its lowest bitrate. But it's not transparent while classic lossy encoders are able to be close to that state with the same samples (see https://hydrogenaud.io/index.php/topic,120007.0.html). Moreover, the poor performances with trumpet sample is a big warning that 200 kbps with Wavpack can lead to strong distortions. It should be more clear with a selection dedicated to problem samples. On the other side, the classical music group also reveals that 200 kbps could also sound very well to the end user.

Adding a fourth encoder was finally a good choice (at least at this bitrate). Porcus was right to insist on that matter. And default (-g) setting really fills the gap between the fastest and the high mode. Difference is not always audible but at the end there's a clear trend. The high (hx) mode seems to be the sweet spot here: excellent encoding speed and audible improvement over inferior settings. The highest mode (hhx6) also brings a small overall improvement but effect can be very significant on some samples (Couperin, Velvet, You Look Good To Me).
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #41 –
And now, a tricky little statistical exercise, revisiting the results of my previous tests on the exact same samples.
A comparison can be made between Wavpack and other lossy encoders: AAC, Opus, MP3, USAC.

links to previous results :
AAC and OPUS (nov.2020): https://hydrogenaud.io/index.php/topic,120166.0.html
USAC (june-2021): https://hydrogenaud.io/index.php/topic,121099.0.html
MP3 (june-2020): https://hydrogenaud.io/index.php/topic,119333.0.html





I don't recommend generalizing these results or considering them as a definitive guide, but they do allow for an approximate assessment of the relative performance of different formats at various bitrates. The comparison will likely be more interesting with the group of 10 samples dedicated to contemporary music. It is noteworthy that Wavpack at 200 kbps is roughly comparable in quality to Opus and AAC at 140 kbps. Not bad at all!
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #42 –
Thanks so much for this guru, extremely interesting.
WavPack 5.8.1 -b384hx6cmv / qaac64 2.83 -V 100

Re: Need advices for listening tests (Wavpack lossy)

Reply #43 –
It is risky to conclude from so little, but FWIW: Those confidence intervals are quite narrow. The single worst -hx score in #41 is 4.0?  Every non-wv codec in the diagram has at least one worse datapoint.

Sure 200 is more than 142/137, and the number of 5's isn't overwhelming, but this doesn't look bad at all. 

Re: Need advices for listening tests (Wavpack lossy)

Reply #44 –
Very interesting and a pleasant surprise.  Personally, I always considered 250k as a minimum, perhaps 230 with a -hx4 or more.

So the 256 / b3 mighty be a 'sweet-spot' .  Lets wait and see .

Re: Need advices for listening tests (Wavpack lossy)

Reply #45 –
Guru, wonderful tests!

Observations:

  • WavPack's hybrid mode, with almost lowest possible lossy bitrate of 200kbps, is good on typical use cases, including storing classical music, with all settings achieving better than 3.7 MOS.
  • However, WavPack's lossy part at 200kbps is also somewhat 'dicey', with Trumpet performing worse than 2.0(Annoying) on solo, and there are other lossy part music far from the original music on techno and pops.
  • ADPCM at 3bit/sample, which is more compact than the typical 4bit/sample, actually probably performs better than a normal PCM at 8bit, dithered with noise shaping.

Re: Need advices for listening tests (Wavpack lossy)

Reply #46 –
I did some visualization rework of the Guruboolez's great test (what I did was reordering, combining, and that's it)



machine-readable text:
Spoiler (click to show/hide)

Re: Need advices for listening tests (Wavpack lossy)

Reply #47 –
Thanks Kamedo, it's much better!

Some updates on the test. It's difficult for me to make progress during the week. I managed to complete the 49th sample yesterday. So, I've finished the group named BILLBOARD and I'm halfway through the KAMEDO2 samples. I still have to test the KILLER SAMPLES, and then I'll be done with the first round of the test, which is also the easiest (around 200 kbps).

Here are the results for the BILLBOARD group:






The results are quite different from those of the previous two groups. Here, even the LOW ANCHOR samples are close to transparency. I remind you that these samples are random excerpts from a list of the best-selling music of the 2010s. They were not selected because they presented known or potential problems. Therefore, these samples are more useful for low or medium bitrate tests where nothing is absolutely transparent than for high bitrate tests where difficult passages are needed to identify differences. But I also remind that 200 kbps is the minimum bitrate for Wavpack, so these ten samples are still interesting to see how the format performs on music that is both ordinary and popular. Technically speaking, these are tracks mastered according to contemporary and commercial standards: reduced dynamics, maximum volume. I had to lower the volume to listen them.

The most striking aspect here is the quality achieved by encodings that are supposed to sound bad. Even when reduced to 8-bit sampling without any compression techniques, several of these samples remained transparent to my ears! I even made a mistake with one, rating the original instead of the degraded version (hence, the final score was adjusted to 5.0). The 3-bit ADPCM is slightly better because it doesn't produce audible hiss with the few excerpts that aren't too loud and whose average volume allows for some background noise to be heard at 8-bit. For Wavpack, only two samples differentiate the various tested modes: 'GIRLS LIKE YOU' and 'TIK TOK'. Statistically speaking, no encoding is better than another if we limit ourselves to this section of the test. Here's a zoomed version of the results.



I think we can still draw a conclusion from this segment of the test: Wavpack at 200 kbps has more than enough resources to handle this type of music, which is dynamically very compressed and does not present a problematic range of instruments. At least most often (problem moments may rise).

I'll post more here when finished (but i'll be next week) then I'll summarize the whole test in a different topic.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #48 –
Only just noticed this test effort, very interesting, guruboolez!  Just one question: do, on your previous test in post #40 above*, the WV-... encodings average at exactly 200 kbps on these samples? Do you have detailed bit-rate averages? Or did I overlook this information?

Chris

* https://hydrogenaud.io/index.php/topic,127451.msg1059784.html#msg1059784
If I don't reply to your reply, it means I agree with you.

Re: Need advices for listening tests (Wavpack lossy)

Reply #49 –
Quote from: C.R.Helmrich on
Only just noticed this test effort, very interesting, guruboolez!  Just one question: do, on your previous test in post #40 above*, the WV-... encodings average at exactly 200 kbps on these samples? Do you have detailed bit-rate averages? Or did I overlook this information?
Hi Christian,
you're correct, I haven't post any bitrate table yet. And to be honest, bitrate isn't tested at all for now. I don't need it at this stage of the test but I still have to do this.
Some data:
I have 55 short samples now, encoded with four variations (f/g/hx1/hhx6). That makes 220 files. Average bitrate is 206 kbps, with lowest at 198 kbps (Classical A.28. Beethoven) and highest at 234 kbps (fatboy_30sec encoded with hhx6).
Which means that -b2 doesn't use 2.0 bit per sample but is closer to 2.2…2.3 bps.
Wavpack Hybrid -c4hx6