Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Need advices for listening tests (Wavpack lossy) (Read 9302 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Need advices for listening tests (Wavpack lossy)

Reply #50 –
Quote from: bryant on
I have to say I’m very impressed with how well 200 kbps lossy WavPack is doing, despite my suggestion that it’s not recommended. I’m sure that problem samples will trip it up in the near future, but it’s still encouraging. I wonder if the new DNS is responsible, or whether is was always this good.
Yes, lately I performed some quick (an hour or two total) personal ABX tests with newest WavPack and LossyWAV and with their lowest settings I managed to only find 1 fragment of 1 track where I was able to ABX WavPack, and I wasn't able to ABX LossyWAV U at all. Either I'm getting deaf or you made some major strides because several versions ago I wasn't very impressed with it (LossyWAV seemed much better). Now I surely consider switching to one of these as I seem to be much more immune to bitdepth/noise artifacts than typical psychoacoustic/frequency domain preecho, ringing, etc.

That being said I'm very interested in results of this listening test and hoping for LossyWAV too!

Re: Need advices for listening tests (Wavpack lossy)

Reply #51 –
That pretty much surmises all my thoughts on my own dealings with lossy encoding.

But I still bet, in my particular case, that's more down to piling up the years.
As my sig implies (the older, the 'lossier'), ageing can be sometimes a blessing in disguise. :))
• Listen to the music, not the media it's on
• The older, the 'lossier'

Re: Need advices for listening tests (Wavpack lossy)

Reply #52 –
I just finished the latest set of musical samples, which includes all 20 selected by Kamedo2. Here are the charts:






There are no major surprises here. The same hierarchy as before is observed. There is a significant rating gap between mode -g and mode -hx1 (about 0.30 points). The Tom's Diner sample posed the most difficulty for Wavpack to my ears. I believe I detected my first "artifact." It is still noise that is heard, but the defect reminds me of the issues caused by the LAME encoder before version 3.98 was finalized on very tonal samples (here the sample isn't tonal). It creates something like sound blotches in certain areas. This is particularly striking with modes -f and -g; the two higher modes are slightly better but do not solve the problem. I did not expect to hear such sharp noise increases in this sample.
Statistically speaking, the very slow hhx6 mode isn't better than the ultra fast -f mode. But there's a clear trend.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #53 –
Here is now the compilation of the 55 musical (and few speech) samples tested so far:





And a zoomed version without anchors:



The overall result is more statistically interesting. Here, it is statistically valid to assert that mode -f is qualitatively inferior to mode -hhx6 (with a probability greater than 95%). It is also evident that several samples are rated at levels where a disturbance can be heard. The highest-quality mode alone has ten samples rated below 4. This represents nearly 20% of the tracks tested, which is significant in absolute terms. I cannot say that Wavpack is transparent at 200 kbps.

However, Wavpack is already transparent at 200 kbps for 22 samples that achieved the maximum score, and nearly transparent for 40 of the 55 samples that scored 4.5 or higher. Overall quality is therefore very good, but a few glitches can still be heard here and there. The last group of samples will, moreover, focus on excerpts known to be challenging for Wavpack, and the results at 200 kbps are expected to be predictably poor.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #54 –
More precise statistics:

Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 55
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              329         392.31
Testers (blocks)    54         179.12
Codecs eval'd        5          98.18   19.64   46.09  0.00E+000
Error              270         115.01    0.43
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.245

Means:

higherxx highx    normal   fast     ADPCM    8bit     
  4.55     4.41     4.19     4.04     3.37     3.05   

---------------------------- p-value Matrix ---------------------------

         highx    normal   fast     ADPCM    8bit     
higherxx 0.274    0.004*   0.000*   0.000*   0.000*   
highx             0.071    0.003*   0.000*   0.000*   
normal                     0.232    0.000*   0.000*   
fast                                0.000*   0.000*   
ADPCM                                        0.009*   
-----------------------------------------------------------------------

higherxx6 is better than normal, fast, ADPCM, 8bit
highx is better than fast, ADPCM, 8bit
normal is better than ADPCM, 8bit
fast is better than ADPCM, 8bit
ADPCM is better than 8bit

With these 55 samples and 1 listener:
-hxx6 > -g & -f
-hx1 > -f
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #55 –
I need to tackle the final group of samples, all of which have been specifically selected for their ability to highlight weaknesses in WavPack encodings.
But instead of using low anchors and the four parameters considered so far, I’m wondering if it might be more useful to test other options. For instance, I could explore a mode more extensively, such as h / hx / hx4 / hx6. The advantage would be that it becomes easier to assess the benefits of these parameters. Of course, other combinations are also possible—I’m open to suggestions, they're always helpful.

Given that some of the samples I’ve set aside are particularly challenging for WavPack, I believe the variations in quality will be significant, possibly even very pronounced. I think in any case that only this type of signal will allow us to hear differences between very close modes.
I also think testing up to eight parameters could be feasible with such samples. As mentioned earlier, I would remove any reference to a low anchor here to save space (I strongly suspect they face fewer issues than WavPack and can therefore no longer serve their initial purpose).

The parameters that come to mind would be:
-f ? maybe to get the worst and see how far it can improve at the same bitrate
-g: default
-gx: default +
-h: high
-hx: high+
-hx4: high++ ( a true gem according to Shadowking)
-hx6: high+++ (to see if a gem can shine further?)
-hh
-hhx6

That makes 9 settings. If I had to remove one, discarding the -f setting could be logical (would someone interested by audio quality use the fast and thus lowest quality mode?).
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #56 –
I think the maximum mode -hhx6 is a necessity...

Re: Need advices for listening tests (Wavpack lossy)

Reply #57 –
I totally agree with you, I will definitely include the highest quality mode :)
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #58 –
Quote from: guruboolez on
(would someone interested by audio quality use the fast and thus lowest quality mode?).
It doesn't seem like they would.
• Listen to the music, not the media it's on
• The older, the 'lossier'

Re: Need advices for listening tests (Wavpack lossy)

Reply #59 –
Oh, putting more in? Only at bitrate 200, or to plan a higher rate too?

*IF* you are to do 9 settings, I would rather select -hhx4 than -hx6. Because "x4" is likely a more reasonable choice than "x6", and you will still get an "x4 vs x6" comparison (only at the "hh" level).

By the way: is it just me, or does "b2" encode even faster atop "-g, -h, -hh", but significantly slower atop anything with x?
E.g. -hb2 faster than -h, but -hxb2 slower than -hx?
(It wouldn't be surprising if you got at least some return on the extra effort ...)

... that consideration may influence choice of "-hh" vs "-hhx". The former is justified if you want to ask for a relationship between -hx and -hh, we don't know whether the "x" or the "extra h" will hit hardest.

Re: Need advices for listening tests (Wavpack lossy)

Reply #60 –
Yes, I plan to conduct the same test at higher bitrates. Among the samples available for this test, most do not appear to be transparent at 400 kbps with intermediate settings.

I take note of your suggestion. The idea of having four variations of the same preset seemed interesting to me. But upon reflection, focusing on the -hh mode makes sense in a quality evaluation test. It is true that the -hh mode is not the most attractive for WavPack's lossless mode. But for lossy, it makes sense.

Regarding speed, I haven't tested it. Your remark might find a resonance in David's comments about the -b2 mode (Reply #14).
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #61 –
I conducted two beta tests of a configuration that initially included 11 and then 9 contenders. Since each sample needs to be evaluated against all the others, the complexity quickly becomes exponential. Additionally, the challenge is compounded by listening fatigue, which comes quickly. As a result, I ended up with inconsistent findings (e.g., hh<g), issues I had managed to avoid by reducing the number of Wavpack encodings to 4 in the previous 55 test samples. Therefore, I need to scale back my ambitions. I plan to retain the four previous settings (f/g/hx/hhx6) and add two more to better understand the differences between the various possible levels. I will include the -hx4 mode suggested by Shadowking, who is truly an authority on the subject. This leaves one final spot to be determined. Maybe something between hx4 and hhx6. -hhx sounds like a good choice. hhx4 as suggested by Porcus is maybe too close to hhx6 and also too far from the previous step (hx4).
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #62 –
So, down to a more managable number.

Currently you are at
-f
-g
-hx
-hhx6

What you don't get from those four:
 * As mentioned: evaluating any benefits from "up one mode" (say, -g to -h) vs "up one x".
You could resolve that by adding -hh. Then you get one such comparison that might  very well yield ambiguous results.
I kinda suggest that, as it gives just one more setting to try atop -g and -hx. And -g and -hx both make sense. -g as it is the default for whatever historical reason, and -hx is what the dev uses, and has been proposed as the recommended setting (by @shadowking, who can weigh in on whether that is still a fave). 
 * The -hx4 setting, which is slow but on some signals (and we don't know whether lossy-mode is such a situation) does very well. (Example file in the retail, and in ktf's test on 96 kHz sources, the "x4" makes at least the same difference as flac to frog)
BUT: you do have -hhx6. And the big difference "-x"-wise is between -x3 and -x4 (in addition to, I guess, between -x0 (= "no x") and -x1 (= -x) !).

Maybe sacrifice "-f", arguing that those who actually select settings will not choose it?
But it might be good enough once you step up bitrate ... we don't know! ;) 

Re: Need advices for listening tests (Wavpack lossy)

Reply #63 –
Thanks Porcus for all your suggestions.

Let's take for basis this selection:

g
h
hx
hx4
hh
hhx
hhx6

fast mode is discarded for testing all killer samples.
default mode is included as the fastest setting tested. But no extra option (assuming that WV lossy users will most likely use any -h or --hh variation.
h - hx -hx4: now we have some useful comparison between  g and h, between h and hx and between hx and hxn
same for hh-hhx-hhx6.
=> 7 contenders, one more than my upper limit. I can try. I think it's manageable.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #64 –
I gathered 15 samples for this separate group. I launched the test and so far the result seems promising.

I can share one result (sample was never posted here — it comes from the same album as the Bibilolo sample I posted here twenty years ago):

Code: [Select]
ABC/HR for Java, Version 0.5b, 07 mars 2025
Testname: 

Tester: 

1L = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hhx6.wv
2L = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2g.wv
3L = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hhx.wv
4L = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hh.wv
5R = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2h.wv
6R = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hx.wv
7L = D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hx4.wv

---------------------------------------
General Comments: tested range: 8.74 - 13.76 (high pitched signal)
---------------------------------------
1L File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hhx6.wv
1L Rating: 3.5
1L Comment: Very impressive improvement compared to the five worse encodings
---------------------------------------
2L File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2g.wv
2L Rating: 1.0
2L Comment: blizzard noise. 1.0 is not excessive
---------------------------------------
3L File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hhx.wv
3L Rating: 2.2
3L Comment: noise is shifted on the right channel?! Slightly less noisy than its twin 6R
---------------------------------------
4L File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hh.wv
4L Rating: 1.8
4L Comment: significant reduction in noise compared to the two worst files
---------------------------------------
5R File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2h.wv
5R Rating: 1.1
5R Comment: marginally less worse than 1R
---------------------------------------
6R File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hx.wv
6R Rating: 2.0
6R Comment: noise is shifted on the right channel?!
---------------------------------------
7L File: D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hx4.wv
7L Rating: 3.5
7L Comment: Very impressive improvement compared to the five worse encodings. Very similar to 1L
---------------------------------------

ABX Results:
D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hhx6.wv vs D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hx4.wv
    3 out of 9, pval = 0.91


---- Detailed ABX results ----
D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hhx6.wv vs D:\samples WAVPACK\SELECTION 4\Bibilolo8_b2hx4.wv
Playback Range: 08.744 to 13.766
    10:30:46 AM f 0/1 pval = 1.0
    10:30:48 AM p 1/2 pval = 0.75
    10:30:50 AM f 1/3 pval = 0.875
    10:30:56 AM f 1/4 pval = 0.937
    10:31:01 AM f 1/5 pval = 0.968
    10:31:06 AM f 1/6 pval = 0.984
    10:31:17 AM p 2/7 pval = 0.937
    10:31:19 AM p 3/8 pval = 0.855
    10:31:25 AM f 3/9 pval = 0.91


The results are consistent with this sample, except for hh<h which is suspicious. This kind of possible mistake are expected and ANOVA or FRIEDMAN analysis is there to clean up the global vision.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #65 –
Wavpack Test, Day 16

I have completed the last fifteen samples, all focused on pushing WavPack to its limits. The extreme difficulty of most of these samples allowed me to test more parameters and better understand their impact on sound quality. It is much harder to perceive these gains on standard samples.

Without further ado, here are the results:





And a zoomed version:





Several lessons can be drawn from this:

- The qualitative collapse of Wavpack on this selection is very clear. I did remove the LOW ANCHOR from the equation, which increases the risk of scoring too low. But if you listen to the sound quality of a number of these encodings, it will be difficult to further degrade the signal and the final score.

- The transition between -g and -h provides a significant and statistically valid gain; the transition between -h and -hx also provides a significant gain, but it is not guaranteed. Indeed, on the Fuechen_II sample, -h seemed noticeably better than -hx; the same goes for the Sachico sample, where the -x switch adds a perfectly clear artifact (a sort of tick-tick-tick sound that repeats at short intervals).

- The -hh mode and its variants are really the ones to favor. On the one hand, because the gap between -h and -hh is 0.5 points, which is considerable. The difference is statistically valid. Adding the -x switch doesn't bring much (+0.12 and not validated). On the Insolence sample, -hxx is clearly inferior to -hx. Once again, the -x mode has a generally positive impact but seems to also introduce defects.

- The parameters -xn where n>3 are by far the ones to favor. Between -h and -hx4: +1.04 points! Between -hh and -hhx6: +0.82 points. It is with these modes that some major defects really fade away and improvements are audible on all critical samples. The issues remain perfectly audible, but the qualitative leap is very significant. -hhx6 really brings something extra compared to -hx4 (+0.24 points). If the gap is not statistically significant, it is only because it could not be heard on 4 of the 15 samples. Otherwise, for the other 11, -hhx6 appeared as the best of all, sometimes with a clear lead (for example: on Metamorphose).

- There are artifacts with Wavpack. The contribution of the different parameters does not only consist of increasing or decreasing the noise uniformly. On certain passages, the noise is shifted to one channel only (Bibilolo8, Bibilolo12, Metamorphose to some extent). With other samples, a very clear tick-tick-tick-tick sound can be heard (Sachico, Insolence).


Few words about these samples. Some are known (Atem_Lied, Metamorphose, Furious, Glockenspiel). I went hunting before starting this test and caught a few more. They are available on this thread with complete information:
https://hydrogenaud.io/index.php/topic,127551.0.html

There are real instruments: a piccolo flute, an electric guitar, a glockenspiel, a bird whistle. But the most terrific (or terrible) samples comes from electronic recordings.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #66 –
Interesting that "Sachico" sample, where to me the most fascinating about the artifacts is something else than what you mention, namely that the "breathing"/"air flow" noise is so much more pronounced after "the second of the ping sounds" from 2 to 4 seconds. Not so strange, as during that segment there is a hf tone up there, apparently giving the encoder too much to handle.
(Tried a few more settings on that, this difference between 2 to 4 seconds and later is very pronounced on -g/-gx/-h/-hx ... and there is a difference between -hx3 and -hx4 too.)

... this at the risk of invoking TOS#8, but hey: it is already established that it struggles on that sample.

Re: Need advices for listening tests (Wavpack lossy)

Reply #67 –
for what is described as artifacts;

Does -x4 or more solve it (make noise more uniform)?
Does switching to fixed Noise shaping give more uniform noise (-s.5 or -s0 ) for these cases ?

Re: Need advices for listening tests (Wavpack lossy)

Reply #68 –
I was trying to ABX some PWM-like music from ZX Spectrum, AY/FM chip music, Amiga modules with extreme stereo separation, but failed to find any difference at -hhx6 and LossyWAV U. I succeeded only with an ambient electronic track. Will try to ABX your samples today and if I'm not deaf will give LossyWAV U a quick spin too...

Guru do you keep some headroom with replaygain or something? WavPack lossy tends to raise peaks a lil and at least Atem_Lied got True Peak clipping after encoding.

EDIT: Fuechen I & II are True Peak clipped even before encoding.

Re: Need advices for listening tests (Wavpack lossy)

Reply #69 –
Quote from: shadowking on
for what is described as artifacts;

Does -x4 or more solve it (make noise more uniform)?
Does switching to fixed Noise shaping give more uniform noise (-s.5 or -s0 ) for these cases ?
At the moment, I’d rather avoid getting sidetracked by advanced settings and stick to my test. Soon, I plan to carry out the same test with a higher throughput (-b3). The samples are available online, and I warmly encourage the most enthusiastic individuals to explore various experiments (I’d be glad to take part and share my feedback based on the insights gathered).

About artifact:
The tic tic tic that sounds obvious can also be seen:



The amount of additional noise is remarkable. You can clearly see fine vertical lines of noise appearing at very short intervals with the -x parameter. These lines almost certainly match what I heard during the test.
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #70 –
Quote from: rutra80 on
Guru do you keep some headroom with replaygain or something? WavPack lossy tends to raise peaks a lil and at least Atem_Lied got True Peak clipping after encoding.
Album Gain and Track Gain for each samples (except those from unknown source) are available here:
https://hydrogenaud.io/index.php/topic,127551.0.html

During my test volume was rather low to avoid listening fatigue.
The only volume modification was normalisation on the ABC/HR side. I never use replay gain on short samples. I sometimes have to turn the volume knob. But this is normal practice when classical samples are mixed with modern productions.


I don't know if it might help, here are Atem_Lied full track True Peak values (Case 0.6.14 component):
<TRUEPEAK_SCANNER_ALBUM_GAIN> : -0,71 dB
<TRUEPEAK_SCANNER_ALBUM_LUFS_I> : -17,29 LUFS
<TRUEPEAK_SCANNER_ALBUM_MAX_LUFS_M> : -0,69 LUFS
<TRUEPEAK_SCANNER_ALBUM_MAX_LUFS_S> : -4,95 LUFS
<TRUEPEAK_SCANNER_ALBUM_PLR> : 17,14 LU
<TRUEPEAK_SCANNER_ALBUM_RMS> : -24,72 dBFS
<TRUEPEAK_SCANNER_CLIPPED_SAMPLES_ALBUM> : 0
<TRUEPEAK_SCANNER_CLIPPED_SAMPLES_TRACK> : 0
<TRUEPEAK_SCANNER_PEAK_POSITION> : 8:53.023
<TRUEPEAK_SCANNER_TRACK_GAIN> : -2,22 dB
<TRUEPEAK_SCANNER_TRACK_LUFS_I> : -15,78 LUFS
<TRUEPEAK_SCANNER_TRACK_MAX_LUFS_M> : -3,22 LUFS
<TRUEPEAK_SCANNER_TRACK_MAX_LUFS_S> : -7,90 LUFS
<TRUEPEAK_SCANNER_TRACK_PLR> : 15,24 LU
<TRUEPEAK_SCANNER_TRACK_RMS> : -22,79 dBFS
Wavpack Hybrid -c4hx6

Re: Need advices for listening tests (Wavpack lossy)

Reply #71 –

Heavens to Betsy! 😮

I think some will agree there will be our impressions/assumptions before this test and a bunch of certainties after it. 

To think not too long ago I was in the impression it all boiled down to the noise floor being audible or not.
Still on -b2 and I'm already flabbergasted!
• Listen to the music, not the media it's on
• The older, the 'lossier'

Re: Need advices for listening tests (Wavpack lossy)

Reply #72 –
Great set of samples, at -hhx6 most artifacts are obvious and annoying indeed. I gave a quick spin against LossyWAV U and yet it performs much better (none are annoying and some indistinguishable), after passing to FLAC -8ep all the files are larger (some several dozen kbps but some twice as large). Tomorrow I'll either try matching WavPack lossy kbps to LossyFLAC ones, or pass it as LossyWV and compare again.

Re: Need advices for listening tests (Wavpack lossy)

Reply #73 –
 Forcing L/R stereo (-j0) eliminates the tic-tic artifact at the expense of more overall noise.
Higher setting like -b3hx6 sounds good.

Re: Need advices for listening tests (Wavpack lossy)

Reply #74 –
LossyWV (U & -hhx6) turned out significantly larger than LossyFLAC (U & -ep8) so I matched kbps of WavPack lossy versions to LossyFLAC by bruteforcing -b:

Atem_Lied - WV not annoying, LF imperceptible
Bibilolo8 - WV imperceptible, LF imperceptible
Bibilolo12 - WV imperceptible, LF imperceptible
Fuechen_1 - WV imperceptible, LF imperceptible
Fuechen_II - WV imperceptible, LF imperceptible
Furious - WV not annoying, LF imperceptible
Guitar_Three - WV imperceptible, LF imperceptible
Insolence - WV imperceptible, LF imperceptible
Keiser - WV slightly annoying, LF imperceptible
Metamorphose - WV imperceptible, LF imperceptible
Piccolo_B - WV not annoying, LF slightly annoying
Sachico - WV not annoying, LF imperceptible
Silver_Apples - WV imperceptible, LF imperceptible
SQAM_glockenspiel_cut - WV imperceptible, LF imperceptible
Suspend - WV slightly annoying, LF imperceptible