Interested by performance of various lame encoders/settings at ~130 kbps (I'm using a small flash memory player), and also bored by unproductive quarrel about this question, I've decided to test them myself.
The tests consist on simple ABCHR ratings, without ABX confrontations. When two (or three) files sounded very similar, I didn’t insist, and gave both the same note. ANOVA and TUKEY analysis will conclude the test.
• SAMPLES
The 18 samples used in last Roberto's test. In other words, it’s a wide panel of musical genre. I'm more interested in “classical” music, but this musical genre includes too many situations, and therefore need a dedicated test to be really useful. Maybe I’ll do it later…
• CHALLENGERS
— The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps]
— The "untested" 3.96.1 with --preset 130 [ABR 130 kbps]
— The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps]
The average bitrate of 3.96.1 VBR mode is 129 kbps, and extreme values are 89 kbps and 153 kbps.
• RESULTS
(http://audiotests.free.fr/tests/2004.11/lame130_results.png)
• ANALYSIS (basic)
Results are really contrasted: lame 3.96.1 VBR appears as the big winner (1st on 72% of tested samples), and 3.96.1 ABR as the worse (last on 72% of total samples). The old lame release sounded
better than 3.96.1 ABR/VBR on one sample only [getiton.wav], but on the other side was never ranked as last.
Important thing to note: there are much more differences between 3.90.3 ABR and 3.96.1 VBR than between 3.90.3 ABR and 3.96.1 ABR!• ANALYSIS (advanced)
Using ANOVA tool first and then TUKEY¨PARAMETRIC analysis:
ANOVA ANALYSIS: FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 18
Critical significance: 0.05
Significance of data: 1.32E-004 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 53 67.05
Testers (blocks) 17 42.65
Codecs eval'd 2 9.97 4.99 11.75 1.32E-004
Error 34 14.43 0.42
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.441
Means:
3.96.V 3.90.A 3.96.A
3.48 2.82 2.44
---------------------------- p-value Matrix ---------------------------
3.90.A 3.96.A
3.96.V 0.004* 0.000*
3.90.A 0.096
-----------------------------------------------------------------------
3.96.V is better than 3.90.A, 3.96.A
PARAMETRIC TUKEY ANALYSISFRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis
Number of listeners: 18
Critical significance: 0.05
Tukey's HSD: 0.533
Means:
3.96.V 3.90.A 3.96.A
3.48 2.82 2.44
-------------------------- Difference Matrix --------------------------
3.90.A 3.96.A
3.96.V 0.667* 1.039*
3.90.A 0.372
-----------------------------------------------------------------------
3.96.V is better than 3.90.A, 3.96.A
In both cases, lame 3.96.1 VBR is said better than lame ABR (both 3.90.3 and 3.96.1). Difference between 3.90.3 ABR and 3.96.1 is not significant enough to conclude with confidence on lame 3.90.3 superiority.
Conclusion: for ~130 kbps encodings, lame 3.96.1 is better (for me) than 3.90.3
P.S. ABC/HR logs are available here (http://audiotests.free.fr/tests/2004.11/lame_130_ABC.zip)
— The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps]
— The "untested" 3.96.1 with --preset 130 [ABR 130 kbps]
— The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps]
In reality 3.96.1 ABR and VBR have been tested by some people before the last multiformat 128 test when choosing the competitors. Nevertheless, i agree with your results: 3.96.1 -V5 --athaa-sensitivity 1 is better the the others. 3.96.1 ABR is really worse than the past version but Gabriel is working well and i already listened a good 3.97 alpha version. Warbling and HF problems seem resolved
P.S. ABC/HR logs are available here (http://www.foobar2000.net/divers/tests/2004.11/lame_130_ABC.zip)
I'm curious about the sample where 3.96.1 VBR received a low score. There are some samples where 3.90.3 is still superior (i noted that too). I perceived a sort of watery/ringing sound that --athaa-sensitivity 1 reduces but not eliminate completely (http://www.hydrogenaudio.org/forums/index....topic=21079&hl= (http://www.hydrogenaudio.org/forums/index.php?showtopic=21079&hl=)).
What's your motivation for scoring so little ?
Thanks guru !
I also tested some samples with V5 vs. ABR (lame 3.96.1) and my results were the same as yours. VBR is superior for me as well.
I'm curious about the sample where 3.96.1 VBR received a low score (...)
What's your motivation for scoring so little ?
[a href="index.php?act=findpost&pid=253980"][{POST_SNAPBACK}][/a]
Are you talking about one precise sample, or all problem samples for 3.96.1-VBR?
I didn't use any low anchor -> notation could be exagerated in some case. Nevertheless, when a file was ranked 2.0/5, it means that I was really annoyed by some distortions (not necessary ringing, but also metallic/acid colouration).
[span style='font-size:8pt;line-height:100%']
P.S. Sorry, I thought I had reply to your question. I don't know what happened...[/span]
These results confirm what has been stated by [proxima], myself and others before: 3.96.1's CBR/ABR mode has some serious regressions compared to 3.90.3, but Gabriel is working hard on fixing them.
I find the performance of -V5 to be seriously impressive for its bitrate (even though my music tends to bloat up to 150kbps with this setting and is still far from transparent -> see sincealways).
Hi Guru,
I've just read your results (very interesting as usual, thanks) but i don't understand something... Why didn't you compare the 3.96.1 VBR with modified 3.90.3 "--alt-preset medium"???
This wouldn't have made sense?
Anyway thanks for your test results... (was finally using 3.96.1 for some weeks however)
Best regards
--alt-preset medium would have come out quite a bit over the target bitrate of 130kbps.
--alt-preset medium would have come out quite a bit over the target bitrate of 130kbps.
[a href="index.php?act=findpost&pid=254088"][{POST_SNAPBACK}][/a]
Ok I thought -apm would get a bitrate sightly inferior to the 130kbps target... (i remember about 115 kbps/s for a CDA but it was "chanson" not classical thought)
Anyway not a matter but i thought comparing the two codec with VBR would have been helpfull...
Preset medium (similar to V4 in 3.96) is targetting 150-160kbps overall
Preset medium (similar to V4 in 3.96) is targetting 150-160kbps overall
[a href="index.php?act=findpost&pid=254154"][{POST_SNAPBACK}][/a]
Hi Gabriel,
Thanks for the info... Seems that i've been mistaken cause i used -apm only once time with one CDA of song (live of Jeanne Cherhal) which gave me an average bitrate about 115kbps...
Anyway a single shot encoding (~tracks thought) didi'nt allow généralisation, that's why i mistaken...
Sorry..
regards,
Tanguy
Are you talking about one precise sample
Those coloured orange in your test.
I ask because i have found some samples (LisztBMinor, Blackwater) where VBR 3.96 still suffers a little with ringing/HF problems even with --athaa-sensitivity 1.
Just for understanding:
3.90.3 ABR and 3.96.1 VBR both average at 130kbit
Yet still in your test the bitrates on the samples are usually higher with 3.96.1 VBR
Am i right when concluding that 3.96.1 VBR appears to identify "problem-samples" more reliably and ups the bitrate? Or am i misinterpreting something?
- Lyx
Are you talking about one precise sample
Those coloured orange in your test.
I ask because i have found some samples (LisztBMinor, Blackwater) where VBR 3.96 still suffers a little with ringing/HF problems even with --athaa-sensitivity 1.
[a href="index.php?act=findpost&pid=254179"][{POST_SNAPBACK}][/a]
I can't remember exactly what problems I heard on these samples (I'm at work, and I don't have access to the log), but 3.96.1 problems are not limited to ringing. Correcting this problem won't solve all issues.
Am i right when concluding that 3.96.1 VBR appears to identify "problem-samples" more reliably and ups the bitrate? Or am i misinterpreting something?[a href="index.php?act=findpost&pid=254221"][{POST_SNAPBACK}][/a]
Yes, that's what makes VBR so wonderful.
But it also makes the target bitrate less controllable.
But across a wide range of music/genres it
should average around 130kbps.