lame 3.90.3 vs lame 3.96.1 at ~130 kbps
2004-11-15 08:37:47
Interested by performance of various lame encoders/settings at ~130 kbps (I'm using a small flash memory player), and also bored by unproductive quarrel about this question, I've decided to test them myself. The tests consist on simple ABCHR ratings, without ABX confrontations. When two (or three) files sounded very similar, I didn’t insist, and gave both the same note. ANOVA and TUKEY analysis will conclude the test. • SAMPLES The 18 samples used in last Roberto's test. In other words, it’s a wide panel of musical genre. I'm more interested in “classical” music, but this musical genre includes too many situations, and therefore need a dedicated test to be really useful. Maybe I’ll do it later… • CHALLENGERS — The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps] — The "untested" 3.96.1 with --preset 130 [ABR 130 kbps] — The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps] The average bitrate of 3.96.1 VBR mode is 129 kbps, and extreme values are 89 kbps and 153 kbps. • RESULTS • ANALYSIS (basic) Results are really contrasted: lame 3.96.1 VBR appears as the big winner (1st on 72% of tested samples), and 3.96.1 ABR as the worse (last on 72% of total samples). The old lame release sounded better than 3.96.1 ABR/VBR on one sample only [getiton.wav], but on the other side was never ranked as last. Important thing to note: there are much more differences between 3.90.3 ABR and 3.96.1 VBR than between 3.90.3 ABR and 3.96.1 ABR! • ANALYSIS (advanced) Using ANOVA tool first and then TUKEY¨PARAMETRIC analysis:ANOVA ANALYSIS: FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/ Blocked ANOVA analysis Number of listeners: 18 Critical significance: 0.05 Significance of data: 1.32E-004 (highly significant) --------------------------------------------------------------- ANOVA Table for Randomized Block Designs Using Ratings Source of Degrees Sum of Mean variation of Freedom squares Square F p Total 53 67.05 Testers (blocks) 17 42.65 Codecs eval'd 2 9.97 4.99 11.75 1.32E-004 Error 34 14.43 0.42 --------------------------------------------------------------- Fisher's protected LSD for ANOVA: 0.441 Means: 3.96.V 3.90.A 3.96.A 3.48 2.82 2.44 ---------------------------- p-value Matrix --------------------------- 3.90.A 3.96.A 3.96.V 0.004* 0.000* 3.90.A 0.096 ----------------------------------------------------------------------- 3.96.V is better than 3.90.A, 3.96.A PARAMETRIC TUKEY ANALYSIS FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/ Tukey HSD analysis Number of listeners: 18 Critical significance: 0.05 Tukey's HSD: 0.533 Means: 3.96.V 3.90.A 3.96.A 3.48 2.82 2.44 -------------------------- Difference Matrix -------------------------- 3.90.A 3.96.A 3.96.V 0.667* 1.039* 3.90.A 0.372 ----------------------------------------------------------------------- 3.96.V is better than 3.90.A, 3.96.A In both cases, lame 3.96.1 VBR is said better than lame ABR (both 3.90.3 and 3.96.1). Difference between 3.90.3 ABR and 3.96.1 is not significant enough to conclude with confidence on lame 3.90.3 superiority. Conclusion: for ~130 kbps encodings, lame 3.96.1 is better (for me) than 3.90.3 P.S. ABC/HR logs are available here