Skip to main content

Topic: lame 3.90.3 vs lame 3.96.1 at ~130 kbps (Read 13687 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Interested by performance of various lame encoders/settings at ~130 kbps (I'm using a small flash memory player), and also bored by unproductive quarrel about this question, I've decided to test them myself.
The tests consist on simple ABCHR ratings, without ABX confrontations. When two (or three) files sounded very similar, I didn’t insist, and gave both the same note. ANOVA and TUKEY analysis will conclude the test.

• SAMPLES

The 18 samples used in last Roberto's test. In other words, it’s a wide panel of musical genre. I'm more interested in “classical” music, but this musical genre includes too many situations, and therefore need a dedicated test to be really useful. Maybe I’ll do it later…

• CHALLENGERS

— The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps]
— The "untested" 3.96.1 with --preset 130 [ABR 130 kbps]
— The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps]

The average bitrate of 3.96.1 VBR mode is 129 kbps, and extreme values are 89 kbps and 153 kbps.

• RESULTS




• ANALYSIS (basic)

Results are really contrasted: lame 3.96.1 VBR appears as the big winner (1st on 72% of tested samples), and 3.96.1 ABR as the worse (last on 72% of total samples). The old lame release sounded better than 3.96.1 ABR/VBR on one sample only [getiton.wav], but on the other side was never ranked as last. Important thing to note: there are much more differences between 3.90.3 ABR and 3.96.1 VBR than between 3.90.3 ABR and 3.96.1 ABR!

• ANALYSIS (advanced)

Using ANOVA tool first and then TUKEY¨PARAMETRIC analysis:

ANOVA ANALYSIS:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 18
Critical significance:  0.05
Significance of data: 1.32E-004 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               53          67.05
Testers (blocks)    17          42.65
Codecs eval'd        2           9.97    4.99   11.75  1.32E-004
Error               34          14.43    0.42
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.441

Means:

3.96.V   3.90.A   3.96.A  
 3.48     2.82     2.44  

---------------------------- p-value Matrix ---------------------------

        3.90.A   3.96.A  
3.96.V   0.004*   0.000*  
3.90.A            0.096    
-----------------------------------------------------------------------

3.96.V is better than 3.90.A, 3.96.A


PARAMETRIC TUKEY ANALYSIS

Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Tukey HSD analysis

Number of listeners: 18
Critical significance:  0.05
Tukey's HSD:   0.533

Means:

3.96.V   3.90.A   3.96.A  
 3.48     2.82     2.44  

-------------------------- Difference Matrix --------------------------

        3.90.A   3.96.A  
3.96.V     0.667*   1.039*
3.90.A              0.372  
-----------------------------------------------------------------------

3.96.V is better than 3.90.A, 3.96.A


In both cases, lame 3.96.1 VBR is said better than lame ABR (both 3.90.3 and 3.96.1). Difference between 3.90.3 ABR and 3.96.1 is not significant enough to conclude with confidence on lame 3.90.3 superiority.


Conclusion: for ~130 kbps encodings, lame 3.96.1 is better (for me) than 3.90.3


P.S. ABC/HR logs are available here
  • Last Edit: 29 December, 2005, 03:40:17 PM by guruboolez

  • [proxima]
  • [*][*][*]
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #1
Quote
— The "tested" 3.90.3 with --alt-preset 130 [ABR 130 kbps]
— The "untested" 3.96.1 with --preset 130 [ABR 130 kbps]
— The ignored 3.96.1 -V 5 --athaa-sensitivity 1 [VBR ~130 kbps]


In reality 3.96.1 ABR and VBR have been tested by some people before the last multiformat 128 test when choosing the competitors. Nevertheless, i agree with your results: 3.96.1  -V5 --athaa-sensitivity 1 is better the the others. 3.96.1 ABR is really worse than the past version but Gabriel is working well and i already listened a good 3.97 alpha version. Warbling and HF problems seem resolved 
Quote
P.S. ABC/HR logs are available here

I'm curious about the sample where 3.96.1 VBR received a low score. There are some samples where 3.90.3 is still superior (i noted that too). I perceived a sort of watery/ringing sound that --athaa-sensitivity 1 reduces but not eliminate completely (http://www.hydrogenaudio.org/forums/index....topic=21079&hl=).
What's your motivation for scoring so little ?
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1

  • shadowking
  • [*][*][*][*][*]
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #2
Thanks guru !

I also tested some samples with V5 vs. ABR (lame 3.96.1) and my results were the same as yours. VBR is superior for me as well.
wavpack -b4x4s1c

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #3
Quote
I'm curious about the sample where 3.96.1 VBR received a low score (...)
What's your motivation for scoring so little ?
[a href="index.php?act=findpost&pid=253980"][{POST_SNAPBACK}][/a]


Are you talking about one precise sample, or all problem samples for 3.96.1-VBR?
I didn't use any low anchor -> notation could be exagerated in some case. Nevertheless, when a file was ranked 2.0/5, it means that I was really annoyed by some distortions (not necessary ringing, but also metallic/acid colouration).


[span style='font-size:8pt;line-height:100%']P.S. Sorry, I thought I had reply to your question. I don't know what happened...[/span]

  • dev0
  • [*][*][*][*][*]
  • Developer
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #4
These results confirm what has been stated by [proxima], myself and others before: 3.96.1's CBR/ABR mode has some serious regressions compared to 3.90.3, but Gabriel is working hard on fixing them.
I find the performance of -V5 to be seriously impressive for its bitrate (even though my music tends to bloat up to 150kbps with this setting and is still far from transparent -> see sincealways).
  • Last Edit: 15 November, 2004, 11:46:59 AM by dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

  • Tang
  • [*][*][*]
  • Members (Donating)
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #5
Hi Guru,
I've just read your results (very interesting as usual, thanks) but i don't understand something... Why didn't you compare the 3.96.1 VBR with modified 3.90.3 "--alt-preset medium"???
This wouldn't have made sense?

Anyway thanks for your test results... (was finally using 3.96.1 for some weeks however)

Best regards

  • dev0
  • [*][*][*][*][*]
  • Developer
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #6
--alt-preset medium would have come out quite a bit over the target bitrate of 130kbps.
"To understand me, you'll have to swallow a world." Or maybe your words.

  • Tang
  • [*][*][*]
  • Members (Donating)
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #7
Quote
--alt-preset medium would have come out quite a bit over the target bitrate of 130kbps.
[a href="index.php?act=findpost&pid=254088"][{POST_SNAPBACK}][/a]

Ok I thought -apm would get a bitrate sightly inferior to the 130kbps target... (i remember about 115 kbps/s for a CDA but it was "chanson" not classical thought)
Anyway not a matter but i thought comparing the two codec with VBR would have been helpfull...

  • Gabriel
  • [*][*][*][*][*]
  • Developer
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #8
Preset medium (similar to V4 in 3.96) is targetting 150-160kbps overall

  • Tang
  • [*][*][*]
  • Members (Donating)
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #9
Quote
Preset medium (similar to V4 in 3.96) is targetting 150-160kbps overall
[a href="index.php?act=findpost&pid=254154"][{POST_SNAPBACK}][/a]

Hi Gabriel,
Thanks for the info... Seems that i've been mistaken cause i used -apm only once time with one CDA of song (live of Jeanne Cherhal) which gave me an average bitrate about 115kbps...
Anyway a single shot encoding (~tracks thought) didi'nt allow généralisation, that's why i mistaken...
Sorry..

regards,
Tanguy

  • [proxima]
  • [*][*][*]
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #10
Quote
Are you talking about one precise sample

Those coloured orange in your test.
I ask because i have found some samples (LisztBMinor, Blackwater) where VBR 3.96 still suffers a little with ringing/HF problems even with --athaa-sensitivity 1.
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1

  • Lyx
  • [*][*][*][*][*]
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #11
Just for understanding:

3.90.3 ABR and 3.96.1 VBR both average at 130kbit
Yet still in your test the bitrates on the samples are usually higher with 3.96.1 VBR

Am i right when concluding that 3.96.1 VBR appears to identify "problem-samples" more reliably and ups the bitrate? Or am i misinterpreting something?

- Lyx
I am arrogant and I can afford it because I deliver.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #12
Quote
Quote
Are you talking about one precise sample

Those coloured orange in your test.
I ask because i have found some samples (LisztBMinor, Blackwater) where VBR 3.96 still suffers a little with ringing/HF problems even with --athaa-sensitivity 1.
[a href="index.php?act=findpost&pid=254179"][{POST_SNAPBACK}][/a]

I can't remember exactly what problems I heard on these samples (I'm at work, and I don't have access to the log), but 3.96.1 problems are not limited to ringing. Correcting this problem won't solve all issues.

  • dev0
  • [*][*][*][*][*]
  • Developer
lame 3.90.3 vs lame 3.96.1 at ~130 kbps
Reply #13
Quote
Am i right when concluding that 3.96.1 VBR appears to identify "problem-samples" more reliably and ups the bitrate? Or am i misinterpreting something?[a href="index.php?act=findpost&pid=254221"][{POST_SNAPBACK}][/a]

Yes, that's what makes VBR so wonderful.
But it also makes the target bitrate less controllable.
But across a wide range of music/genres it should average around 130kbps.
  • Last Edit: 16 November, 2004, 04:19:40 AM by dev0
"To understand me, you'll have to swallow a world." Or maybe your words.