Skip to main content

Topic: My LAME tests (Read 2980 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • tr1n1tr0n
  • [*]
My LAME tests
Hi forum people,

After following the various posts arguing for and against the transparency of LAME and it's various bitrates and encoding methods, I decided to put my ears to the test and try a blind test with a variety of modes.

The piece of music used was track 5 from "Faith - A Message From The Spirits" on the Soul Jazz label, which is an album of religious music from around the world.  More specifically, the piece in question is a solo percussion piece on Djouba, which sound more or less like bongos (to me anyway!) so as you can imagine, the music is fast with great transient attack.

Music ripped with EAC Secure/C2, encoded with LAME 3.94 a11 17Feb build, decoded using the MAD 0.14.2b plugin, listened to through an Echo Gina 20 digitally connected to a Marantz PM-75 amp, monitored on Sennheiser HD580 headphones.  ABC/HR was used to audition the samples.

Results:

All cbr files are stereo, q2.  All preset files use their respective settings in LAME.
"CBR 192 EAC Filt" is 192Kbit Stereo Q2, with a lowpass filter (19383-19916Hz) that EAC includes, even if no command line option is set.

ABC/HR Version 0.9b, 30 August 2002
Testname:

1R = D:\cdrips\test\preset 128.wav
2L = D:\cdrips\test\cbr 256 no filt.wav
3L = D:\cdrips\test\preset standard.wav
4L = D:\cdrips\test\preset cbr 192.wav
5L = D:\cdrips\test\cbr 192 eac filt.wav
6L = D:\cdrips\test\cbr 192 no filt.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: D:\cdrips\test\preset 128.wav
1R Rating: 2.0
1R Comment:
---------------------------------------
2L File: D:\cdrips\test\cbr 256 no filt.wav
2L Rating: 4.7
2L Comment:
---------------------------------------
3L File: D:\cdrips\test\preset standard.wav
3L Rating: 4.4
3L Comment:
---------------------------------------
4L File: D:\cdrips\test\preset cbr 192.wav
4L Rating: 3.9
4L Comment:
---------------------------------------
5L File: D:\cdrips\test\cbr 192 eac filt.wav
5L Rating: 4.3
5L Comment:
---------------------------------------
6L File: D:\cdrips\test\cbr 192 no filt.wav
6L Rating: 4.6
6L Comment:
---------------------------------------


So, summing up, this test indicates to me that MP3, even if encoded by the latest LAME engine, is definitely not transparent up to and including 256Kbit.  Not once was an MP3 mistakenly rated above the reference CD-Audio sample.  The source material is very simple (solo bongo) which one would assume would be an easy task for an encoder.  The standard preset fares quite well, but at the cost of file size (an 8% premium over 192Kbit CBR).  Filtering does not benefit the sound, as shown by the difference in rating between the 5th and 6th samples, which are otherwise the same.  To further investigate the comparison between filtered and non-filtered mp3, I made up a second ABC/HR test, this time comparing samples 5 and 6, the non-filtered sample as the reference, and the filtered sample as the comparator, 8 times over.

ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = D:\cdrips\test\cbr 192 eac filt.wav
2L = D:\cdrips\test\cbr 192 eac filt.wav
3L = D:\cdrips\test\cbr 192 eac filt.wav
4R = D:\cdrips\test\cbr 192 eac filt.wav
5R = D:\cdrips\test\cbr 192 eac filt.wav
6L = D:\cdrips\test\cbr 192 eac filt.wav
7R = D:\cdrips\test\cbr 192 eac filt.wav
8L = D:\cdrips\test\cbr 192 eac filt.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: D:\cdrips\test\cbr 192 eac filt.wav
1L Rating: 4.0
1L Comment:
---------------------------------------
2L File: D:\cdrips\test\cbr 192 eac filt.wav
2L Rating: 4.0
2L Comment:
---------------------------------------
3L File: D:\cdrips\test\cbr 192 eac filt.wav
3L Rating: 4.0
3L Comment:
---------------------------------------
4R File: D:\cdrips\test\cbr 192 eac filt.wav
4R Rating: 4.0
4R Comment:
---------------------------------------
5L File: D:\cdrips\test\cbr 192 no filt.wav
5L Rating: 4.0
5L Comment:
---------------------------------------
6L File: D:\cdrips\test\cbr 192 eac filt.wav
6L Rating: 4.0
6L Comment:
---------------------------------------
7R File: D:\cdrips\test\cbr 192 eac filt.wav
7R Rating: 4.0
7R Comment:
---------------------------------------
8L File: D:\cdrips\test\cbr 192 eac filt.wav
8L Rating: 4.0
8L Comment:
---------------------------------------

Only once did the filtered sample fare better than the untouched, unfiltered sound.  So there you have it.

My LAME tests
Reply #1
Yeah, MP3 certainly has its faults, which is why it's not recommended unless you need hardware compatability.

I did notice, however, that you used an alpha of LAME; the recommended version here is 3.90.2.  You also didn't try alt-preset extreme, though that may not make much of a difference.  Could you try it again with 3.90.2 or 3.92?

A solo bongo is actually quite hard for any encoder; the sound has quick transients and, worse still, has no masking noises around it. 

Try your tests again with MPC, and see if it is transparent for you.

  • KikeG
  • [*][*][*][*][*]
  • Developer
My LAME tests
Reply #2
For the sake of rigorousness (statistical validity), I advise to try to ABX at least the highest quality encodes.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
My LAME tests
Reply #3
Quote
For the sake of rigorousness (statistical validity), I advise to try to ABX at least the highest quality encodes.


He implicitly ABX'ed it 6/6 globally. But I suspect you mean doing an ABX of the highest scoring sample (CBR 256) seperately?
  • Last Edit: 19 February, 2003, 03:28:22 AM by Garf

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
My LAME tests
Reply #4
Quote
so as you can imagine, the music is fast with great transient attack.


You contradict yourself with this...

Quote
The source material is very simple (solo bongo)


..and this. Sharp transients are a known worst case scenario for MP3.

Quote
So, summing up, this test indicates to me that MP3, even if encoded by the latest LAME engine, is definitely not transparent up to and including 256Kbit.


Not for all music out there in existence, no. No lossy codec can ever always be transparent, and I doubt you'll find anyone making that claim.

We are generally not talking about absolute transparency since it's an unattainable dream. But LAME APS for example will generate MP3's that are indistinguishable from the CD for almost anything out there. That doesn't mean there aren't exceptions.

I wonder what your point was? MP3 is not perfect? Well, no.
  • Last Edit: 19 February, 2003, 03:29:46 AM by Garf

  • KikeG
  • [*][*][*][*][*]
  • Developer
My LAME tests
Reply #5
Quote
Quote
For the sake of rigorousness (statistical validity), I advise to try to ABX at least the highest quality encodes.


He implicitly ABX'ed it 6/6 globally. But I suspect you mean doing an ABX of the highest scoring sample (CBR 256) seperately?

yes, I'd try to ABX with p<5% (better p<1%) for individual encodes vs. original, at least the better scored ones (CBR256, CBR192) and APS. Maybe the better encodes were correctly identified by chance. Also, in my experience CBR256 in Lame 3.90.2 is always worse than APS in pre-echo, but maybe this changes in Lame 3.94?

Another round of abc/hr to check consistency of results would be good too.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
My LAME tests
Reply #6
Quote
Also, in my experience CBR256 in Lame 3.90.2 is always worse than APS in pre-echo, but maybe this changes in Lame 3.94?

I don't know - it would be good to use the latest trusted encoder anyway (3.90.2), you can get flawed results very easily with alpha versions.
  • Last Edit: 19 February, 2003, 05:30:54 AM by Garf

  • LordofStars
  • [*][*][*][*]
My LAME tests
Reply #7
Some people have found the preecho of standard from 3.94 to be worse than  3.90.2
r3mix zealot.