Skip to main content

Topic: Personal Listening Test of AAC and LAME encoders (old test, translated (Read 11251 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC and LAME encoders (old test, translated
Abstract:
This is a translation of an old test, finished in October 2011, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between libfaac(FAAC, abr), Apple AAC-LC(cvbr), NeroAACEnc(abr), and LAME, at 64kbps, 96kbps, 128kbps and 192kbps.
 
Encoders:
libfaac (FAAC) 1.28
qaac 0.94
nero 1.5.4.0
LAME 3.99

Settings:
64kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -ab 64k out.mp4
 qaac --cvbr 64 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 64000 -of out.mp4
 lame399 -h --preset 64 in.wav out.mp3
96kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -ab 96k out.mp4
 qaac --cvbr 96 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 96000 -of out.mp4
 lame399 -h --preset 96 in.wav out.mp3

128kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -aq 95 out.mp4
 qaac_0.94 --cvbr 128 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 128000 -of out.mp4
 lame399 -h -V5.6 in.wav out.mp3
192kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -aq 155 out.mp4
 qaac_0.94 --cvbr 192 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 192000 -of out.mp4
 lame399 -h -V2 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1

Results






Conclusions & Observations:
qaac(AAC) > NeroAACEnc(AAC) > LAME(MP3) > libfaac(AAC)

Apple AAC had the best sound quality. libfaac was the worst among the 4 encoder tested.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              119          36.04
Testers (blocks)    14          2.24
Codecs eval'd        7          29.71    4.24  101.63  0.00E+000
Error              98          4.09    0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.148

Means:

qaac96  nero96  lame96  faac96  nero64  qaac64  lame64  faac64
  3.61    2.95    2.78    2.43    2.40    2.37    2.30    1.83

---------------------------- p-value Matrix ---------------------------

         nero96  lame96  faac96  nero64  qaac64  lame64  faac64
qaac96  0.000*  0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
nero96            0.020*  0.000*  0.000*  0.000*  0.000*  0.000*
lame96                    0.000*  0.000*  0.000*  0.000*  0.000*
faac96                              0.755    0.449    0.085    0.000*
nero64                                      0.656    0.156    0.000*
qaac64                                                0.328    0.000*
lame64                                                        0.000*
-----------------------------------------------------------------------

qaac96 is better than nero96, lame96, faac96, nero64, qaac64, lame64, faac64
nero96 is better than lame96, faac96, nero64, qaac64, lame64, faac64
lame96 is better than faac96, nero64, qaac64, lame64, faac64
faac96 is better than faac64
nero64 is better than faac64
qaac64 is better than faac64
lame64 is better than faac64
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              119          49.21
Testers (blocks)    14          7.98
Codecs eval'd        7          33.08    4.73  56.76  0.00E+000
Error              98          8.16    0.08
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.209

Means:

qaac192  nero192  qaac128  lame192  faac192  nero128  faac128  lame128
  4.82    4.76    4.15    4.14    4.03    3.79    3.37    3.30

---------------------------- p-value Matrix ---------------------------

         nero192  qaac128  lame192  faac192  nero128  faac128  lame128
qaac192  0.614    0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
nero192          0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
qaac128                    0.900    0.245    0.001*  0.000*  0.000*
lame192                            0.299    0.002*  0.000*  0.000*
faac192                                      0.029*  0.000*  0.000*
nero128                                              0.000*  0.000*
faac128                                                        0.508
-----------------------------------------------------------------------

qaac192 is better than qaac128, lame192, faac192, nero128, faac128, lame128
nero192 is better than qaac128, lame192, faac192, nero128, faac128, lame128
qaac128 is better than nero128, faac128, lame128
lame192 is better than nero128, faac128, lame128
faac192 is better than nero128, faac128, lame128
nero128 is better than faac128, lame128

Raw data:
Code: [Select]
% AAC/MP3 64kbps+96kbps ABC/HR test score.
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
%64kbps 64kbps 64kbps 64kbps 96kbps 96kbps 96kbps 96kbps
%features 7 AAC-LC AAC-LC AAC-LC MP3 AAC-LC AAC-LC AAC-LC MP3
faac64 qaac64 nero64 lame64 faac96 qaac96 nero96 lame96
1.85 2.15 1.6 1.95 2.2 3.35 2.6 2.45
1.7 2.3 2.4 2.4 2.25 3.65 3.15 2.75
1.75 2.5 2.4 2.2 2.55 3.75 2.8 2.8
1.75 2.25 2.25 2.2 2.25 3.1 2.85 2.5
1.85 2.4 2.3 2.4 2.8 3.2 2.85 2.65
1.9 2.35 2.6 2.2 2.45 4.55 3.1 2.8
1.75 2.25 2.5 2.2 2.5 3.5 2.8 2.45
1.75 2.4 2.35 2.3 2.4 3.55 3.05 2.85
1.8 2.45 2.55 2.3 2.25 3.6 2.9 2.65
1.85 2.2 2.45 2.3 2.5 3.8 2.9 2.75
1.8 2.4 2.3 2.3 2.45 3.15 2.75 2.75
1.95 2.6 2.65 2.45 2.75 3.35 3 2.8
1.95 2.55 2.65 2.4 2.4 3.65 3.25 3.05
1.9 2.25 2.65 2.35 2.45 3.35 3.3 2.85
1.9 2.5 2.4 2.5 2.2 4.6 3 3.55
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart_to_Heart Pops
%samples Tom's_Diner Vocal
Code: [Select]
% AAC/MP3 128kbps+192kbps ABC/HR test score.
%features 6 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
%features 7 AAC-LC AAC-LC AAC-LC MP3 AAC-LC AAC-LC AAC-LC MP3
faac128 qaac128 nero128 lame128 faac192 qaac192 nero192 lame192
3 4.45 3.8 3.45 3.6 4.8 5 4.7
3.35 4.25 3.95 2.95 3.9 5 5 3.65
3.45 4.5 4.1 3.4 4 5 4.7 4.4
3.05 4.15 3.8 3.05 4.05 4.45 4.6 4.05
3 3.35 3.4 3.2 3.65 4.8 5 3.85
3.8 5 5 4.45 5 5 5 5
3.9 4.8 4.75 3.5 4.4 5 5 4.2
3.1 4 3.7 3.25 3.9 5 5 4.15
3.7 3.95 3.2 3.5 3.95 4.8 3.95 3.9
3.05 3.95 3.65 3.1 3.5 5 5 3.95
3.95 3.75 3.3 3.3 4.4 4.35 4 4.1
3.3 3.8 3.35 3.25 3.85 4.65 4.2 3.9
3.15 4.1 3.6 3.05 4 4.7 5 4.1
3.7 4.05 3.7 3.25 4.1 5 5 4.3
3.05 4.15 3.6 2.8 4.1 4.7 5 3.8
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart_to_Heart Pops
%samples Tom's_Diner Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

  • IgorC
  • [*][*][*][*][*]
Personal Listening Test of AAC and LAME encoders (old test, translated
Reply #1
Kamedo2,

Yes, that is the test!

It's interesting to see that Nero and Apple are on par at 192k. However an average score describes only a part of a whole picture.
The minimum score (worst case) at 192k: Nero - 4.0, Apple - 4.4.

Have You tried Fraunhofer AAC encoder from the last Winamp?

  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC and LAME encoders (old test, translated
Reply #2
It's interesting to see that Nero and Apple are on par at 192k. However an average score describes only a part of a whole picture.
The minimum score (worst case) at 192k: Nero - 4.0, Apple - 4.4.

Have You tried Fraunhofer AAC encoder from the last Winamp?

Yes, sometimes it makes sense to look at the minimum score and a distribution.
Apple beats Nero at 128kbps, and it's natural to think Apple also beats Nero at 192kbps.
Insignificance doesn't mean there are no differences at all, and actual average can be anywhere within the CI bar.
(Although this claim is hard to verify, because both encoders are very close to the original and offer very good qualities.)

I haven't tested the Fraunhofer AAC. Those on the web are the only listening test results I have. I don't test anything and secretly withhold the results.

  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC and LAME encoders (old test, translated
Reply #3
The results of FAAC, Nero, and LAME are still relevant today. FAAC and Nero haven't changed at all since 2010, and as for LAME, a tiny bitrate bloat issue was fixed but no other quality improvement.

  • IgorC
  • [*][*][*][*][*]
Personal Listening Test of AAC and LAME encoders (old test, translated
Reply #4
Ohm, I wanted to ask You if You have ever tried FhG AAC winamp encoder or consider to .
While its performance is well known at 96k,  it's not so at 128-192k.

  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC and LAME encoders (old test, translated
Reply #5
Ohm, I wanted to ask You if You have ever tried FhG AAC winamp encoder or consider to .
While its performance is well known at 96k,  it's not so at 128-192k.

No, I have never tried FhG. I don't have a winamp.