Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of AAC and LAME encoders (old test, translated (Read 20554 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of AAC and LAME encoders (old test, translated

Abstract:
This is a translation of an old test, finished in October 2011, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between libfaac(FAAC, abr), Apple AAC-LC(cvbr), NeroAACEnc(abr), and LAME, at 64kbps, 96kbps, 128kbps and 192kbps.
 
Encoders:
libfaac (FAAC) 1.28
qaac 0.94
nero 1.5.4.0
LAME 3.99

Settings:
64kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -ab 64k out.mp4
 qaac --cvbr 64 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 64000 -of out.mp4
 lame399 -h --preset 64 in.wav out.mp3
96kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -ab 96k out.mp4
 qaac --cvbr 96 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 96000 -of out.mp4
 lame399 -h --preset 96 in.wav out.mp3

128kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -aq 95 out.mp4
 qaac_0.94 --cvbr 128 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 128000 -of out.mp4
 lame399 -h -V5.6 in.wav out.mp3
192kbps
 ffmpeg18607 -y -i in.wav -acodec libfaac -aq 155 out.mp4
 qaac_0.94 --cvbr 192 -o out.mp4 in.wav
 neroAacEnc -if in.wav -lc -br 192000 -of out.mp4
 lame399 -h -V2 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1

Results






Conclusions & Observations:
qaac(AAC) > NeroAACEnc(AAC) > LAME(MP3) > libfaac(AAC)

Apple AAC had the best sound quality. libfaac was the worst among the 4 encoder tested.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              119          36.04
Testers (blocks)    14           2.24
Codecs eval'd        7          29.71    4.24   101.63  0.00E+000
Error               98           4.09    0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.148

Means:

qaac96   nero96   lame96   faac96   nero64   qaac64   lame64   faac64
  3.61     2.95     2.78     2.43     2.40     2.37     2.30     1.83

---------------------------- p-value Matrix ---------------------------

         nero96   lame96   faac96   nero64   qaac64   lame64   faac64
qaac96   0.000*   0.000*   0.000*   0.000*   0.000*   0.000*   0.000*
nero96            0.020*   0.000*   0.000*   0.000*   0.000*   0.000*
lame96                     0.000*   0.000*   0.000*   0.000*   0.000*
faac96                              0.755    0.449    0.085    0.000*
nero64                                       0.656    0.156    0.000*
qaac64                                                0.328    0.000*
lame64                                                         0.000*
-----------------------------------------------------------------------

qaac96 is better than nero96, lame96, faac96, nero64, qaac64, lame64, faac64
nero96 is better than lame96, faac96, nero64, qaac64, lame64, faac64
lame96 is better than faac96, nero64, qaac64, lame64, faac64
faac96 is better than faac64
nero64 is better than faac64
qaac64 is better than faac64
lame64 is better than faac64
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              119          49.21
Testers (blocks)    14           7.98
Codecs eval'd        7          33.08    4.73   56.76  0.00E+000
Error               98           8.16    0.08
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.209

Means:

qaac192  nero192  qaac128  lame192  faac192  nero128  faac128  lame128
  4.82     4.76     4.15     4.14     4.03     3.79     3.37     3.30

---------------------------- p-value Matrix ---------------------------

         nero192  qaac128  lame192  faac192  nero128  faac128  lame128
qaac192  0.614    0.000*   0.000*   0.000*   0.000*   0.000*   0.000*
nero192           0.000*   0.000*   0.000*   0.000*   0.000*   0.000*
qaac128                    0.900    0.245    0.001*   0.000*   0.000*
lame192                             0.299    0.002*   0.000*   0.000*
faac192                                      0.029*   0.000*   0.000*
nero128                                               0.000*   0.000*
faac128                                                        0.508
-----------------------------------------------------------------------

qaac192 is better than qaac128, lame192, faac192, nero128, faac128, lame128
nero192 is better than qaac128, lame192, faac192, nero128, faac128, lame128
qaac128 is better than nero128, faac128, lame128
lame192 is better than nero128, faac128, lame128
faac192 is better than nero128, faac128, lame128
nero128 is better than faac128, lame128

Raw data:
Code: [Select]
% AAC/MP3 64kbps+96kbps ABC/HR test score.
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
%64kbps 64kbps 64kbps 64kbps 96kbps 96kbps 96kbps 96kbps
%features 7 AAC-LC AAC-LC AAC-LC MP3 AAC-LC AAC-LC AAC-LC MP3
faac64	qaac64	nero64	lame64	faac96	qaac96	nero96	lame96
1.85	2.15	1.6	1.95	2.2	3.35	2.6	2.45	
1.7	2.3	2.4	2.4	2.25	3.65	3.15	2.75	
1.75	2.5	2.4	2.2	2.55	3.75	2.8	2.8	
1.75	2.25	2.25	2.2	2.25	3.1	2.85	2.5	
1.85	2.4	2.3	2.4	2.8	3.2	2.85	2.65	
1.9	2.35	2.6	2.2	2.45	4.55	3.1	2.8	
1.75	2.25	2.5	2.2	2.5	3.5	2.8	2.45	
1.75	2.4	2.35	2.3	2.4	3.55	3.05	2.85	
1.8	2.45	2.55	2.3	2.25	3.6	2.9	2.65	
1.85	2.2	2.45	2.3	2.5	3.8	2.9	2.75	
1.8	2.4	2.3	2.3	2.45	3.15	2.75	2.75	
1.95	2.6	2.65	2.45	2.75	3.35	3	2.8	
1.95	2.55	2.65	2.4	2.4	3.65	3.25	3.05	
1.9	2.25	2.65	2.35	2.45	3.35	3.3	2.85	
1.9	2.5	2.4	2.5	2.2	4.6	3	3.55	
%samples	41_30sec	hihats
%samples	finalfantasy	cemb
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	orch
%samples	mybloodrusts	guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart_to_Heart	Pops
%samples	Tom's_Diner	Vocal
Code: [Select]
% AAC/MP3 128kbps+192kbps ABC/HR test score.
%features 6 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
%features 7 AAC-LC AAC-LC AAC-LC MP3 AAC-LC AAC-LC AAC-LC MP3
faac128	qaac128	nero128	lame128	faac192	qaac192	nero192	lame192
3	4.45	3.8	3.45	3.6	4.8	5	4.7	
3.35	4.25	3.95	2.95	3.9	5	5	3.65	
3.45	4.5	4.1	3.4	4	5	4.7	4.4	
3.05	4.15	3.8	3.05	4.05	4.45	4.6	4.05	
3	3.35	3.4	3.2	3.65	4.8	5	3.85	
3.8	5	5	4.45	5	5	5	5	
3.9	4.8	4.75	3.5	4.4	5	5	4.2	
3.1	4	3.7	3.25	3.9	5	5	4.15	
3.7	3.95	3.2	3.5	3.95	4.8	3.95	3.9	
3.05	3.95	3.65	3.1	3.5	5	5	3.95	
3.95	3.75	3.3	3.3	4.4	4.35	4	4.1	
3.3	3.8	3.35	3.25	3.85	4.65	4.2	3.9	
3.15	4.1	3.6	3.05	4	4.7	5	4.1	
3.7	4.05	3.7	3.25	4.1	5	5	4.3	
3.05	4.15	3.6	2.8	4.1	4.7	5	3.8
%samples	41_30sec	hihats
%samples	finalfantasy	cemb
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	orch
%samples	mybloodrusts	guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart_to_Heart	Pops
%samples	Tom's_Diner	Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

Personal Listening Test of AAC and LAME encoders (old test, translated

Reply #1 –
Kamedo2,

Yes, that is the test!

It's interesting to see that Nero and Apple are on par at 192k. However an average score describes only a part of a whole picture.
The minimum score (worst case) at 192k: Nero - 4.0, Apple - 4.4.

Have You tried Fraunhofer AAC encoder from the last Winamp?

Personal Listening Test of AAC and LAME encoders (old test, translated

Reply #2 –
Quote from: IgorC on
It's interesting to see that Nero and Apple are on par at 192k. However an average score describes only a part of a whole picture.
The minimum score (worst case) at 192k: Nero - 4.0, Apple - 4.4.

Have You tried Fraunhofer AAC encoder from the last Winamp?

Yes, sometimes it makes sense to look at the minimum score and a distribution.
Apple beats Nero at 128kbps, and it's natural to think Apple also beats Nero at 192kbps.
Insignificance doesn't mean there are no differences at all, and actual average can be anywhere within the CI bar.
(Although this claim is hard to verify, because both encoders are very close to the original and offer very good qualities.)

I haven't tested the Fraunhofer AAC. Those on the web are the only listening test results I have. I don't test anything and secretly withhold the results.

Personal Listening Test of AAC and LAME encoders (old test, translated

Reply #3 –
The results of FAAC, Nero, and LAME are still relevant today. FAAC and Nero haven't changed at all since 2010, and as for LAME, a tiny bitrate bloat issue was fixed but no other quality improvement.

Personal Listening Test of AAC and LAME encoders (old test, translated

Reply #4 –
Ohm, I wanted to ask You if You have ever tried FhG AAC winamp encoder or consider to .
While its performance is well known at 96k,  it's not so at 128-192k.

Personal Listening Test of AAC and LAME encoders (old test, translated

Reply #5 –
Quote from: IgorC on
Ohm, I wanted to ask You if You have ever tried FhG AAC winamp encoder or consider to .
While its performance is well known at 96k,  it's not so at 128-192k.

No, I have never tried FhG. I don't have a winamp.

Re: Personal Listening Test of AAC and LAME encoders (old test, translated

Reply #6 –
Oops, images deleted.