Skip to main content

Topic: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans (Read 13548 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              119          88.91
Testers (blocks)    14          8.12
Codecs eval'd        7          65.64    9.38  60.70  0.00E+000
Error              98          15.14    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.285

Means:

qaac192  wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
  4.78    4.47    4.05    4.00    3.73    2.85    2.83    2.74

---------------------------- p-value Matrix ---------------------------

         wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
qaac192  0.035*  0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
wmaen192          0.004*  0.001*  0.000*  0.000*  0.000*  0.000*
qaac128                    0.694    0.025*  0.000*  0.000*  0.000*
lame192                            0.063    0.000*  0.000*  0.000*
wmaen128                                    0.000*  0.000*  0.000*
lame128                                              0.889    0.445
voaac192                                                      0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128

Raw data:
Code: [Select]
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

  • cd.vs.mp3
  • [*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #1
Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results



Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              119          88.91
Testers (blocks)    14          8.12
Codecs eval'd        7          65.64    9.38  60.70  0.00E+000
Error              98          15.14    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.285

Means:

qaac192  wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
  4.78    4.47    4.05    4.00    3.73    2.85    2.83    2.74

---------------------------- p-value Matrix ---------------------------

         wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
qaac192  0.035*  0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
wmaen192          0.004*  0.001*  0.000*  0.000*  0.000*  0.000*
qaac128                    0.694    0.025*  0.000*  0.000*  0.000*
lame192                            0.063    0.000*  0.000*  0.000*
wmaen128                                    0.000*  0.000*  0.000*
lame128                                              0.889    0.445
voaac192                                                      0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128

Raw data:
Code: [Select]
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

Excellent work! Congratulations!
  • Last Edit: 01 June, 2014, 12:00:59 PM by cd.vs.mp3

  • eahm
  • [*][*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #2
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?
  • Last Edit: 01 June, 2014, 12:10:59 PM by eahm

  • TomasPin
  • [*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #3
Is FFmpeg's AAC based on FDK-AAC?

Nope. It's another, more experimental encoder called "libvo_aacenc". FFmpeg does support FDK though I believe you have to compile it yourself to use it.

(I'm still waiting for the stable version of Handbrake to add support for FDK btw... But that's another topic.)
A man and his music: http://tubular.net/

  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #4
Is FFmpeg's AAC based on FDK-AAC?

FFmpeg's native AAC, vo-aacenc(VisualOn AAC encoder from Android), FDK-AAC(A standalone library of the Fraunhofer FDK AAC code from Android);
These three are different things, although all are usable from FFmpeg.
vo-aacenc is extremely bad like shown in this test, FDK-AAC is a state-of-the-art encoder(its quality is comparable to Apple AAC=qaac), FFmpeg's native AAC is slightly better than vo-aacenc.

  • saratoga
  • [*][*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #5
I'm curious which WMA encoder revision was used with WMAEncode.

  • cd.vs.mp3
  • [*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #6
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?


I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.
  • Last Edit: 02 June, 2014, 01:30:28 AM by cd.vs.mp3

  • Robertina
  • [*][*][*][*][*]
  • Banned
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #7
I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.

Post edits are possible within 60 minutes.
This is HA. Not the Jerry Springer Show.

  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #8
I'm curious which WMA encoder revision was used with WMAEncode.

Code: [Select]
General
Complete name                  : ********************************\11ff_wmae029_192k.wma
Format                         : Windows Media
File size                      : 730 KiB
Duration                       : 29s 953ms
Overall bit rate mode          : Constant
Overall bit rate               : 200 Kbps
Maximum Overall bit rate       : 193 Kbps
Encoded date                   : UTC 2012-01-03 17:30:15.121

Audio
ID                             : 1
Format                         : WMA
Format version                 : Version 2
Codec ID                       : 161
Codec ID/Info                  : Windows Media Audio
Description of the codec       : Windows Media Audio 9.2 - 192 kbps, 44 kHz, stereo 1-pass CBR
Duration                       : 29s 952ms
Bit rate mode                  : Constant
Bit rate                       : 192 Kbps
Channel(s)                     : 2 channels
Sampling rate                  : 44.1 KHz
Bit depth                      : 16 bits
Stream size                    : 702 KiB (96%)
Language                       : Japanese



  • eahm
  • [*][*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #9
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #10
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.

  • Mach-X
  • [*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #11
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.

  • eahm
  • [*][*][*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #12
It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.

What? Crazy, I had no idea.

Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #13
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.


what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx

  • LithosZA
  • [*][*][*]
Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Reply #14
Quote
what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx


Windows Phone does support it, but the market share of Windows Phone is really low at the moment:
http://bgr.com/2014/07/01/windows-phone-market-share-3/
  • Last Edit: 12 August, 2014, 01:49:54 AM by LithosZA