HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Kamedo2 on 2013-10-02 18:13:57

Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Kamedo2 on 2013-10-02 18:13:57
Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003 (http://www.hydrogenaudio.org/forums/index.php?showtopic=98003)

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results
(http://i40.tinypic.com/29olvt3.png)
(http://i39.tinypic.com/2mx2yk8.png)

Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              119          88.91
Testers (blocks)    14          8.12
Codecs eval'd        7          65.64    9.38  60.70  0.00E+000
Error              98          15.14    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.285

Means:

qaac192  wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
  4.78    4.47    4.05    4.00    3.73    2.85    2.83    2.74

---------------------------- p-value Matrix ---------------------------

        wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
qaac192  0.035*  0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
wmaen192          0.004*  0.001*  0.000*  0.000*  0.000*  0.000*
qaac128                    0.694    0.025*  0.000*  0.000*  0.000*
lame192                            0.063    0.000*  0.000*  0.000*
wmaen128                                    0.000*  0.000*  0.000*
lame128                                              0.889    0.445
voaac192                                                      0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128

Raw data:
Code: [Select]
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: cd.vs.mp3 on 2014-06-01 16:57:19
Abstract:
This is a translation of an old test, finished in January 2012, and translated from Japanese to English in October 2013 for convenience.
Blind Comparison between Apple AAC-LC, Vo-aacenc(VisualOn), Microsoft WMA(standard), LAME CBR, at 128kbps and 192kbps.

Encoders:
qaac 1.18
FFmpeg r.36030
WMAEncode 0.2.9
LAME 3.98.4

Settings:
qaac --cvbr 128 -o out.mp4 in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 128k out.mp4
WMAEncode64 in.wav out.wma --bitrate 128
lame -q 0 -b 128 in.wav out.mp3

qaac --cvbr 192 -o out.mp in.wav
ffmpeg -y -i in.wav -acodec libvo_aacenc -ab 192k out.mp
WMAEncode64 in.wav out.wma --bitrate 192
lame -q 0 -b 192 in.wav out.mp3


Samples:
15 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003 (http://www.hydrogenaudio.org/forums/index.php?showtopic=98003)

Hardwares:
ABC/HR for Java 0.53a + MHP-A1(1st), SE-DIR800CII(2nd), took average between two.

Results
(http://i40.tinypic.com/29olvt3.png)
(http://i39.tinypic.com/2mx2yk8.png)

Conclusions & Observations:
The Apple AAC had the best quality among the 4 encoders tested. The Apple AAC was clearly superior than the Microsoft WMA. LAME MP3 was worse than the WMA, and VisualOn vo-aacenc was the worst encoder.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              119          88.91
Testers (blocks)    14          8.12
Codecs eval'd        7          65.64    9.38  60.70  0.00E+000
Error              98          15.14    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.285

Means:

qaac192  wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
  4.78    4.47    4.05    4.00    3.73    2.85    2.83    2.74

---------------------------- p-value Matrix ---------------------------

        wmaen192 qaac128  lame192  wmaen128 lame128  voaac192 voaac128
qaac192  0.035*  0.000*  0.000*  0.000*  0.000*  0.000*  0.000*
wmaen192          0.004*  0.001*  0.000*  0.000*  0.000*  0.000*
qaac128                    0.694    0.025*  0.000*  0.000*  0.000*
lame192                            0.063    0.000*  0.000*  0.000*
wmaen128                                    0.000*  0.000*  0.000*
lame128                                              0.889    0.445
voaac192                                                      0.532
-----------------------------------------------------------------------

qaac192 is better than wmaen192, qaac128, lame192, wmaen128, lame128, voaac192, voaac128
wmaen192 is better than qaac128, lame192, wmaen128, lame128, voaac192, voaac128
qaac128 is better than wmaen128, lame128, voaac192, voaac128
lame192 is better than lame128, voaac192, voaac128
wmaen128 is better than lame128, voaac192, voaac128

Raw data:
Code: [Select]
% AAC/WMA/MP3 128/192kbps ABC/HR score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
qaac128 voaac128 wmaen128 lame128 qaac192 voaac192 wmaen192 lame192
%qaac vo-aacenc wmaencode lame qaac vo-aacenc wmaencode lame
%features 6 AAC AAC WMA MP3 AAC AAC WMA MP3
%features 7 128kbps 128kbps 128kbps 128kbps 192kbps 192kbps 192kbps 192kbps
3.850 2.600 3.300 2.100 5.000 2.750 3.650 3.300
4.350 3.400 4.000 2.900 4.600 3.850 5.000 3.700
4.550 2.700 4.600 2.350 5.000 2.900 5.000 4.500
3.800 2.800 3.100 2.300 4.650 3.150 4.200 3.950
3.400 2.450 3.350 2.450 5.000 2.400 4.050 3.350
4.550 2.750 5.000 3.350 5.000 2.800 4.650 4.650
5.000 2.350 3.900 2.750 5.000 2.350 5.000 3.900
4.100 2.300 3.600 3.100 5.000 2.300 4.250 4.450
3.800 2.450 2.650 2.950 4.450 2.550 3.500 3.200
3.950 2.300 4.350 2.850 5.000 2.300 4.700 5.000
3.650 2.850 3.600 3.300 4.250 3.200 4.600 4.100
3.750 2.600 3.700 2.800 4.150 2.650 4.600 3.950
3.800 3.800 3.700 3.450 5.000 2.850 5.000 3.950
3.700 3.050 3.500 2.800 4.600 3.450 4.450 3.850
4.550 2.700 3.550 3.300 5.000 2.950 4.450 4.100
%samples 41_30sec hihats
%samples finalfantasy cemb
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre orch
%samples mybloodrusts guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Vocal
It's not strange that some scores get 0.05 scale, as I tested twice per each sample.

Excellent work! Congratulations!
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: eahm on 2014-06-01 17:10:08
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: TomasPin on 2014-06-01 22:28:48
Is FFmpeg's AAC based on FDK-AAC?

Nope. It's another, more experimental encoder called "libvo_aacenc". FFmpeg does support FDK though I believe you have to compile it yourself to use it.

(I'm still waiting for the stable version of Handbrake (http://handbrake.fr/) to add support for FDK btw... But that's another topic.)
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Kamedo2 on 2014-06-02 01:41:12
Is FFmpeg's AAC based on FDK-AAC?

FFmpeg's native AAC, vo-aacenc(VisualOn AAC encoder from Android), FDK-AAC(A standalone library of the Fraunhofer FDK AAC code from Android);
These three are different things, although all are usable from FFmpeg.
vo-aacenc is extremely bad like shown in this test, FDK-AAC is a state-of-the-art encoder(its quality is comparable to Apple AAC=qaac), FFmpeg's native AAC is slightly better than vo-aacenc.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: saratoga on 2014-06-02 01:44:13
I'm curious which WMA encoder revision was used with WMAEncode.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: cd.vs.mp3 on 2014-06-02 06:29:04
cd.vs.mp3, remove the full useless quote please.

Thanks Kamedo2 for testing, WMA is clearly still really really good (I don't know why everyone thinks it always sucked?) and surprisingly FFmpeg's AAC is really bad. Is FFmpeg's AAC based on FDK-AAC?


I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Robertina on 2014-06-02 07:14:39
I'm new in the forum. I can't find the edit button to remove the unnecessary quote. Sorry. I only see "Quote" and "Reply". It is strange, because I do see it in this post.

Post edits are possible within 60 minutes.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Kamedo2 on 2014-06-05 14:12:54
I'm curious which WMA encoder revision was used with WMAEncode.

Code: [Select]
General
Complete name                  : ********************************\11ff_wmae029_192k.wma
Format                         : Windows Media
File size                      : 730 KiB
Duration                       : 29s 953ms
Overall bit rate mode          : Constant
Overall bit rate               : 200 Kbps
Maximum Overall bit rate       : 193 Kbps
Encoded date                   : UTC 2012-01-03 17:30:15.121

Audio
ID                             : 1
Format                         : WMA
Format version                 : Version 2
Codec ID                       : 161
Codec ID/Info                  : Windows Media Audio
Description of the codec       : Windows Media Audio 9.2 - 192 kbps, 44 kHz, stereo 1-pass CBR
Duration                       : 29s 952ms
Bit rate mode                  : Constant
Bit rate                       : 192 Kbps
Channel(s)                     : 2 channels
Sampling rate                  : 44.1 KHz
Bit depth                      : 16 bits
Stream size                    : 702 KiB (96%)
Language                       : Japanese


Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: eahm on 2014-06-05 15:22:06
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Kamedo2 on 2014-06-05 17:21:26
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Mach-X on 2014-06-14 13:24:43
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: eahm on 2014-06-14 15:55:42
It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.

What? Crazy, I had no idea.
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Speechless on 2014-08-12 01:38:58
Kamedo2, no particular reason to ask this but why did you try WMA Standard instead of WMA Pro?

I like the backwards hardware compatibility of WMA Standard.


It wouldn't be of much use since there's next to zero hardware support. Even Windows Phone doesn't support WMA Pro.


what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx (http://msdn.microsoft.com/en-us/library/windows/apps/ff462087(v=vs.105).aspx)
Title: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: LithosZA on 2014-08-12 06:47:55
Quote
what an ignorant... Windows Phone Support WMA and WMA Pro!

http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx (http://msdn.microsoft.com/en-us/library/wi...(v=vs.105).aspx)


Windows Phone does support it, but the market share of Windows Phone is really low at the moment:
http://bgr.com/2014/07/01/windows-phone-market-share-3/ (http://bgr.com/2014/07/01/windows-phone-market-share-3/)
Title: Re: Personal Listening Test of AAC, WMA, and MP3 encoders (old test, trans
Post by: Kamedo2 on 2020-09-28 20:05:18
Oops, files deleted.

(https://listening-test.coresv.net/img2/wma1c.png)
(https://listening-test.coresv.net/img2/wma2c.png)
SimplePortal 1.0.0 RC1 © 2008-2021