Skip to main content

Topic: Personal Listening Test of AAC encoders available from FFmpeg (Read 12912 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Kamedo2
  • [*][*][*][*]
Personal Listening Test of AAC encoders available from FFmpeg
Abstract:
Blind sound quality comparison between the FFmpeg's native AAC encoder (-c:a aac, from the 2016/01/17 version) and the Fraunhofer FDK AAC encoder library(-c:a libfdk_aac), at 64 kbps, 96 kbps and 128kbps.

Encoders:
FFmpeg N-77891-g757248e (2016/01/17 version, it was latest at the time but TNS algorithm was fixed since then.)
fdk-aac-0.1.4 (still the latest as of 2016/01/25)

Settings:
ffmpeg77891 -i in.wav -c:a aac -b:a 64k out.mp4
ffmpeg77891 -i in.wav -c:a aac -b:a 96k out.mp4
ffmpeg77891 -i in.wav -c:a aac -b:a 128k out.mp4
ffmpeg77891 -i in.wav -c:a libfdk_aac -b:a 64k out.mp4
ffmpeg77891 -i in.wav -c:a libfdk_aac -b:a 96k out.mp4
ffmpeg77891 -i in.wav -c:a libfdk_aac -b:a 128k out.mp4

Samples:
Total 15 samples from my corpus

Hardwares:
Sony PSP-3000 + RP-HJE150.

Results:




Conclusions & Observations:
The FDK-AAC was the clear winner. Compared to 2016/01/17 version of the FFmpeg's native encoder, Fraunhofer's FDK-AAC library delivered the same sound quality at 32kbps reduced bitrates.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               89          55.70
Testers (blocks)    14           1.91
Codecs eval'd        5          47.62    9.52   107.92  0.00E+000
Error               70           6.18    0.09
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.216

Means:

fdk128   ffa128   fdk96k   ffa96k   fdk64k   ffa64k  
  4.33     3.47     3.43     2.83     2.69     2.02  

---------------------------- p-value Matrix ---------------------------

         ffa128   fdk96k   ffa96k   fdk64k   ffa64k  
fdk128   0.000*   0.000*   0.000*   0.000*   0.000*  
ffa128            0.668    0.000*   0.000*   0.000*  
fdk96k                     0.000*   0.000*   0.000*  
ffa96k                              0.201    0.000*  
fdk64k                                       0.000*  
-----------------------------------------------------------------------

fdk128 is better than ffa128, fdk96k, ffa96k, fdk64k, ffa64k
ffa128 is better than ffa96k, fdk64k, ffa64k
fdk96k is better than ffa96k, fdk64k, ffa64k
ffa96k is better than ffa64k
fdk64k is better than ffa64k


Raw data:
Code: [Select]
64k	96k	128k	fdk64k	fdk96k	fdk128k
%feature 7 FFmpeg's native AAC encoder FFmpeg's native AAC encoder FFmpeg's native AAC encoder FDK AAC encoder FDK AAC encoder FDK AAC encoder
%feature 10 -c:a aac -c:a aac -c:a aac -c:a libfdk_aac -c:a libfdk_aac -c:a libfdk_aac
%feature 11 -b:a 64k -b:a 96k -b:a 128k -b:a 64k -b:a 96k -b:a 128k
%feature 12 66 kbps 99 kbps 134 kbps 66 kbps 98 kbps 130 kbps
1.800 2.600 3.300 2.600 3.600 4.200
2.000 2.800 3.500 2.700 3.200 4.400
2.000 2.900 3.300 2.700 3.900 4.100
2.100 3.300 3.900 2.600 3.800 4.200
2.200 2.700 3.500 3.200 3.800 4.000
1.800 2.700 3.300 2.500 3.300 3.900
2.200 2.800 3.400 2.900 3.700 5.000
2.000 3.100 3.600 2.700 3.800 4.300
1.900 2.500 3.500 3.400 3.800 4.000
2.100 2.900 3.500 2.400 2.800 3.900
1.800 2.800 2.900 2.300 3.200 3.900
2.200 2.400 3.400 2.900 3.500 4.100
2.200 2.800 3.500 2.600 3.300 5.000
2.100 3.000 3.700 2.500 2.700 5.000
1.900 3.100 3.800 2.300 3.000 5.000
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples HearttoHeart Pops
%samples Tom'sDiner Acappella

Bitrates:
The FDK AAC tend to have less bitrate deviations. The unit is in bps.
Code: [Select]
%bitrate
ffa64k ffa96k ffa128k fdk64k fdk96k fdk128k
65930 99003 135514 65879 97884 129902
66156 99536 132771 65877 97904 129932
65964 97857 129806 66190 98218 130228
66147 99804 134617 65974 97973 129987
67298 100032 137166 66165 98170 130176
66006 97904 129911 66231 98173 130189
65891 97881 129837 66147 98105 130097
65856 99404 134801 65952 97938 129933
66221 98460 130945 66905 98863 130954
66192 98948 134048 65994 97973 129996
65977 98068 130546 65878 97819 129819
65914 97978 130939 66125 98066 130101
66540 99989 137550 66110 98114 130146
65900 98942 133352 65969 97978 130014
66434 103135 141370 66179 98205 130166

  • LithosZA
  • [*][*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #1
The ffmpeg wiki page on AAC Encoding recommends the native aac encoder above libfdk_aac:
Quote
Which encoder should I use? What provides the best quality?
For AAC-LC the likely answer is: Native FFmpeg AAC encoder (aac) >= libfdk_aac > libfaac.

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #2
The ffmpeg wiki page on AAC Encoding recommends the native aac encoder above libfdk_aac:
That statement is likely to be overstated.

  • xnor
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #3
I also read some people suggesting that the ffmpeg implementation sounds as good or better than fdk ... which is clearly nonsense.
"I hear it when I see it."

  • 2012
  • [*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #4
I also read some people suggesting that the ffmpeg implementation sounds as good or better than fdk ... which is clearly nonsense.

They are probably just repeating the wild claim that was made in the announcement in FFmpeg's front page:
Quote
The encoder is transparent at 128kbps for most samples tested with artifacts only appearing in extreme cases. Subjective quality tests put the encoder to be of equal or greater quality than most of the other encoders available to the public.
saldl: A lightweight well-featured CLI downloader optimized for speed and early preview.
https://saldl.github.io

  • xnor
  • [*][*][*][*][*]
  • Developer
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #5
Oh right, yeah I guess that's the case. Sheesh, even open source needs to do some marketing I guess.
"I hear it when I see it."

  • Enig123
  • [*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #6
Are there someone still working on aac codec of ffmpeg? They may get some ideas/theories on how to improve the performance of the encoder based on listening test like this.

  • CoRoNe
  • [*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #7
The FDK-AAC was the clear winner. Compared to 2016/01/17 version of the FFmpeg's native encoder, Fraunhofer's FDK-AAC library delivered the same sound quality at 32kbps reduced bitrates.
Quote from: https://trac.ffmpeg.org/wiki/Encode/HighQualityAudio
libfdk_aac > aac
NOTE: as of 2017 this is no longer necessarily the case. The built in aac encoder is quite good.
Do you happen to know, Kamedo2, how the native encoder of today fares against FDK-AAC?
  • Last Edit: 08 September, 2017, 03:29:26 PM by CoRoNe

  • Kamedo2
  • [*][*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #8
Do you happen to know, Kamedo2, how the native encoder of today fares against FDK-AAC?
No, I don't know.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #9
Let's be realists.

Development of codec takes years.

ffmpeg AAC was far inferior to FDK. You can't change it in 1 year. Even not  2 o 3 years considering how many projects ffmpeg has.

Let alone an idea to compare it to such encoders as  Apple or Fraunhofer (Winamp edition) AAC.

It hadn't change during several past years  and it won't in next years. The rest is a lie.

 


  • mycroft
  • [*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #10
It hadn't change during several past years  and it won't in next years. The rest is a lie.
You are proved to be liar.

  • IgorC
  • [*][*][*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #11
darling, whatever suits you.

  • dutch109
  • [*][*][*]
Re: Personal Listening Test of AAC encoders available from FFmpeg
Reply #12
OP has provided detailed data of a listening test that shows that FDK-ACC is superior to FFmpeg's AAC encoder, despite the latter having been improved in 2016.

FFmpeg has provided no such data to back up their claim that their encoder is superior.
Opus 96 kb/s (Android) / Vorbis -q5 (PC) / WavPack -hhx6m (Archive)
Developer of r128gain, SACAD and https://playnoise.com/