Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of FFmpeg AAC encoders (Read 19124 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of FFmpeg AAC encoders

Abstract:
Blind Comparison between FFmpeg native experimental AAC encoder with and without v4 patch, and FDK-AAC via FFmpeg, at 96kbps and 128kbps.

Encoders:
FFmpeg r55212(it was a current git head in August 4th 2013)
FFmpeg r55212 with aac-improvements-wip-v4-vbr.patch applied
https://ffmpeg.org/trac/ffmpeg/ticket/2686
fdk-aac-0.1.1 -acodec libfdk_aac via FFmpeg r55212
http://sourceforge.net/projects/opencore-amr/files/fdk-aac/

Settings:
ffmpeg55212 -y -i input.wav -c:a aac -strict experimental -b:a 96k output.mp4
ffmpeg55212_patchv4 -y -i input.wav -c:a aac -strict experimental -b:a 96k output.mp4
ffmpeg55212 -y -i input.wav -c:a libfdk_aac -b:a 96k -afterburner 1 output.mp4

ffmpeg55212 -y -i input.wav -c:a aac -strict experimental -b:a 128k output.mp4
ffmpeg55212_patchv4 -y -i input.wav -c:a aac -strict experimental -b:a 128k output.mp4
ffmpeg55212 -y -i input.wav -c:a libfdk_aac -b:a 128k -afterburner 1 output.mp4

Samples:
20 Sounds of various genres, including difficult samples.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

Hardwares:
Sony PSP-3000 + RP-HT560.

Results



Conclusions & Observations:
The FDK-AAC had a significant and big quality advantage, compared to the FFmpeg experimental native AAC encoder.
Also, the positive effect of the v4 patch was confirmed in 128kbps. v4 patch certainly improves the audio quality.


Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 20
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              119          72.61
Testers (blocks)    19          10.68
Codecs eval'd        5          46.58    9.32   57.67  0.00E+000
Error               95          15.35    0.16
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.252

Means:

fdk_128k fv4_128k fdk_96k  fex_128k fv4_96k  fex_96k
  4.11     3.34     3.17     3.00     2.43     2.19

---------------------------- p-value Matrix ---------------------------

         fv4_128k fdk_96k  fex_128k fv4_96k  fex_96k
fdk_128k 0.000*   0.000*   0.000*   0.000*   0.000*
fv4_128k          0.184    0.009*   0.000*   0.000*
fdk_96k                    0.184    0.000*   0.000*
fex_128k                            0.000*   0.000*
fv4_96k                                      0.068
-----------------------------------------------------------------------

fdk_128k is better than fv4_128k, fdk_96k, fex_128k, fv4_96k, fex_96k
fv4_128k is better than fex_128k, fv4_96k, fex_96k
fdk_96k is better than fv4_96k, fex_96k
fex_128k is better than fv4_96k, fex_96k
Raw data:
Code: [Select]
% FFmpeg AAC 96kbps and 128kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
% [url=http://zak.s206.xrea.com/bitratetest/graphmaker3.htm]http://zak.s206.xrea.com/bitratetest/graphmaker3.htm[/url]
fex_96k	fv4_96k	fdk_96k	fex_128k	fv4_128k	fdk_128k
%feature 7 96kbps 96kbps 96kbps 128kbps 128kbps 128kbps
1.800	2.100	3.100	2.400	2.900	3.400	
2.100	2.600	3.000	2.800	3.200	3.800	
2.500	3.300	2.800	3.600	4.100	3.900	
2.200	2.500	3.000	3.100	3.500	3.700	
2.100	2.400	3.400	2.600	3.200	3.800	
2.400	2.700	3.200	3.000	4.000	3.600	
2.100	1.900	4.300	2.300	2.400	5.000	
2.800	3.100	4.000	3.800	4.100	4.500	
1.900	2.000	3.000	2.300	2.200	3.500	
1.900	2.200	3.200	2.700	3.900	4.200	
2.300	2.300	2.700	3.200	3.400	3.900	
2.200	2.500	2.600	2.900	3.400	3.700	
2.100	2.000	2.700	3.400	3.200	4.300	
2.700	3.200	3.100	3.600	3.800	5.000	
2.000	1.900	2.900	2.900	2.400	5.000	
2.700	2.900	3.100	4.200	4.400	4.000	
2.000	2.400	3.500	2.500	3.300	5.000	
2.100	2.400	3.300	2.900	3.400	3.700	
1.800	1.900	3.600	2.600	2.800	4.200	
2.200	2.300	2.900	3.200	3.200	3.900	
%samples	41_30sec	hihats
%samples	finalfantasy	cemb
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	orch
%samples	mybloodrusts	guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart_to_Heart	Pops
%samples	Tom's_Diner	Vocal
%samples	Reunion_Blues	Jazz
%samples	French	Speech
%samples	undelete	Pops
%samples	Dimmu_Borgir	Metal
%samples	Run_up	Pops
Bitrates:
Code: [Select]
%bitrates
98047	99633	97884	130027	132691	129902
97968	97493	97904	129893	128481	129932
98086	99534	98218	130174	131700	130228
98023	99672	97973	130025	132839	129987
98185	101107	98170	130210	134467	130176
98057	98083	98173	130099	130354	130189
97976	98413	98105	130058	130253	130097
98036	99669	97938	129986	132192	129933
98551	101322	98863	130614	134139	130954
98001	99409	97973	130011	131928	129996
97916	98599	97819	129968	130927	129819
98023	98929	98066	130070	131320	130101
98152	99640	98114	130198	132861	130146
98017	99379	97978	129990	131970	130014
98610	98271	98205	130611	129521	130166
98733	99587	97895	130899	132309	129893
98194	99502	97905	130235	128148	129938
97896	99076	97932	129981	131508	129981
98019	98110	98048	130155	130212	130136
98020	99044	98177	130087	131553	130171
v4 patch had the most bitrate fluctuation.

Personal Listening Test of FFmpeg AAC encoders

Reply #1 –
Thanks for sharing, this is certainly interesting, especially the fact that fdk-aac is "almost never" transparent at 128 kbits for you?
Did you test/plan to test quality based settings of fdk-aac as well?
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

Personal Listening Test of FFmpeg AAC encoders

Reply #2 –
I wonder if the quality of the internal FFMPEG aac encoder has reached the quality of libfaac yet?

Based on past experiences the quality of the encoders is ranked as such:
libfdk_aac > libfaac > aac > libvo-aacenc

Personal Listening Test of FFmpeg AAC encoders

Reply #3 –
Quote from: smok3 on
Thanks for sharing, this is certainly interesting, especially the fact that fdk-aac is "almost never" transparent at 128 kbits for you?
Did you test/plan to test quality based settings of fdk-aac as well?

Apple used to advertise AAC 128 kbps as "transparent", but according to my past test, Apple AAC consistently scores around 4.1. So I'd say that fdk-aac is an excellent encoder in 128kbps, the quality is even comparable to the mighty Apple, when encoding music at 128kbps. At 96kbps, Apple AAC typically gets 3.6, so fdk-aac may sound worse than the Apple. I may test the VBR fdk-aac in a distant future, but I don't expect too much from employing the VBR.

Personal Listening Test of FFmpeg AAC encoders

Reply #4 –
Quote from: LithosZA on
I wonder if the quality of the internal FFMPEG aac encoder has reached the quality of libfaac yet?

Based on past experiences the quality of the encoders is ranked as such:
libfdk_aac > libfaac > aac > libvo-aacenc

Based on this test, Claudio Freire is now writing a new patch that fixes the weak point exposed in this test. It may beat the outdated libfaac with margin, but it's too early to draw any conclusions. My recommendation is to wait for the new patch.

Why do people stick to the old libfaac? Libfaac is extremely outdated, exceptional low quality in 64kbps, unfriendly, and non-free. There should be all the reason to pick libfdk_aac over libfaac.

Personal Listening Test of FFmpeg AAC encoders

Reply #5 –
Quote from: Kamedo2 on
Why do people stick to the old libfaac? Libfaac is extremely outdated, exceptional low quality in 64kbps, unfriendly, and non-free. There should be all the reason to pick libfdk_aac over libfaac.

A recent discussion related to that: http://www.hydrogenaudio.org/forums/index....howtopic=102580

And thanks for taking the time to do this, Kamedo2.

Personal Listening Test of FFmpeg AAC encoders

Reply #6 –
Is there anybody who have color vision deficiency or who have difficulty reading the results for other reasons? Should I change how I plot?

Personal Listening Test of FFmpeg AAC encoders

Reply #7 –
Kamedo2,

The plot is ok.

Interesting test. You're all into open source AAC encoders. Do You know if there will be a long term development of ffmpeg AAC encoder?
There were some attempts to develop it during last years but it hasn't come any close to commercial AAC encoders in terms of quality.
Honestly I doubt there will be a big development of classic MDCT audio encoders. The viable options are Opus, USAC  or completely new  formats in future.       

Personal Listening Test of FFmpeg AAC encoders

Reply #8 –
Quote from: IgorC on
Do You know if there will be a long term development of ffmpeg AAC encoder?

klaussfreire's new v5 patch is out for test now, and and it's the another major improvement of ABR.
He's now writing a v6 patch, which is I think a minor revision of v5, including optimization of VBR.