Abstract:
Personal blind sound quality comparison of the Opus, HE-AAC v2, xHE-AAC at ~40kbps at 48kHz.
LAME MP3 encoder at -V5 (~131kbps), also 48kHz, was used as a high anchor.
Encoders:
The latest versions as of 2021 May were used, but the xHE-AAC encoder exhale had a quality update during the test.
The official build opus-tools-0.2-opus-1.3 was used as the Opus encoder.
The FFmpeg r.102573 with mstorsjo / fdk-aac v2.0.2 was used as the HE-AAC v2 encoder.
The exhale-V1.1.5-1592b40c_x64 was used as the xHE-AAC (USAC) encoder.
The LAME 3.100.1 was used as the MP3 encoder, high anchor.
Settings:
lossless 44.1kHz source to lossless 48kHz conversion ... refalac64 in.wav --rate 48000 -D -b 32 -o in.48kHz.wav
opus encodes ... opusenc --bitrate 39 in.48kHz.wav out.mp3
HE-AAC v2 encodes ... ffmpeg -i in.48kHz.wav -c:a libfdk_aac -profile:a aac_he_v2 -b:a 42k out.mp4
xHE-AAC encodes with SBR ... exhale a in.48kHz.wav out.mp4
MP3 encodes ... lame -V 5 in.48kHz.wav out.mp3
Sample tracks:
15 sound samples from Kamedo2's samples.
Total 15 diverse music and speech sound samples.
Hardware:
Sony PSP-3000 + Panasonic RP-HT560 (1st), AKG K712 (2nd).
Results:
Results (by hardware):
The 1st is Panasonic RP-HT560, the 2nd is AKG K712.
Conclusions & Observations:
- Opus, FDK-AAC, exhale were rated better than 2.0 (Annoying) on most tracks.
- LAME was used as a high anchor, and rated better than 3.0 (Slightly annoying) on most tracks.
- LAME at setting V5 (131 kbps) is still significantly better than the latest codecs at 40 kbps.
- Outside of the high anchor, there were no big fidelity difference among these three encoders. It is unclear whether which was the winner.
- There were no significant difference in fidelity observed by the two hardware tested.
Anova analysis:
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 15
Critical significance: 0.05
Significance of data: 7.34E-007 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 59 20.71
Testers (blocks) 14 4.55
Codecs eval'd 3 8.43 2.81 15.26 7.34E-007
Error 42 7.73 0.18
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.316
Means:
mp3128 exh36k opu36k fdk36k
3.75 2.97 2.90 2.81
---------------------------- p-value Matrix ---------------------------
exh36k opu36k fdk36k
mp3128 0.000* 0.000* 0.000*
exh36k 0.673 0.313
opu36k 0.555
-----------------------------------------------------------------------
mp3128 is better than exh36k, opu36k, fdk36k
Raw data:
Opus 1.3 FDK-AAC 2.0.2 exhale 1.1.5 LAME 3.100.1
%feature 7 Opus HE-AAC v2 xHE-AAC(USAC) MP3
%feature 10 --bitrate 39 -c:a libfdk_aac -profile:a aac_he_v2 -b:a 42k a -V 5
%feature 11 albums:~40kbps albums:~43kbps albums:~40kbps albums:~131kbps
%feature 12 tracks:~46kbps tracks:~43kbps tracks:~45kbps tracks:~143kbps
%genre Kamedo2's 15 sample tracks
2.250 2.200 2.800 3.700
3.200 3.900 3.300 2.900
2.700 2.200 2.950 3.800
2.750 2.400 2.950 3.900
3.800 3.000 2.800 4.150
2.000 1.800 3.050 3.750
3.950 3.450 3.600 3.250
3.400 3.200 2.750 4.100
3.050 2.650 2.800 4.000
2.100 2.900 2.550 3.750
2.000 2.550 2.600 3.950
3.100 2.750 2.900 3.700
3.450 3.400 3.050 3.700
2.600 2.750 3.150 3.900
3.200 3.000 3.300 3.700
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella
Raw data (separated by hardware):
opus-1st opus-2nd he-aac-v2-1st he-aac-v2-2nd xhe-aac-1st xhe-aac-2nd mp3-1st mp3-2nd
2.1 2.4 2.3 2.1 2.7 2.9 3.8 3.6
2.9 3.5 3.9 3.9 3.4 3.2 3.3 2.5
2.9 2.5 2.1 2.3 3.2 2.7 3.9 3.7
2.8 2.7 2.3 2.5 3.1 2.8 4.1 3.7
3.7 3.9 2.9 3.1 2.7 2.9 4.2 4.1
1.9 2.1 1.7 1.9 3.2 2.9 3.7 3.8
4.1 3.8 3.9 3.0 3.7 3.5 3.3 3.2
3.5 3.3 3.2 3.2 3.0 2.5 4.1 4.1
3.2 2.9 2.1 3.2 2.9 2.7 3.9 4.1
2.1 2.1 3.1 2.7 2.7 2.4 3.9 3.6
1.9 2.1 2.5 2.6 2.7 2.5 4.1 3.8
2.9 3.3 2.7 2.8 2.9 2.9 3.7 3.7
3.5 3.4 3.3 3.5 2.9 3.2 3.8 3.6
2.7 2.5 2.6 2.9 3.5 2.8 4.1 3.7
3.3 3.1 3.1 2.9 3.4 3.2 3.7 3.7
Bitrates:
Opus 1.3 FDK-AAC 2.0.2 exhale 1.1.5 LAME 3.100.1
%feature 7 Opus HE-AAC v2 xHE-AAC(USAC) MP3
%feature 10 --bitrate 39 -c:a libfdk_aac -profile:a aac_he_v2 -b:a 42k a -V 5
%feature 11 albums:~40kbps albums:~43kbps albums:~40kbps albums:~131kbps
%feature 12 tracks:~46kbps tracks:~43kbps tracks:~45kbps tracks:~143kbps
%y_axis File-based bitrates of each track [bps]
%bitrate
Opus 1.3 FDK-AAC 2.0.2 exhale 1.1.5 LAME 3.100.1
46331 43282 55803 159545
56552 43277 42408 113421
44465 43593 45380 141851
42608 43387 46175 152219
53898 43559 44766 174336
41318 43588 46528 137064
44302 43542 39861 135788
46242 43303 46629 148985
47125 44295 52523 157022
45743 43367 43348 129806
36764 43224 42329 153864
46754 43495 44439 141483
43211 43515 44221 143500
42659 43323 43100 138090
47066 43481 42887 119125
ALBUMS
40416 42802 39730 131059
Topic: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz (Read 9263 times)
previous topic - next topic