Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz (Read 11089 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Abstract:
Personal blind sound quality comparison of the Opus, HE-AAC v2, xHE-AAC at ~40kbps at 48kHz.
LAME MP3 encoder at -V5 (~131kbps), also 48kHz, was used as a high anchor.

Encoders:
The latest versions as of 2021 May were used, but the xHE-AAC encoder exhale had a quality update during the test.
The official build opus-tools-0.2-opus-1.3 was used as the Opus encoder.
The FFmpeg r.102573 with  mstorsjo / fdk-aac v2.0.2 was used as the HE-AAC v2 encoder.
The exhale-V1.1.5-1592b40c_x64 was used as the xHE-AAC (USAC) encoder.
The LAME 3.100.1 was used as the MP3 encoder, high anchor.

Settings:
lossless 44.1kHz source to lossless 48kHz conversion ... refalac64 in.wav --rate 48000 -D -b 32 -o in.48kHz.wav
opus encodes ... opusenc --bitrate 39 in.48kHz.wav out.mp3
HE-AAC v2 encodes ... ffmpeg -i in.48kHz.wav -c:a libfdk_aac -profile:a aac_he_v2 -b:a 42k out.mp4
xHE-AAC encodes with SBR ... exhale a in.48kHz.wav out.mp4
MP3 encodes ... lame -V 5 in.48kHz.wav out.mp3

Sample tracks:
15 sound samples from Kamedo2's samples.
Total 15 diverse music and speech sound samples.

Hardware:
Sony PSP-3000 + Panasonic RP-HT560 (1st), AKG K712 (2nd).

Results:



Results (by hardware):
The 1st is Panasonic RP-HT560, the 2nd is AKG K712.



Conclusions & Observations:
  • Opus, FDK-AAC, exhale were rated better than 2.0 (Annoying) on most tracks.
  • LAME was used as a high anchor, and rated better than 3.0 (Slightly annoying) on most tracks.
  • LAME at setting V5 (131 kbps) is still significantly better than the latest codecs at 40 kbps.
  • Outside of the high anchor, there were no big fidelity difference among these three encoders. It is unclear whether which was the winner.
  • There were no significant difference in fidelity observed by the two hardware tested.

Anova analysis:
Code: [Select]

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 7.34E-007 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               59          20.71
Testers (blocks)    14           4.55
Codecs eval'd        3           8.43    2.81   15.26  7.34E-007
Error               42           7.73    0.18
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.316

Means:

mp3128   exh36k   opu36k   fdk36k  
  3.75     2.97     2.90     2.81  

---------------------------- p-value Matrix ---------------------------

         exh36k   opu36k   fdk36k  
mp3128   0.000*   0.000*   0.000*  
exh36k            0.673    0.313   
opu36k                     0.555   
-----------------------------------------------------------------------

mp3128 is better than exh36k, opu36k, fdk36k


Raw data:
Code: [Select]
Opus 1.3	FDK-AAC 2.0.2	exhale 1.1.5	LAME 3.100.1
%feature 7 Opus HE-AAC v2 xHE-AAC(USAC) MP3
%feature 10 --bitrate 39 -c:a libfdk_aac -profile:a aac_he_v2 -b:a 42k a -V 5
%feature 11 albums:~40kbps albums:~43kbps albums:~40kbps albums:~131kbps
%feature 12 tracks:~46kbps tracks:~43kbps tracks:~45kbps tracks:~143kbps
%genre Kamedo2's 15 sample tracks
2.250 2.200 2.800 3.700
3.200 3.900 3.300 2.900
2.700 2.200 2.950 3.800
2.750 2.400 2.950 3.900
3.800 3.000 2.800 4.150
2.000 1.800 3.050 3.750
3.950 3.450 3.600 3.250
3.400 3.200 2.750 4.100
3.050 2.650 2.800 4.000
2.100 2.900 2.550 3.750
2.000 2.550 2.600 3.950
3.100 2.750 2.900 3.700
3.450 3.400 3.050 3.700
2.600 2.750 3.150 3.900
3.200 3.000 3.300 3.700
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella

Raw data (separated by hardware):
Code: [Select]
opus-1st	opus-2nd	he-aac-v2-1st	he-aac-v2-2nd	xhe-aac-1st	xhe-aac-2nd	mp3-1st	mp3-2nd
2.1 2.4 2.3 2.1 2.7 2.9 3.8 3.6
2.9 3.5 3.9 3.9 3.4 3.2 3.3 2.5
2.9 2.5 2.1 2.3 3.2 2.7 3.9 3.7
2.8 2.7 2.3 2.5 3.1 2.8 4.1 3.7
3.7 3.9 2.9 3.1 2.7 2.9 4.2 4.1
1.9 2.1 1.7 1.9 3.2 2.9 3.7 3.8
4.1 3.8 3.9 3.0 3.7 3.5 3.3 3.2
3.5 3.3 3.2 3.2 3.0 2.5 4.1 4.1
3.2 2.9 2.1 3.2 2.9 2.7 3.9 4.1
2.1 2.1 3.1 2.7 2.7 2.4 3.9 3.6
1.9 2.1 2.5 2.6 2.7 2.5 4.1 3.8
2.9 3.3 2.7 2.8 2.9 2.9 3.7 3.7
3.5 3.4 3.3 3.5 2.9 3.2 3.8 3.6
2.7 2.5 2.6 2.9 3.5 2.8 4.1 3.7
3.3 3.1 3.1 2.9 3.4 3.2 3.7 3.7

Bitrates:



Code: [Select]
Opus 1.3	FDK-AAC 2.0.2	exhale 1.1.5	LAME 3.100.1
%feature 7 Opus HE-AAC v2 xHE-AAC(USAC) MP3
%feature 10 --bitrate 39 -c:a libfdk_aac -profile:a aac_he_v2 -b:a 42k a -V 5
%feature 11 albums:~40kbps albums:~43kbps albums:~40kbps albums:~131kbps
%feature 12 tracks:~46kbps tracks:~43kbps tracks:~45kbps tracks:~143kbps
%y_axis File-based bitrates of each track [bps]
%bitrate
Opus 1.3 FDK-AAC 2.0.2 exhale 1.1.5 LAME 3.100.1
46331 43282 55803 159545
56552 43277 42408 113421
44465 43593 45380 141851
42608 43387 46175 152219
53898 43559 44766 174336
41318 43588 46528 137064
44302 43542 39861 135788
46242 43303 46629 148985
47125 44295 52523 157022
45743 43367 43348 129806
36764 43224 42329 153864
46754 43495 44439 141483
43211 43515 44221 143500
42659 43323 43100 138090
47066 43481 42887 119125

ALBUMS
40416 42802 39730 131059

Re: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Reply #1
Quote
LAME at setting V5 (131 kbps) is still significantly better than the latest codecs at 40 kbps.
I'm not shocked, given that ~128 kbps mp3 was pretty good, perhaps even good enough for many people, in a public 2008 listening test. http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm

Or another way to put it, it's not shocking to me that, despite its limitations, mp3 has better quality with over 3x the bitrate.

Re: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Reply #2
Thanks for this nice and interesting test Kamedo2 :)
All competitors are tied. But the most interesting conclusion in my opinion is the sound quality level reached at this bitrate : MOS close to 3. It's not a wonderfully enjoying quality but it's something usable and probably enjoyable in many cases.

Exhale xHE-AAC seems to be more constant and never reached the “annoying” level in these 15 samples: it's an excellent thing. Quality is more balanced and this encoder is apparently more constant and robust than OPUS and HE-AACv2.

AAC and Opus are very close at this bitrate. Your conclusion perfectly matches the “offical” graph (and claim) from OPUS:

source: https://opus-codec.org/comparison/

Tested samples are fine in my opinion: nothing too hard, general music. You describe the selection as “Total 15 diverse music and speech sound samples” but correct me if I'm wrong there are only musical samples, right?

Re: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Reply #3
You describe the selection as “Total 15 diverse music and speech sound samples” but correct me if I'm wrong there are only musical samples, right?
You're right, only musical samples were included.
I'd like to fix the description, but it seems like there is no way to correct the post now.

Re: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Reply #4
Thank You for this test, Kamedo2  :) 
Quote
Hardware:
Sony PSP-3000 + Panasonic RP-HT560 (1st), AKG K712 (2nd).
It's interesting to observe variations of average scores using different headphones. While for Opus and HEv2 it is pretty the same it's not so for exhale.


Re: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Reply #5
@Kamedo2
Can You please comment on how new pair of headphones  has affected your listening tests as your previous  tests were done using Panasonic RP-HT560?   Was it easier, faster or overall experience is pretty the same?  Does K712 have more extended frequency range, lower distortion to your judgement etc?

Thank You.

 

Re: Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz

Reply #6
@Kamedo2
Can You please comment on how new pair of headphones  has affected your listening tests as your previous  tests were done using Panasonic RP-HT560?   Was it easier, faster or overall experience is pretty the same?  Does K712 have more extended frequency range, lower distortion to your judgement etc?

Thank You.

The fist thing to notice is that K712 plays at noticeably smaller volume.
It covers the entire ear and more comfortable than the RP-HT560.
When it comes to discerning extended frequency range, not much different.