Abstract:
Personal blind sound quality comparison of the Opus and the exhale xHE-AAC (USAC) encoders.
Encoders:
The latest versions as of 2021 April were used.
The official build opus-tools-0.2-opus-1.3 was used as the Opus encoder.
The exhale-V1.1.4-cc4151b9_x64 was used as the xHE-AAC (USAC) encoder.
According to the developer, there are no quality change between the 1.1.4 and the exhale 1.1.5.
Opus Settings (~70kbps):
opusenc --bitrate 62 in.wav out.opus
xHE-AAC Settings (~70kbps):
exhale was tested with and without the SBR encoding.
exhale c is the setting with the SBR encoding.
exhale 1 was tested with both the 32kHz and 44.1kHz input.
refalac64 in.wav --rate 32000 -D -b 32 -o in-32kHz.wav
exhale 1 in_32kHz.wav out.32000Hz.nosbr.mp4
exhale 1 in.wav out.44100Hz.nosbr.mp4
exhale c in.wav out.44100Hz.sbr.mp4
Opus Settings (~86kbps):
opusenc --bitrate 80 in.wav out.opus
xHE-AAC Settings (~86kbps):
exhale was tested with and without the SBR encoding.
exhale e is the setting with the SBR encoding.
exhale 2 in.wav out.44100Hz.nosbr.mp4
exhale e in.wav out.44100Hz.sbr.mp4
Sample tracks:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 diverse music and speech sound samples.
Hardware:
Sony PSP-3000 + RP-HT560.
Results:
Tracks list:
Conclusions & Observations:
- Despite the low bitrates, Opus was rated more than 3.0 on most tracks.
- The rate of the exhale was also high, if the best setting was picked.
- Using about 16kbps higher bitrate significantly improved the sound quality (p=0.000).
- In the ~70kbps exhale settings, converting the input sound into 32kHz before the exhale encoding had better effect on the output quality (p=0.003).
- In the ~70kbps exhale settings, using SBR feature may improve the output sound quality, but we cannot be certain from this test alone (p=0.057). Using the 32kHz sampling rate without the SBR feature is more reliable way.
- In the ~86kbps exhale settings, using SBR feature may degrade the output sound quality, but we cannot be certain from this test alone (p=0.084).
- It is unclear whether opus was the winner, on both ~70kbps settings (p=0.076) and the ~86kbps settings (p=0.111), if the exhale was allowed to change the sampling rate into 32kHz and disable the SBR settings.
Anova analysis:
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 27
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 188 38.85
Testers (blocks) 26 13.54
Codecs eval'd 6 10.78 1.80 19.28 0.00E+000
Error 156 14.54 0.09
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.164
Means:
opu80k xh80k2 xh88ke opu64k xh64k3 xh64kc xh64k1
3.92 3.79 3.64 3.57 3.43 3.34 3.18
---------------------------- p-value Matrix ---------------------------
xh80k2 xh88ke opu64k xh64k3 xh64kc xh64k1
opu80k 0.111 0.001* 0.000* 0.000* 0.000* 0.000*
xh80k2 0.084 0.012* 0.000* 0.000* 0.000*
xh88ke 0.424 0.011* 0.000* 0.000*
opu64k 0.076 0.005* 0.000*
xh64k3 0.286 0.003*
xh64kc 0.057
-----------------------------------------------------------------------
opu80k is better than xh88ke, opu64k, xh64k3, xh64kc, xh64k1
xh80k2 is better than opu64k, xh64k3, xh64kc, xh64k1
xh88ke is better than xh64k3, xh64kc, xh64k1
opu64k is better than xh64kc, xh64k1
xh64k3 is better than xh64k1
Raw data:
opus.62k exhale.1_32kHz exhale.1 exhale.c opus.80k exhale.2 exhale.e
%feature 5 SBR enabled SBR enabled
%feature 6 48kHz(fixed) 32kHz 44.1kHz 44.1kHz 48kHz(fixed) 44.1kHz 44.1kHz
%feature 7 Opus xHE-AAC (USAC) xHE-AAC (USAC) xHE-AAC (USAC) Opus xHE-AAC (USAC) xHE-AAC (USAC)
%feature 8 ≒64kbps ≒64kbps ≒64kbps ≒64kbps ≒80kbps ≒80kbps ≒80kbps
%genre Kamedo2's 15 sample
2.400 3.400 2.800 3.200 3.100 3.800 3.500
2.900 4.100 3.100 3.300 3.500 3.800 3.200
3.200 3.400 2.700 3.800 3.900 4.200 3.700
3.100 3.600 2.900 3.400 3.700 3.900 3.800
4.200 3.200 2.800 3.100 4.300 3.400 3.800
3.800 3.600 3.900 2.800 4.100 3.700 2.800
3.900 3.300 2.900 3.400 3.800 3.500 3.500
3.500 3.400 3.200 3.300 3.900 3.800 4.100
2.900 3.500 3.400 3.200 3.600 3.900 3.800
3.500 3.200 2.600 2.800 3.700 3.600 3.300
3.500 2.900 2.700 2.800 3.900 3.700 3.100
3.700 3.500 3.200 3.300 3.800 3.600 3.900
3.600 3.400 2.900 3.900 4.200 3.700 4.300
2.900 2.700 2.600 2.700 3.800 3.600 3.300
3.100 2.800 2.700 2.600 3.700 3.400 3.300
%genre IgorC's 12 sample
4.100 3.500 3.600 3.100 4.200 3.800 3.200
4.400 3.100 3.700 3.200 4.200 3.800 3.500
4.100 3.900 3.500 3.100 4.300 3.800 3.300
3.400 3.700 3.200 3.800 3.700 4.200 4.100
3.300 2.900 2.800 2.900 3.900 3.700 3.100
4.100 3.500 3.600 3.700 4.300 3.900 3.800
3.800 3.200 2.800 3.600 4.100 3.400 3.700
3.600 3.400 3.100 3.700 3.600 3.800 3.900
3.900 3.600 3.900 3.900 4.200 3.700 4.100
3.200 3.700 3.500 3.400 3.600 3.800 3.900
4.200 4.100 4.000 4.100 4.300 4.400 4.200
4.200 3.900 3.700 4.000 4.400 4.300 4.100
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella
%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech
Bitrates:
opus.62k exhale.1_32kHz exhale.1 exhale.c opus.80k exhale.2 exhale.e
%bitrate
73541 87569 80218 81199 95249 96833 104028
82934 68322 70776 58106 103687 85669 82409
70562 78701 76656 67404 90624 93945 93430
67597 75989 72969 68502 86942 88539 90542
83964 72871 72271 69271 107602 89230 95502
65405 76979 77163 64635 83760 90984 87770
67762 67039 63194 61219 84811 77129 84183
73479 78271 75570 71365 94913 91817 97155
72357 87082 84782 72109 92233 99839 95125
72197 72303 70099 67653 92551 85964 94803
63895 72056 70826 65889 83000 85863 88973
72916 76197 74314 68295 93070 90621 92831
67815 72736 69377 67638 86549 84703 92090
67855 72474 70436 63394 87487 85528 83817
68512 71395 69441 64523 82994 85566 90219
78994 77419 76047 69405 103220 90768 94019
90460 88465 89677 79211 117074 104521 100253
82867 81995 78370 75990 107081 95468 107020
86735 84614 87039 68243 109700 104426 87674
69926 73064 72773 64572 89108 88835 86566
83625 67835 69040 57987 106781 85543 87132
75347 79339 75534 72506 95983 92273 97080
69710 81516 75998 86717 91347 92649 109027
59436 62915 61911 62957 75877 76115 85728
75511 77804 77281 65768 95042 91216 87071
56687 57407 55188 59437 67418 70100 85110
73034 71268 69321 69985 93052 85483 95719