Skip to main content

Notice

Please be aware that much of the software linked to or mentioned on this forum is niche and therefore infrequently downloaded. Lots of anti-virus scanners and so-called malware detectors like to flag infrequently downloaded software as bad until it is either downloaded enough times, or its developer actually bothers with getting each individual release allow listed by every single AV vendor. You can do many people a great favor when encountering such a "problem" example by submitting them to your AV vendor for examination. For almost everything on this forum, it is a false positive.
Topic: Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder (Read 603 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder

Abstract:
Personal blind sound quality comparison of the Opus and the exhale xHE-AAC (USAC) encoders.

Encoders:
The latest versions as of 2021 April were used.
The official build opus-tools-0.2-opus-1.3 was used as the Opus encoder.
The exhale-V1.1.4-cc4151b9_x64 was used as the xHE-AAC (USAC) encoder.
According to the developer, there are no quality change between the 1.1.4 and the exhale 1.1.5.

Opus Settings (~70kbps):
opusenc --bitrate 62 in.wav out.opus

xHE-AAC Settings (~70kbps):
exhale was tested with and without the SBR encoding.
exhale c is the setting with the SBR encoding.
exhale 1 was tested with both the 32kHz and 44.1kHz input.

refalac64 in.wav --rate 32000 -D -b 32 -o in-32kHz.wav
exhale 1 in_32kHz.wav out.32000Hz.nosbr.mp4
exhale 1 in.wav out.44100Hz.nosbr.mp4
exhale c in.wav out.44100Hz.sbr.mp4

Opus Settings (~86kbps):
opusenc --bitrate 80 in.wav out.opus

xHE-AAC Settings (~86kbps):
exhale was tested with and without the SBR encoding.
exhale e is the setting with the SBR encoding.
exhale 2 in.wav out.44100Hz.nosbr.mp4
exhale e in.wav out.44100Hz.sbr.mp4

Sample tracks:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 diverse music and speech sound samples.

Hardware:
Sony PSP-3000 + RP-HT560.

Results:



Tracks list:



Conclusions & Observations:
  • Despite the low bitrates, Opus was rated more than 3.0 on most tracks.
  • The rate of the exhale was also high, if the best setting was picked.
  • Using about 16kbps higher bitrate significantly improved the sound quality (p=0.000).
  • In the ~70kbps exhale settings, converting the input sound into 32kHz before the exhale encoding had better effect on the output quality (p=0.003).
  • In the ~70kbps exhale settings, using SBR feature may improve the output sound quality, but we cannot be certain from this test alone (p=0.057). Using the 32kHz sampling rate without the SBR feature is more reliable way.
  • In the ~86kbps exhale settings, using SBR feature may degrade the output sound quality, but we cannot be certain from this test alone (p=0.084).
  • It is unclear whether opus was the winner, on both ~70kbps settings (p=0.076) and the ~86kbps settings (p=0.111), if the exhale was allowed to change the sampling rate into 32kHz and disable the SBR settings.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              188          38.85
Testers (blocks)    26          13.54
Codecs eval'd        6          10.78    1.80   19.28  0.00E+000
Error              156          14.54    0.09
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.164

Means:

opu80k   xh80k2   xh88ke   opu64k   xh64k3   xh64kc   xh64k1  
  3.92     3.79     3.64     3.57     3.43     3.34     3.18  

---------------------------- p-value Matrix ---------------------------

         xh80k2   xh88ke   opu64k   xh64k3   xh64kc   xh64k1  
opu80k   0.111    0.001*   0.000*   0.000*   0.000*   0.000*  
xh80k2            0.084    0.012*   0.000*   0.000*   0.000*  
xh88ke                     0.424    0.011*   0.000*   0.000*  
opu64k                              0.076    0.005*   0.000*  
xh64k3                                       0.286    0.003*  
xh64kc                                                0.057   
-----------------------------------------------------------------------

opu80k is better than xh88ke, opu64k, xh64k3, xh64kc, xh64k1
xh80k2 is better than opu64k, xh64k3, xh64kc, xh64k1
xh88ke is better than xh64k3, xh64kc, xh64k1
opu64k is better than xh64kc, xh64k1
xh64k3 is better than xh64k1


Raw data:
Code: [Select]
opus.62k	exhale.1_32kHz	exhale.1	exhale.c	opus.80k	exhale.2	exhale.e
%feature 5 SBR enabled SBR enabled
%feature 6 48kHz(fixed) 32kHz 44.1kHz 44.1kHz 48kHz(fixed) 44.1kHz 44.1kHz
%feature 7 Opus xHE-AAC (USAC) xHE-AAC (USAC) xHE-AAC (USAC) Opus xHE-AAC (USAC) xHE-AAC (USAC)
%feature 8 ≒64kbps ≒64kbps ≒64kbps ≒64kbps ≒80kbps ≒80kbps ≒80kbps
%genre Kamedo2's 15 sample
2.400 3.400 2.800 3.200 3.100 3.800 3.500
2.900 4.100 3.100 3.300 3.500 3.800 3.200
3.200 3.400 2.700 3.800 3.900 4.200 3.700
3.100 3.600 2.900 3.400 3.700 3.900 3.800
4.200 3.200 2.800 3.100 4.300 3.400 3.800
3.800 3.600 3.900 2.800 4.100 3.700 2.800
3.900 3.300 2.900 3.400 3.800 3.500 3.500
3.500 3.400 3.200 3.300 3.900 3.800 4.100
2.900 3.500 3.400 3.200 3.600 3.900 3.800
3.500 3.200 2.600 2.800 3.700 3.600 3.300
3.500 2.900 2.700 2.800 3.900 3.700 3.100
3.700 3.500 3.200 3.300 3.800 3.600 3.900
3.600 3.400 2.900 3.900 4.200 3.700 4.300
2.900 2.700 2.600 2.700 3.800 3.600 3.300
3.100 2.800 2.700 2.600 3.700 3.400 3.300

%genre IgorC's 12 sample
4.100 3.500 3.600 3.100 4.200 3.800 3.200
4.400 3.100 3.700 3.200 4.200 3.800 3.500
4.100 3.900 3.500 3.100 4.300 3.800 3.300
3.400 3.700 3.200 3.800 3.700 4.200 4.100
3.300 2.900 2.800 2.900 3.900 3.700 3.100
4.100 3.500 3.600 3.700 4.300 3.900 3.800
3.800 3.200 2.800 3.600 4.100 3.400 3.700
3.600 3.400 3.100 3.700 3.600 3.800 3.900
3.900 3.600 3.900 3.900 4.200 3.700 4.100
3.200 3.700 3.500 3.400 3.600 3.800 3.900
4.200 4.100 4.000 4.100 4.300 4.400 4.200
4.200 3.900 3.700 4.000 4.400 4.300 4.100

%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella

%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech

Bitrates:



Code: [Select]
opus.62k	exhale.1_32kHz	exhale.1	exhale.c	opus.80k	exhale.2	exhale.e
%bitrate
73541 87569 80218 81199 95249 96833 104028
82934 68322 70776 58106 103687 85669 82409
70562 78701 76656 67404 90624 93945 93430
67597 75989 72969 68502 86942 88539 90542
83964 72871 72271 69271 107602 89230 95502
65405 76979 77163 64635 83760 90984 87770
67762 67039 63194 61219 84811 77129 84183
73479 78271 75570 71365 94913 91817 97155
72357 87082 84782 72109 92233 99839 95125
72197 72303 70099 67653 92551 85964 94803
63895 72056 70826 65889 83000 85863 88973
72916 76197 74314 68295 93070 90621 92831
67815 72736 69377 67638 86549 84703 92090
67855 72474 70436 63394 87487 85528 83817
68512 71395 69441 64523 82994 85566 90219
78994 77419 76047 69405 103220 90768 94019
90460 88465 89677 79211 117074 104521 100253
82867 81995 78370 75990 107081 95468 107020
86735 84614 87039 68243 109700 104426 87674
69926 73064 72773 64572 89108 88835 86566
83625 67835 69040 57987 106781 85543 87132
75347 79339 75534 72506 95983 92273 97080
69710 81516 75998 86717 91347 92649 109027
59436 62915 61911 62957 75877 76115 85728
75511 77804 77281 65768 95042 91216 87071
56687 57407 55188 59437 67418 70100 85110
73034 71268 69321 69985 93052 85483 95719

Re: Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder

Reply #1
Impressive test Kamedo2. Very nice work!

At ~64 kbps I very recently got the same conclusion:
These results can also be compared to my personal test done last year between 32 KHz and 44 KHz at 64 kbps. 32 KHz was better to my taste. Today, Exhale with SBR probably sound as good if not better (with a plain and full frequency sound due to SBR).
• In other words there's an audible bitrate starvation for exhale at 64 kbps that either resampling to 32 KHz or SBR technique solves (with some drawbacks: preecho and smearing).

• Opus is statistically tied but seems stronger. Sharp attacks samples like castanets.wav or eig.wav help to widen the gap between the two formats.

• It could be noted that exhale c is a bit under 64 kbps. On my big table exhale c is 60.9 kbps while exhale 1 is 64.7 (and Opus -62 should be ~64.5 kbps). So exhale c is both better and smaller than exhale 1 (and is also smaller than 1 resampled to 32 KHz). I wouldn't bother anymore with 32 KHz resampling but rather keep SBRed Exhale at ~64 kbps and below. It could make the encoder a bit more user friendly at least on a GUI side (a bit like EZ CD Audio converter is doing).



80 kbps results are also very interesting.
• SBR seems to show some limitation here and exhale's starvation is clearly less obvious with non-SBR settings (from exhale 1 to 2 : +0.61 point ; from opus 62 to 80: +0.35; exhale -c to -e: +0.30). 80 kbps is clearly a mixed zone for Exhale and it's unclear if SBR is better or not. It can also be a matter of subjective taste.

• Between Exhale and Opus: Exhale -2 gets the highest mark on a sample (4.40) and Opus the lowest one (3.10). On the whole picture Opus performs very well. On my table Opus -80 reaches 83.5 kbps and Exhale -2 79.5 kbps. A score of 3.80…3.90 is quite good, more than acceptable, especially for a trained user like you kamedo2. A good listening experience, with minor issues and a lot of saved space.

Re: Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder

Reply #2
@Kamedo2  Excellent work! I was also wondering what is the difference in quality between SBR vs Standard mode. I must admit that I hoped for further improvement when using SBR mode at 80k. Probably not as relevant as it is for 64k or lower.

@Kamedo2  @guruboolez

Is this test at 64k - 80k also relevant when using bluetooth device such as headphones/speakers or is better to use higher bitrate source which is better at transcoding when using bluetooth device? Should 80k be of acceptable quality also for bluetooth?
As I understand, no matter which format/codec is used, transcoding to bluetooth codec always occur.
WavPack v5.4 -b450hh

Re: Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder

Reply #3
@Kamedo2  @guruboolez
Is this test at 64k - 80k also relevant when using bluetooth device such as headphones/speakers or is better to use higher bitrate source which is better at transcoding when using bluetooth device? Should 80k be of acceptable quality also for bluetooth?
As I understand, no matter which format/codec is used, transcoding to bluetooth codec always occur.

Common Bluetooth codecs offer very high fidelity. So you can safely ignore the fact that your 80k file will be transcoded to SBC, AAC, or aptX.
However, because the storage is so cheap now, I find no reason to dive into the 80kbps, when I know that the majority of the score will be less than 4.0, unless I am in a desperate effort to save space. Maybe I will use 128kbps (exhale 5, or opusenc --bitrate 128) in your scenario.

 

Re: Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder

Reply #4
@Kamedo2
Thanks for explanation.
Which audio player can play sbc directly?
WavPack v5.4 -b450hh

 
SimplePortal 1.0.0 RC1 © 2008-2021