Abstract:
Blind sound quality comparison between experimental and released Opus encoders, at 32 kbps, 48 kbps, 64 kbps and 80kbps.
Encoders:
Opus 1.1, with opus-tools 0.1.9 https://archive.mozilla.org/pub/opus/win32/opus-tools-0.1.9-win32.zip
Opus 1.1.1-rc-49-g5db9e14 branch exp_lbr_tune, with opus-tools 0.1.9.
Settings:
opusenc --bitrate 32 in.wav out.opus
opusenc --bitrate 48 in.wav out.opus
opusenc --bitrate 64 in.wav out.opus
opusenc --bitrate 80 in.wav out.opus
opusenc --bitrate 32 in.wav out.opus
opusenc --bitrate 48 in.wav out.opus
opusenc --bitrate 64 in.wav out.opus
opusenc --bitrate 80 in.wav out.opus
Samples:
Total 15 samples from my corpus, tested twice per sample.
Hardwares:
Sony PSP-3000 + RP-HT560(1st), RP-HJE150(2nd).
Results:
The change of scores in exp_lbr_tune. They are mostly positive.
The bitrate vs quality graph.
Conclusions & Observations:
The Opus development branch exp_lbr_tune, has solid quality improvements in 32kbps and 48 kbps.
In 64 kbps and 80kbps, there are no big difference in quality between the exp_lbr_tune and the trunk Opus 1.1.
Anova analysis:
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 15
Critical significance: 0.05
Significance of data: 1.11E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 119 92.40
Testers (blocks) 14 6.66
Codecs eval'd 7 81.87 11.70 296.65 1.11E-016
Error 98 3.86 0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.144
Means:
exp80k org80k org64k exp64k exp48k org48k exp32k org32k
4.14 4.13 3.90 3.87 3.23 3.05 2.24 1.82
---------------------------- p-value Matrix ---------------------------
org80k org64k exp64k exp48k org48k exp32k org32k
exp80k 0.854 0.001* 0.000* 0.000* 0.000* 0.000* 0.000*
org80k 0.002* 0.001* 0.000* 0.000* 0.000* 0.000*
org64k 0.714 0.000* 0.000* 0.000* 0.000*
exp64k 0.000* 0.000* 0.000* 0.000*
exp48k 0.015* 0.000* 0.000*
org48k 0.000* 0.000*
exp32k 0.000*
-----------------------------------------------------------------------
exp80k is better than org64k, exp64k, exp48k, org48k, exp32k, org32k
org80k is better than org64k, exp64k, exp48k, org48k, exp32k, org32k
org64k is better than exp48k, org48k, exp32k, org32k
exp64k is better than exp48k, org48k, exp32k, org32k
exp48k is better than org48k, exp32k, org32k
org48k is better than exp32k, org32k
exp32k is better than org32k
Raw data:
exp32k exp48k exp64k exp80k org32k org48k org64k org80k
1.650 3.100 3.500 3.950 1.800 2.800 3.650 4.000
2.600 3.650 4.100 4.350 1.950 3.200 4.150 4.350
2.350 3.400 4.000 4.250 1.550 3.100 3.900 4.200
1.700 2.900 3.750 4.050 1.700 2.850 3.950 4.000
3.050 3.700 4.200 4.300 2.000 3.400 4.100 4.250
1.750 2.350 3.450 3.700 1.500 2.400 3.550 3.550
2.650 3.250 4.150 4.250 2.000 3.200 4.150 4.250
2.100 3.450 3.900 4.200 1.750 3.150 3.950 4.200
2.350 3.100 3.450 3.700 1.900 2.850 3.500 3.700
1.650 2.450 3.800 4.250 1.700 2.300 3.750 4.250
2.200 3.200 3.850 4.300 1.800 3.050 3.950 4.300
2.150 3.350 4.050 4.100 1.650 3.050 4.000 4.150
2.650 3.500 4.050 4.400 1.900 3.600 4.000 4.400
1.950 3.450 3.950 4.150 1.650 3.200 4.000 4.150
2.750 3.600 3.850 4.150 2.450 3.600 3.850 4.150
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples HearttoHeart Pops
%samples Tom'sDiner Acappella
Bitrates:
The exp_lbr_tune branch has more bitrate deviations down to 32 kbps.
%bitrate
exp32k exp48k exp64k exp80k org32k org48k org64k org80k
36749 54937 73371 92363 33033 52076 73438 92430
48355 68543 87084 106794 34026 59325 87161 106872
37576 56210 74723 93189 33190 52377 74764 93229
35411 53122 70453 87718 33374 51285 70523 87786
42211 62890 82978 103395 33742 56865 83108 103540
34296 50768 67697 84163 33250 50278 67966 84437
31638 47318 61532 75075 33409 48583 61552 75096
38539 57412 76608 95888 33395 53455 76761 96040
41265 60008 78435 97111 34648 55392 78772 97397
36911 54502 72279 90022 33483 51861 72311 90054
34338 50941 67863 85263 33064 49980 68072 85471
38476 56844 74928 92831 34037 53583 75361 93264
36575 54928 72355 89386 33628 52318 72495 89527
33863 51076 67929 85114 33186 50203 67987 85174
33287 50391 75053 88353 33507 51445 75864 89179
%album bitrate
33564 50005 66320 82681 32480 49071 66477 82842