Skip to main content
Topic: Personal Listening Test of MP3 and Opus (Read 6191 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of MP3 and Opus

Abstract:
Blind sound quality comparison of MP3 and Opus.
Opus was tested at half the bitrate of the MP3.

Encoders:
LAME 3.100 http://www.rarewares.org/mp3-lame-bundle.php
opusenc opus-tools 0.1.10-80-g1a646c4 (using libopus 1.3-rc-12-gbc4ecf19, 2018 July version. This is older than the current version. I started this listening test in 2018 August.) https://hydrogenaud.io/index.php/topic,116059.75.html

Settings:
lame -h -b 96 in.wav out.mp3
lame -h -b 128 in.wav out.mp3
lame -h -b 160 in.wav out.mp3
lame -h -b 192 in.wav out.mp3
opusenc --bitrate 48 in.wav out.opus
opusenc --bitrate 64 in.wav out.opus
opusenc --bitrate 80 in.wav out.opus
opusenc --bitrate 96 in.wav out.opus

Samples:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 sound samples.

Hardwares:
Sony PSP-3000 + RP-HT560.

Results:




Conclusions & Observations:
The Opus at 64kbps setting sounds better than the MP3 at 128kbps.
The Opus at 48kbps setting sounds better than the MP3 at 96kbps.
The Opus is likely to be the first format which delivers the equivalent sound quality as MP3 at half the bitrates, not as a marketing plot but as a fact.
The high efficiency of Opus is more pronounced at the lower rate.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 1.11E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              215          86.75
Testers (blocks)    26          19.92
Codecs eval'd        7          45.15    6.45   54.12  1.11E-016
Error              182          21.69    0.12
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.185

Means:

mp3192   opus96   opus80   mp3160   opus64   mp3128   opus48   mp3_96  
  4.27     4.25     4.07     4.03     3.86     3.51     3.32     2.90  

---------------------------- p-value Matrix ---------------------------

         opus96   opus80   mp3160   opus64   mp3128   opus48   mp3_96  
mp3192   0.844    0.035*   0.012*   0.000*   0.000*   0.000*   0.000*  
opus96            0.055    0.021*   0.000*   0.000*   0.000*   0.000*  
opus80                     0.694    0.026*   0.000*   0.000*   0.000*  
mp3160                              0.066    0.000*   0.000*   0.000*  
opus64                                       0.000*   0.000*   0.000*  
mp3128                                                0.042*   0.000*  
opus48                                                         0.000*  
-----------------------------------------------------------------------

mp3192 is better than opus80, mp3160, opus64, mp3128, opus48, mp3_96
opus96 is better than mp3160, opus64, mp3128, opus48, mp3_96
opus80 is better than opus64, mp3128, opus48, mp3_96
mp3160 is better than mp3128, opus48, mp3_96
opus64 is better than mp3128, opus48, mp3_96
mp3128 is better than opus48, mp3_96
opus48 is better than mp3_96


Raw data:
Code: [Select]
opus48	opus64	opus80	opus96	mp3_96	mp3128	mp3160	mp3192	
2.800 3.700 4.100 4.300 1.800 3.400 4.200 4.400
4.000 4.400 4.600 4.600 3.500 4.200 4.400 4.700
3.100 3.800 3.800 4.400 2.700 3.400 4.100 4.300
3.000 3.600 4.100 4.200 3.300 3.800 4.300 4.500
3.500 4.100 4.200 4.500 2.700 2.900 3.400 3.800
2.400 3.100 3.600 3.800 3.300 3.900 4.100 4.200
3.700 5.000 4.300 5.000 2.400 2.700 3.300 4.000
3.500 4.000 4.100 4.300 3.300 3.800 4.400 4.600
2.400 3.000 3.500 3.700 2.500 3.300 3.800 3.900
2.900 3.600 3.900 4.000 2.600 3.200 4.100 4.200
3.000 3.700 3.900 3.900 2.800 3.200 4.100 4.300
2.600 3.600 4.000 3.900 2.400 3.100 3.900 4.200
3.200 3.600 3.900 4.300 2.400 3.900 4.400 4.500
2.500 3.700 4.200 4.400 2.400 3.400 4.000 4.200
3.100 3.400 3.800 4.000 3.000 3.500 3.700 4.200
4.000 4.100 4.400 4.500 2.800 3.300 3.500 3.700
4.600 4.800 4.700 5.000 2.700 3.300 4.000 4.300
3.500 3.800 3.900 3.700 2.500 2.700 2.800 3.200
3.300 3.600 3.800 4.200 3.500 4.000 4.400 4.600
3.000 3.300 3.500 3.700 3.100 3.800 3.900 4.200
4.400 4.500 4.700 4.600 3.300 3.600 4.200 4.600
2.900 3.800 4.100 4.200 2.400 3.300 4.300 4.400
3.500 3.800 3.900 3.900 2.400 2.900 4.000 4.200
3.900 4.300 4.300 4.500 3.500 4.200 4.400 4.400
3.100 3.600 3.900 4.100 3.300 3.800 4.100 4.300
4.300 4.300 4.600 4.700 4.100 4.400 4.800 5.000
3.500 4.000 4.100 4.400 3.600 3.900 4.300 4.400
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples HearttoHeart Pops
%samples Tom'sDiner Acappella
%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech


Bitrates:
Code: [Select]
%bitrate
opus48 opus64 opus80 opus96 mp3_96 mp3128 mp3160 mp3192

56880 75747 95248 114257 96307 128285 160357 192428
66948 84991 103687 123011 96255 128230 160288 192346
54746 72574 90622 108186 96399 128372 160465 192558
52458 69741 86915 104050 96368 128319 160399 192479
65630 86473 107602 127975 96401 128368 160461 192553
50549 67344 83756 99633 96549 128440 160551 192661
53568 69782 84808 99693 96485 128458 160572 192687
56704 75748 94913 113849 96344 128271 160339 192407
56894 74481 92233 109998 96689 128827 161035 193242
55933 74323 92551 110275 96390 128295 160369 192443
49407 66008 83000 100052 96348 128244 160305 192366
56951 75077 93069 110522 96367 128405 160507 192608
53149 69985 86551 103243 96390 128353 160442 192531
52610 69890 87487 104392 96359 128304 160381 192456
56351 70628 82992 97120 96414 128470 160589 192706
61687 81349 103220 125449 97415 129023 161281 193535
69925 93369 117074 140997 96320 128336 160420 192504
64676 85344 107088 128612 96533 128613 160767 192919
69446 88817 109699 130814 96353 128396 160495 192594
54523 71962 89108 105862 96345 128301 160376 192451
68054 85839 106781 125858 96814 128666 160833 192999
58885 77742 95976 113623 97163 128850 161063 193275
54146 71966 91347 110123 96593 128639 160800 192959
46454 61405 75877 90013 96498 128588 160736 192883
59458 77547 95042 113980 96293 128264 160331 192396
48064 58813 67410 79530 96364 128432 160541 192649
57471 75071 93052 110375 96691 128663 160829 192995

%album bitrate
49285 65423 81510 97711 96031 128029 160037 192044

Re: Personal Listening Test of MP3 and Opus

Reply #1
Interesting, but I wonder why you chose CBR, not VBR using lame.
lame3995o -Q1.7
opus --bitrate 140

Re: Personal Listening Test of MP3 and Opus

Reply #2
Interesting, but I wonder why you chose CBR, not VBR using lame.

CBR was widely used in the past so it's a better anchor.

Re: Personal Listening Test of MP3 and Opus

Reply #3
Thank you for this very interesting test.
I use Opus 96 kbps VBR converted by foobar2000.

Re: Personal Listening Test of MP3 and Opus

Reply #4
Hardwares:
Sony PSP-3000 + RP-HT560.

Wait there's abx homebrew for the psp? This makes me think about dusting off mine to use as a portable player instead of my phone.

Re: Personal Listening Test of MP3 and Opus

Reply #5
Can you add also opus at 32kbps? For other part the result is no surprise for me as 64kbps have been my selected bitrate to use on the go for music (phone, car...), I also use 32kbps for audiobooks and 48kbps for podcasts (latelly I even sometimes use 32kbps for podcasts).

Re: Personal Listening Test of MP3 and Opus

Reply #6
Hardwares:
Sony PSP-3000 + RP-HT560.

Wait there's abx homebrew for the psp? This makes me think about dusting off mine to use as a portable player instead of my phone.

Yes, I have it. http://zak.s206.xrea.com/bitratetest/ABCHR2.4.7z http://zak.s206.xrea.com/bitratetest/pspabchr24.jpg It works with LCFW 6.60 PRO-C2. https://code.google.com/archive/p/procfw/wikis/Installation.wiki

Re: Personal Listening Test of MP3 and Opus

Reply #7
Can you add also opus at 32kbps? For other part the result is no surprise for me as 64kbps have been my selected bitrate to use on the go for music (phone, car...), I also use 32kbps for audiobooks and 48kbps for podcasts (latelly I even sometimes use 32kbps for podcasts).

I have to say that Opus at 32kbps is of low quality at this moment, and recommend upgrading to 48kbps, the rate extensively tested, for any static contents distributions. Memory and bandwidths are cheap now. https://hydrogenaud.io/index.php/topic,113985.0.html https://hydrogenaud.io/index.php/topic,111798.0.html

Re: Personal Listening Test of MP3 and Opus

Reply #8
Kamedo2 every bitrate have its use, normally the configs that I use for my phone is this:
Pure talk -> 32kbps
Talk + incidental music -> 48kbps
Music -> 64kbps

Actually 32kbps is more used that most people here can think, specially in places with data caps and/or low speeds.

And I miss a more aggressive VBR mode for opus, more like a dual rate mode where one can specify different rates for music and voice.

 

Re: Personal Listening Test of MP3 and Opus

Reply #9
Actually 32kbps is more used that most people here can think, specially in places with data caps and/or low speeds.

A devoted podcast subscriber who considers listening to the program is an important part of his/her life, may listen to 1 hour of podcast 30 days a month. In that case...

32kbps -> 0.4 GB/month
48kbps -> 0.6 GB/month
diff        -> 0.2 GB/month

Are you really sure that 0.2GB of cost reduction is more important than improving his/her listening experience?
(I may be missing some problems somewhere. Here in Japan increasing the mobile data cap 2GB more typically costs 200~2,000 JPY/month, and after using up all the data cap the speed is typically limited to 128 kbps~200 kbps.)

Re: Personal Listening Test of MP3 and Opus

Reply #10
From my very limited experience ;Optus 32  - very bad artifacts like no need to abx at all. Even at 64 I think i can still hear it but much better quality than 32. @ 96k I don't know.
wavpack 4.8 -b256hx6c

Re: Personal Listening Test of MP3 and Opus

Reply #11
https://tools.ietf.org/html/rfc6716
8-12 kbit/s for NB speech,    NB (narrowband)      |           4 kHz |                   8 kHz
16-20 kbit/s for WB speech,    WB (wideband)        |           8 kHz |                  16 kHz
28-40 kbit/s for FB speech,    FB (fullband)        |      20 kHz (*) |                  48 kHz
48-64 kbit/s for FB mono music
64-128 kbit/s for FB stereo music.

Re: Personal Listening Test of MP3 and Opus

Reply #12
Music with Opus 1.3 at 32 kbps has too much noise and sibilance. Old samples of USAC from the original verification tests usually sound better to me than Opus 1.3 still. Below 64 kbps and especially below 48 kbps Opus loses more and more to certain higher latency codecs. For speech it is a different story though, 32 kbps is great and if limited to mono then even 16 kbps is good. I look forward to what can be done with LPCnet if the remaining issues can be addressed.

https://tools.ietf.org/html/rfc6716
8-12 kbit/s for NB speech,    NB (narrowband)      |           4 kHz |                   8 kHz
16-20 kbit/s for WB speech,    WB (wideband)        |           8 kHz |                  16 kHz
28-40 kbit/s for FB speech,    FB (fullband)        |      20 kHz (*) |                  48 kHz
48-64 kbit/s for FB mono music
64-128 kbit/s for FB stereo music.

With Opus 1.3 it is more like:

16-32 kbit/s for FB speech
32-64 kbit/s for FB stereo speech
48-128 kbit/s for FB stereo music

Mono music has improved a lot too, incidental mono music is okay down to around 24 kbps now IMO.

Re: Personal Listening Test of MP3 and Opus

Reply #13
[...] more like: [...]
"more like" versus "PROPOSED STANDARD"s "these are the bitrate "sweet spots"" ? You are the speechwriter T.May?

Re: Personal Listening Test of MP3 and Opus

Reply #14
[...] more like: [...]
"more like" versus "PROPOSED STANDARD"s "these are the bitrate "sweet spots"" ? You are the speechwriter T.May?

Why the hostility? That RFC is from seven years ago; since then there have been improvements to Opus especially with low bitrate speech. And those "sweet spots" are just someone's opinion.

Re: Personal Listening Test of MP3 and Opus

Reply #15
[...]are just someone's opinion.
yes, by creators.
[...]  have been improvements to Opus [...]
of course, but how much this "improvements" reduce the "comfortable" bitrate? 10% ? 30% ? or 2-5% ?
Why the hostility? [...]
where? love (ἕτερος) and peace (healthy).

Re: Personal Listening Test of MP3 and Opus

Reply #16
Very interesting test result, thanks for this!

 
SimplePortal 1.0.0 RC1 © 2008-2019