Skip to main content
Topic: Personal Listening Test of MP3/Opus/AAC at 96kbps (Read 19712 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Abstract:
Blind sound quality comparison between LAME 3.99.5 abr, Opus 1.1 vbr, NeroAACEnc vbr, FFmpeg's native encoder with intermediate patch v7 abr, FFmpeg's native encoder with intermediate patch v9b abr, all at 96kbps.

Encoders:
LAME 3.99.5
Opus 1.1 with opus-tools-0.1.9, win32
NeroAACEnc 1.5.4.0
ffmpeg r.70351 with v7 patch applied
ffmpeg r.70351 with v9b patch applied

Settings:
lame --abr 98 -S in.wav out.mp3
opusenc --bitrate 91 in.wav out.opus
NeroAacEnc -q 0.333 -if in.wav -of out.mp4
ffmpeg70351_v7 -y -i in.wav -c:a aac -strict experimental -b:a 96k out.mp4
ffmpeg70351_v9b -y -i in.wav -c:a aac -strict experimental -b:a 96k out.mp4

Samples:
Total 74 samples.
40 samples from 2014 public listening test
25 samples from my corpus
9 samples from SoundExpert

Hardwares:
Sony PSP-3000 + RP-HJE150.

Results:



Conclusions & Observations:
Opus had the best score. Nero was the second, LAME was third, and the FFmpeg's native AAC encoder was the last. In the FFmpeg's native AAC encoder, v9b patch, which enables M/S stereo by default, certainly improves audio quality.
The result was consistent over three set of samples.
Code: [Select]
lame opus nero ffv7 ffv9b
3.65 4.36 3.98 3.20 3.45 - 40 samples from the 2014 public listening test.
3.60 4.21 3.80 2.94 3.32 - 25 samples from my corpus.
3.80 4.39 4.08 3.33 3.61 - 9 samples from SoundExpert.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 74
Critical significance:  0.05
Significance of data: 1.11E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of        Degrees    Sum of    Mean
variation        of Freedom  squares  Square    F      p

Total              369        134.26
Testers (blocks)    73          45.11
Codecs eval'd        4          61.69  15.42  164.04  1.11E-016
Error              292          27.45    0.09
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:  0.099

Means:

opus    nero    lame    ffv9b    ffv7
  4.31    3.93    3.65    3.42    3.13

---------------------------- p-value Matrix ---------------------------

        nero    lame    ffv9b    ffv7
opus    0.000*  0.000*  0.000*  0.000*
nero              0.000*  0.000*  0.000*
lame                      0.000*  0.000*
ffv9b                              0.000*
-----------------------------------------------------------------------

opus is better than nero, lame, ffv9b, ffv7
nero is better than lame, ffv9b, ffv7
lame is better than ffv9b, ffv7
ffv9b is better than ffv7
Raw data:
Code: [Select]
% 96kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
lame opus nero ffv7 ffv9b
4.300 4.500 4.400 3.900 3.400
3.200 3.700 3.500 3.100 2.700
3.200 3.600 2.800 2.000 2.400
3.200 4.300 4.100 3.400 3.500
3.600 5.000 3.800 3.400 2.800
3.600 5.000 3.900 2.400 2.500
3.200 3.500 3.700 2.800 2.700
3.300 4.300 3.800 3.500 3.700
3.800 5.000 5.000 3.600 3.900
3.700 4.100 3.600 3.100 3.700
3.100 3.800 3.500 2.600 3.300
3.800 4.200 3.900 3.400 3.700
3.000 4.300 3.600 3.400 3.200
4.400 5.000 5.000 4.600 4.000
3.200 5.000 4.000 3.000 3.700
3.700 4.400 4.000 3.400 4.100
4.300 5.000 4.400 2.900 3.900
3.000 3.900 3.600 3.200 3.200
3.700 4.000 3.200 2.400 3.000
3.900 4.000 3.800 3.000 3.500
3.600 5.000 4.000 3.300 3.700
3.800 4.600 4.100 2.800 3.100
3.800 4.400 4.100 3.400 3.600
3.300 5.000 4.300 3.100 3.900
4.200 4.300 4.100 3.300 3.900
3.900 4.200 4.100 3.300 3.700
4.000 5.000 4.000 3.300 3.700
3.900 4.100 3.500 2.200 3.100
2.800 4.500 4.200 3.200 3.300
3.600 4.300 4.200 3.200 3.500
3.300 3.700 3.600 2.700 3.000
4.000 3.800 3.900 3.400 3.700
4.100 4.400 4.500 3.200 3.800
4.100 4.400 4.000 3.600 3.700
3.500 4.200 3.800 3.100 3.600
3.800 4.100 4.400 4.500 4.300
4.000 4.400 4.000 3.100 2.900
3.200 4.200 3.900 3.600 3.800
4.700 5.000 5.000 3.600 3.500
3.300 4.300 3.900 3.100 3.200
2.600 3.500 2.900 2.100 2.000
4.200 4.500 4.300 3.200 3.800
3.200 4.300 3.600 3.500 3.300
3.000 4.200 3.900 2.500 3.200
3.600 4.400 3.900 2.700 3.100
2.900 3.200 3.500 2.900 3.300
4.000 4.400 4.200 3.500 2.700
3.500 4.200 3.600 3.000 3.600
3.600 3.900 4.000 3.000 3.200
3.600 4.300 4.200 2.600 3.800
2.700 4.000 3.300 2.500 2.900
3.200 4.200 3.600 3.000 3.400
4.000 4.100 3.800 3.300 3.500
3.700 4.600 4.300 2.900 3.300
3.500 4.000 3.200 2.600 2.900
3.100 3.900 3.800 2.900 3.300
4.000 4.400 3.700 2.800 3.500
3.800 4.100 4.200 2.700 3.300
4.200 5.000 4.500 3.600 3.400
4.100 4.500 3.300 3.000 3.500
3.400 4.200 3.600 2.700 3.300
3.900 4.100 4.100 2.800 3.500
4.100 4.500 3.800 2.800 3.400
4.100 4.300 4.000 3.600 3.900
4.000 4.500 3.700 3.400 3.800
3.800 5.000 4.400 3.600 3.900
3.800 4.000 3.800 3.400 3.500
2.900 4.200 3.700 2.500 2.400
4.400 4.700 4.500 2.800 3.600
4.300 4.500 4.600 3.700 4.100
3.800 3.700 4.000 3.300 3.600
3.800 4.500 4.000 3.400 3.700
3.700 5.000 4.500 4.000 4.100
3.700 3.900 3.200 3.300 3.600
%samples 35_SQAM_glockenspiel_cut
%samples Can't Wait Until Tonight (Dry Wurlitzer Mix)
%samples girl
%samples Robots_old
%samples SinceAlways
%samples trumpet
%samples velvet
%samples Waiting
%samples Asleep__4.11-4.31_
%samples Greatest_Love_of_All_2min57
%samples Hey Tonight
%samples Severance__1.31-1.51_
%samples Shinsho_pool_3min45_4min4
%samples SlavesOfFear
%samples The Chastising of Renegade
%samples TrosYGareg
%samples 4-Sound-English-male.441
%samples 9-Have-big-expensive-car.441
%samples 12-German-male-speech.441
%samples 15-Good-evening.441
%samples 21-classic.441
%samples 24-Greensleeves-Korean-male-speech.441
%samples 25-This-is-the-end.441
%samples 27-last-song-drums-and-trampets.441
%samples bonhemian_rhapsody
%samples clapton_44k
%samples Coral
%samples ExitMusic
%samples liberate
%samples NewYorkCity
%samples sandman
%samples take_your_finger_frin_my_head
%samples Changes
%samples Girl_In_The_Fire__Sample_
%samples Hotel California
%samples Jupiter, the Bringer of Jolity
%samples Last_Of_The_Mohicanz__Sample_
%samples Only Time
%samples Through The Fire And Flames
%samples With Love (Outro)
%samples 10xh_
%samples 11ff_
%samples 12at_
%samples 13by_
%samples 14fe_
%samples 15ma_
%samples 16mb_
%samples 17qu_
%samples 18vr_
%samples 19am_
%samples 20tt_
%samples 21wa_
%samples 22ex_
%samples 23ht_
%samples 24td_
%samples 25rb_
%samples 26fr_
%samples 27ud_
%samples 28db_
%samples 29ru_
%samples 30ge_
%samples 31it_
%samples 32on_
%samples 33eg_
%samples 34ti_
%samples bah
%samples bas
%samples cst
%samples fms
%samples glk
%samples hrp
%samples lob
%samples mof
%samples qrt
Bitrates:
Code: [Select]
lame	opus	nero	ffv7	ffv9b
82948 125361 62801 81081 80653
98436 110962 100291 98107 97965
116012 116258 95232 100872 101298
97325 95065 99314 96381 96061
99217 77112 107985 98599 98829
96412 127339 88749 98686 99195
111820 106411 112453 98908 98936
95839 105495 105361 98094 97927
96293 97746 104058 97824 98399
95785 98364 94045 97773 97881
93888 87390 101949 97795 97910
95429 115218 90252 97804 97912
96425 99162 95230 97900 97938
94595 85344 112534 97800 97881
103867 99087 104226 98556 98830
96790 118446 92321 97829 97980
102910 89496 69023 100251 98870
100492 113557 98093 98261 98484
101974 92795 66003 100148 99052
94949 89161 96873 98283 98461
94965 107150 94367 98193 98360
118635 125803 93728 99969 100336
97083 93523 92307 98389 98919
100165 98239 94310 98496 99561
95094 96261 94612 97737 97896
104619 98265 95464 98987 98548
97232 98966 99626 98022 98236
96157 104856 86407 99389 99200
97895 93067 115111 99014 99370
94729 87840 110421 97782 98123
98767 91653 102367 98125 99461
95061 109266 100849 98161 97555
106313 106911 100472 98769 98161
98436 115158 95749 98156 98205
97095 101181 99000 97960 98413
94042 99816 92879 97714 97762
104073 98453 102137 99086 98965
95373 91143 100387 97777 97856
97443 91866 112188 97731 98863
98112 98539 103093 98519 98908
109930 105413 111017 98946 98326
96602 121097 81298 98239 97675
94384 105946 102901 97895 98013
99296 99680 110560 98178 99173
103081 117451 100662 98795 98484
93911 95599 97343 97893 98030
94620 84439 110052 97824 97877
105362 109264 98588 98668 98775
101652 110234 102814 98253 98510
100889 102213 93441 98520 98707
98263 97457 97981 97996 98111
95839 105495 105361 98094 97927
99931 101133 102671 98468 98991
97231 96784 107328 98151 98465
100069 98630 80604 98493 97079
102605 95588 106593 98318 98889
105499 100977 77273 100230 98775
98180 99440 97389 97915 97974
94480 88641 94130 97896 97940
96716 95641 110092 97849 99193
86139 77265 51642 81463 80028
96571 75645 76995 98262 97660
97690 85590 88768 97846 97770
93433 81743 96805 94695 94745
94885 95011 104750 97612 98640
95015 98436 93806 98114 98258
96489 104414 88582 97933 97991
107692 111287 103567 98748 99283
102419 107750 78932 101344 99407
89110 124905 66436 96138 96174
96227 119820 93563 98276 98551
99124 102445 99667 98264 98563
113757 122520 107873 99624 99079
91105 97839 90284 94306 93194

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #1
Try out oggenc aotuv 6.03b with -q 1 --advanced-encode-option impulse_noisetune=-15 --advanced-encode-option bit_reservoir_bias=0 --advanced-encode-option bit_reservoir_bits=4 --advanced-encode-option bitrate_average_damping=0.1

Some or many of the parameters are maybe too much or unnecessary but just try it out and see what you think. It has to be the aotuv encoder if I remember correctly from my tests.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #2
Is anything commercially released in 96kbps anymore?

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #3
AFAIK some radio streams and YouTube audio content are encoded at 96 kbit/s.

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #4
AFAIK some radio streams and YouTube audio content are encoded at 96 kbit/s.

Exactly.
+1


Kamedo2,  Thank You for sharing this.
Question. Why Nero and not something, You know,  of higher quality AAC encoder?
Or it would be too high anchor for ffmpeg encoder?

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #5
Thank you for the test.
I'm pretty impressed that apart from ffmpeg encodings only few samples had a quality of 'slightly annoying' or below,  and those below were not very far from that. Especially Opus shines, but even Lame provided a quality I didn't expect.
lame3995o -Q1.7
opus --bitrate 140

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #6
Question. Why Nero and not something, You know,  of higher quality AAC encoder?
Or it would be too high anchor for ffmpeg encoder?

Yes, Apple and FDK have even better quality. They are too good to be compared with FFmpeg.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #7
Samples used in this test:

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #8


The results, Opus > Nero > LAME > v9b > v7, remain mostly unchanged by set of samples. It means test conclusions will be similar if we use different set of samples.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #9
Thank You, Kenji 

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #10
The results, Opus?>?Nero?>?LAME?>?v9b?>?v7, remain mostly unchanged by set of samples. It means test conclusions will be similar if we use different set of samples.

Yes, similar and mostly unchanged. But sometimes, the rank order changes due to the test set, e.g. using the randomly picked samples -> the only set where LAME has a lower average score than FFmpeg v9b (on all the other sets, LAME is probably significantly better on average than FFmpeg v9b).

Kamedo's critical corpus #10-24 and my test set (of which only a subset is included here) are, by design, collections of critical items, and this test verifies this nicely (lowest mean scores for all codecs, except FFmpeg v7 on my set). So these two sets should be useful for the planned upcoming high-bitrate test.

Edit: Kamedo, the Waiting samples in your and my set are identical and therefore duplicates, right? What about the Velvet(Realm) sample?

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #11
Kamedo2, may I ask you to share these two binaries?
[blockquote]ffmpeg r.70351 with v7 patch applied
ffmpeg r.70351 with v9b patch applied [/blockquote]
I would like to perform some measurements.

Thank you for this test and sharing detailed results which could be used for further research.
keeping audio clear together - soundexpert.org

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #12
Kamedo's critical corpus #10-24 and my test set (of which only a subset is included here) are, by design, collections of critical items, and this test verifies this nicely (lowest mean scores for all codecs, except FFmpeg v7 on my set). So these two sets should be useful for the planned upcoming high-bitrate test.
Yes, I designed the sets to deliver ease of very high-bitrate testing, while sticking to sounds I hear in actual music scene frequently. Glad to see that conclusion is similar to all other sets of samples.

Edit: Kamedo, the Waiting samples in your and my set are identical and therefore duplicates, right? What about the Velvet(Realm) sample?
Yes, the sample "Waiting" is identical. I apparently found another flaw of FFmpeg on the second test, though
Velvet is a percussion sample, while VelvetRealm is a techno sample.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #13
Kamedo2, may I ask you to share these two binaries?
[blockquote]ffmpeg r.70351 with v7 patch applied
ffmpeg r.70351 with v9b patch applied [/blockquote]

listening-test.coresv.net/70351_with_patch.7z (I will remove it within week)
Note that v7, v9b are intermediate WIP patches that have never deployed in official FFmpeg. The improvement of v7 is only partly applied in the current FFmpeg.


 

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #15
@kamedo2: You've already deleted the zip, so I can't test this by myself.

I found a file that sounds really bad with ffmpeg. I've posted it in this thread:

http://www.hydrogenaud.io/forums/index.php?showtopic=109757

Could you try how better (if any) it encodes with v9b ?

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #16
@[JAZ]: I listened to the sample you posted. Nice sound, and the sound illuminates the problem FFmpeg have. But I could not reproduce your findings, using FFmpeg version N-73515-g3526a12 unpatched and N-70351-g2b40416 v9b patched. I am wondering if you are using more than a month old FFmpeg, before improvements were committed.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #17
I indicated the version used in the uploads post :
The ffmpeg version is N-73818-g9ebe041:
avformat Lavf56.40.101
avcodec Lavc56.49.101 libvo_aacenc

So it is newer than the one you say.

I downloaded it directly from ffmpeg.org website.
Build date is  20/07/2015


Edit:
I just tried an older version that I have (7 june 2015, 72709) and shows the exact same problem.  I should mention that I run windows, and a 64bit compile.
Could it be that the patch is not applied with 64bit?


Edit 2:
Woops!  It looks like I was not using the same audiocodec than you.  As it is shown, i was using the "libvo_aacenc", not the "native" aac.

With your settings:  -c:a aac -strict experimental -b:a 128k  it shows:
  Metadata:
    encoder        : Lavf56.40.101
    Stream #0:0: Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder        : Lavc56.49.101 aac

And the dropouts dissapear. (It is not as good as apple's, but notably better than "visualOn")

Edit3:
I found the page for older releases and just tried 20150302 ( 70358 ) : http://ffmpeg.zeranoe.com/builds/win64/static/

I tried this release, and the 73818 and they sound the same to me ( i.e. vo_aacenc is bad on both, and native aac is "good" on both), for this sample.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #18
The bitrate vs MOS scatter graph.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #19
The conclusion, Opus > Nero > Lame, is radically different from SoundExpert's Opus, AAC, Vorbis, MP3 at 96 kbit/s listening test, Nero > Lame = Opus.

Although Opus 64kbps and Nero 64kbps matched Lame 96kbps on the Google MUSHRA listening test in 2011, on the SoundExpert test, Opus 96kbps behaves poorly, and Nero significantly beats Opus.

The biggest discrepancy comes from Glockenspiel and Harpsichord.





Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #21
From opus-codec.org:

Quote
•Version 1.1 released

5 December, 2013

After more than two years of development, we have released Opus 1.1. This includes:
◦ new analysis code and tuning that significantly improves encoding quality, especially for variable-bitrate (VBR),

This was done to improve tonal items by pumping bits into them IIRC.

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #22
Also for Lame there's a difference between --abr 98 and -V 6.9. At 96 kbps Lame ABR is said to be superior over VBR.
lame3995o -Q1.7
opus --bitrate 140



 
SimplePortal 1.0.0 RC1 © 2008-2019