Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of MP3/Opus/AAC at 96kbps (Read 26667 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Abstract:
Blind sound quality comparison between LAME 3.99.5 abr, Opus 1.1 vbr, NeroAACEnc vbr, FFmpeg's native encoder with intermediate patch v7 abr, FFmpeg's native encoder with intermediate patch v9b abr, all at 96kbps.

Encoders:
LAME 3.99.5
Opus 1.1 with opus-tools-0.1.9, win32
NeroAACEnc 1.5.4.0
ffmpeg r.70351 with v7 patch applied
ffmpeg r.70351 with v9b patch applied

Settings:
lame --abr 98 -S in.wav out.mp3
opusenc --bitrate 91 in.wav out.opus
NeroAacEnc -q 0.333 -if in.wav -of out.mp4
ffmpeg70351_v7 -y -i in.wav -c:a aac -strict experimental -b:a 96k out.mp4
ffmpeg70351_v9b -y -i in.wav -c:a aac -strict experimental -b:a 96k out.mp4

Samples:
Total 74 samples.
40 samples from 2014 public listening test
25 samples from my corpus
9 samples from SoundExpert

Hardwares:
Sony PSP-3000 + RP-HJE150.

Results:



Conclusions & Observations:
Opus had the best score. Nero was the second, LAME was third, and the FFmpeg's native AAC encoder was the last. In the FFmpeg's native AAC encoder, v9b patch, which enables M/S stereo by default, certainly improves audio quality.
The result was consistent over three set of samples.
Code: [Select]
lame opus nero ffv7 ffv9b
3.65 4.36 3.98 3.20 3.45 - 40 samples from the 2014 public listening test.
3.60 4.21 3.80 2.94 3.32 - 25 samples from my corpus.
3.80 4.39 4.08 3.33 3.61 - 9 samples from SoundExpert.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) [url=http://ff123.net/]http://ff123.net/[/url]
Blocked ANOVA analysis

Number of listeners: 74
Critical significance:  0.05
Significance of data: 1.11E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              369         134.26
Testers (blocks)    73          45.11
Codecs eval'd        4          61.69   15.42   164.04  1.11E-016
Error              292          27.45    0.09
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.099

Means:

opus     nero     lame     ffv9b    ffv7
  4.31     3.93     3.65     3.42     3.13

---------------------------- p-value Matrix ---------------------------

         nero     lame     ffv9b    ffv7
opus     0.000*   0.000*   0.000*   0.000*
nero              0.000*   0.000*   0.000*
lame                       0.000*   0.000*
ffv9b                               0.000*
-----------------------------------------------------------------------

opus is better than nero, lame, ffv9b, ffv7
nero is better than lame, ffv9b, ffv7
lame is better than ffv9b, ffv7
ffv9b is better than ffv7
Raw data:
Code: [Select]
% 96kbps ABC/HR Score
% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
lame	opus	nero	ffv7	ffv9b
4.300	4.500	4.400	3.900	3.400	
3.200	3.700	3.500	3.100	2.700	
3.200	3.600	2.800	2.000	2.400	
3.200	4.300	4.100	3.400	3.500	
3.600	5.000	3.800	3.400	2.800	
3.600	5.000	3.900	2.400	2.500	
3.200	3.500	3.700	2.800	2.700	
3.300	4.300	3.800	3.500	3.700	
3.800	5.000	5.000	3.600	3.900	
3.700	4.100	3.600	3.100	3.700	
3.100	3.800	3.500	2.600	3.300	
3.800	4.200	3.900	3.400	3.700	
3.000	4.300	3.600	3.400	3.200	
4.400	5.000	5.000	4.600	4.000	
3.200	5.000	4.000	3.000	3.700	
3.700	4.400	4.000	3.400	4.100	
4.300	5.000	4.400	2.900	3.900	
3.000	3.900	3.600	3.200	3.200	
3.700	4.000	3.200	2.400	3.000	
3.900	4.000	3.800	3.000	3.500	
3.600	5.000	4.000	3.300	3.700	
3.800	4.600	4.100	2.800	3.100	
3.800	4.400	4.100	3.400	3.600	
3.300	5.000	4.300	3.100	3.900	
4.200	4.300	4.100	3.300	3.900	
3.900	4.200	4.100	3.300	3.700	
4.000	5.000	4.000	3.300	3.700	
3.900	4.100	3.500	2.200	3.100	
2.800	4.500	4.200	3.200	3.300	
3.600	4.300	4.200	3.200	3.500	
3.300	3.700	3.600	2.700	3.000	
4.000	3.800	3.900	3.400	3.700	
4.100	4.400	4.500	3.200	3.800	
4.100	4.400	4.000	3.600	3.700	
3.500	4.200	3.800	3.100	3.600	
3.800	4.100	4.400	4.500	4.300	
4.000	4.400	4.000	3.100	2.900	
3.200	4.200	3.900	3.600	3.800	
4.700	5.000	5.000	3.600	3.500	
3.300	4.300	3.900	3.100	3.200	
2.600	3.500	2.900	2.100	2.000	
4.200	4.500	4.300	3.200	3.800	
3.200	4.300	3.600	3.500	3.300	
3.000	4.200	3.900	2.500	3.200	
3.600	4.400	3.900	2.700	3.100	
2.900	3.200	3.500	2.900	3.300	
4.000	4.400	4.200	3.500	2.700	
3.500	4.200	3.600	3.000	3.600	
3.600	3.900	4.000	3.000	3.200	
3.600	4.300	4.200	2.600	3.800	
2.700	4.000	3.300	2.500	2.900	
3.200	4.200	3.600	3.000	3.400	
4.000	4.100	3.800	3.300	3.500	
3.700	4.600	4.300	2.900	3.300	
3.500	4.000	3.200	2.600	2.900	
3.100	3.900	3.800	2.900	3.300	
4.000	4.400	3.700	2.800	3.500	
3.800	4.100	4.200	2.700	3.300	
4.200	5.000	4.500	3.600	3.400	
4.100	4.500	3.300	3.000	3.500	
3.400	4.200	3.600	2.700	3.300	
3.900	4.100	4.100	2.800	3.500	
4.100	4.500	3.800	2.800	3.400	
4.100	4.300	4.000	3.600	3.900	
4.000	4.500	3.700	3.400	3.800	
3.800	5.000	4.400	3.600	3.900	
3.800	4.000	3.800	3.400	3.500	
2.900	4.200	3.700	2.500	2.400	
4.400	4.700	4.500	2.800	3.600	
4.300	4.500	4.600	3.700	4.100	
3.800	3.700	4.000	3.300	3.600	
3.800	4.500	4.000	3.400	3.700	
3.700	5.000	4.500	4.000	4.100	
3.700	3.900	3.200	3.300	3.600	
%samples	35_SQAM_glockenspiel_cut
%samples	Can't Wait Until Tonight (Dry Wurlitzer Mix)
%samples	girl
%samples	Robots_old
%samples	SinceAlways
%samples	trumpet
%samples	velvet
%samples	Waiting
%samples	Asleep__4.11-4.31_
%samples	Greatest_Love_of_All_2min57
%samples	Hey Tonight
%samples	Severance__1.31-1.51_
%samples	Shinsho_pool_3min45_4min4
%samples	SlavesOfFear
%samples	The Chastising of Renegade
%samples	TrosYGareg
%samples	4-Sound-English-male.441
%samples	9-Have-big-expensive-car.441
%samples	12-German-male-speech.441
%samples	15-Good-evening.441
%samples	21-classic.441
%samples	24-Greensleeves-Korean-male-speech.441
%samples	25-This-is-the-end.441
%samples	27-last-song-drums-and-trampets.441
%samples	bonhemian_rhapsody
%samples	clapton_44k
%samples	Coral
%samples	ExitMusic
%samples	liberate
%samples	NewYorkCity
%samples	sandman
%samples	take_your_finger_frin_my_head
%samples	Changes
%samples	Girl_In_The_Fire__Sample_
%samples	Hotel California
%samples	Jupiter, the Bringer of Jolity
%samples	Last_Of_The_Mohicanz__Sample_
%samples	Only Time
%samples	Through The Fire And Flames
%samples	With Love (Outro)
%samples	10xh_
%samples	11ff_
%samples	12at_
%samples	13by_
%samples	14fe_
%samples	15ma_
%samples	16mb_
%samples	17qu_
%samples	18vr_
%samples	19am_
%samples	20tt_
%samples	21wa_
%samples	22ex_
%samples	23ht_
%samples	24td_
%samples	25rb_
%samples	26fr_
%samples	27ud_
%samples	28db_
%samples	29ru_
%samples	30ge_
%samples	31it_
%samples	32on_
%samples	33eg_
%samples	34ti_
%samples	bah
%samples	bas
%samples	cst
%samples	fms
%samples	glk
%samples	hrp
%samples	lob
%samples	mof
%samples	qrt
Bitrates:
Code: [Select]
lame	opus	nero	ffv7	ffv9b
82948	125361	62801	81081	80653
98436	110962	100291	98107	97965
116012	116258	95232	100872	101298
97325	95065	99314	96381	96061
99217	77112	107985	98599	98829
96412	127339	88749	98686	99195
111820	106411	112453	98908	98936
95839	105495	105361	98094	97927
96293	97746	104058	97824	98399
95785	98364	94045	97773	97881
93888	87390	101949	97795	97910
95429	115218	90252	97804	97912
96425	99162	95230	97900	97938
94595	85344	112534	97800	97881
103867	99087	104226	98556	98830
96790	118446	92321	97829	97980
102910	89496	69023	100251	98870
100492	113557	98093	98261	98484
101974	92795	66003	100148	99052
94949	89161	96873	98283	98461
94965	107150	94367	98193	98360
118635	125803	93728	99969	100336
97083	93523	92307	98389	98919
100165	98239	94310	98496	99561
95094	96261	94612	97737	97896
104619	98265	95464	98987	98548
97232	98966	99626	98022	98236
96157	104856	86407	99389	99200
97895	93067	115111	99014	99370
94729	87840	110421	97782	98123
98767	91653	102367	98125	99461
95061	109266	100849	98161	97555
106313	106911	100472	98769	98161
98436	115158	95749	98156	98205
97095	101181	99000	97960	98413
94042	99816	92879	97714	97762
104073	98453	102137	99086	98965
95373	91143	100387	97777	97856
97443	91866	112188	97731	98863
98112	98539	103093	98519	98908
109930	105413	111017	98946	98326
96602	121097	81298	98239	97675
94384	105946	102901	97895	98013
99296	99680	110560	98178	99173
103081	117451	100662	98795	98484
93911	95599	97343	97893	98030
94620	84439	110052	97824	97877
105362	109264	98588	98668	98775
101652	110234	102814	98253	98510
100889	102213	93441	98520	98707
98263	97457	97981	97996	98111
95839	105495	105361	98094	97927
99931	101133	102671	98468	98991
97231	96784	107328	98151	98465
100069	98630	80604	98493	97079
102605	95588	106593	98318	98889
105499	100977	77273	100230	98775
98180	99440	97389	97915	97974
94480	88641	94130	97896	97940
96716	95641	110092	97849	99193
86139	77265	51642	81463	80028
96571	75645	76995	98262	97660
97690	85590	88768	97846	97770
93433	81743	96805	94695	94745
94885	95011	104750	97612	98640
95015	98436	93806	98114	98258
96489	104414	88582	97933	97991
107692	111287	103567	98748	99283
102419	107750	78932	101344	99407
89110	124905	66436	96138	96174
96227	119820	93563	98276	98551
99124	102445	99667	98264	98563
113757	122520	107873	99624	99079
91105	97839	90284	94306	93194

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #1 –
Try out oggenc aotuv 6.03b with -q 1 --advanced-encode-option impulse_noisetune=-15 --advanced-encode-option bit_reservoir_bias=0 --advanced-encode-option bit_reservoir_bits=4 --advanced-encode-option bitrate_average_damping=0.1

Some or many of the parameters are maybe too much or unnecessary but just try it out and see what you think. It has to be the aotuv encoder if I remember correctly from my tests.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #2 –
Is anything commercially released in 96kbps anymore?

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #3 –
AFAIK some radio streams and YouTube audio content are encoded at 96 kbit/s.

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #4 –
Quote from: C.R.Helmrich on
AFAIK some radio streams and YouTube audio content are encoded at 96 kbit/s.

Exactly.
+1


Kamedo2,  Thank You for sharing this.
Question. Why Nero and not something, You know,  of higher quality AAC encoder?
Or it would be too high anchor for ffmpeg encoder?

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #5 –
Thank you for the test.
I'm pretty impressed that apart from ffmpeg encodings only few samples had a quality of 'slightly annoying' or below,  and those below were not very far from that. Especially Opus shines, but even Lame provided a quality I didn't expect.
lame3995o -Q1.7 --lowpass 17

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #6 –
Quote from: IgorC on
Question. Why Nero and not something, You know,  of higher quality AAC encoder?
Or it would be too high anchor for ffmpeg encoder?

Yes, Apple and FDK have even better quality. They are too good to be compared with FFmpeg.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #7 –
Samples used in this test:

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #8 –


The results, Opus　>　Nero　>　LAME　>　v9b　>　v7, remain mostly unchanged by set of samples. It means test conclusions will be similar if we use different set of samples.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #9 –
Thank You, Kenji 

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #10 –
Quote from: Kamedo2 on
The results, Opus?>?Nero?>?LAME?>?v9b?>?v7, remain mostly unchanged by set of samples. It means test conclusions will be similar if we use different set of samples.

Yes, similar and mostly unchanged. But sometimes, the rank order changes due to the test set, e.g. using the randomly picked samples -> the only set where LAME has a lower average score than FFmpeg v9b (on all the other sets, LAME is probably significantly better on average than FFmpeg v9b).

Kamedo's critical corpus #10-24 and my test set (of which only a subset is included here) are, by design, collections of critical items, and this test verifies this nicely (lowest mean scores for all codecs, except FFmpeg v7 on my set). So these two sets should be useful for the planned upcoming high-bitrate test.

Edit: Kamedo, the Waiting samples in your and my set are identical and therefore duplicates, right? What about the Velvet(Realm) sample?

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #11 –
Kamedo2, may I ask you to share these two binaries?
[blockquote]ffmpeg r.70351 with v7 patch applied
ffmpeg r.70351 with v9b patch applied [/blockquote]
I would like to perform some measurements.

Thank you for this test and sharing detailed results which could be used for further research.
keeping audio clear together - soundexpert.org

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #12 –
Quote from: C.R.Helmrich on
Kamedo's critical corpus #10-24 and my test set (of which only a subset is included here) are, by design, collections of critical items, and this test verifies this nicely (lowest mean scores for all codecs, except FFmpeg v7 on my set). So these two sets should be useful for the planned upcoming high-bitrate test.
Yes, I designed the sets to deliver ease of very high-bitrate testing, while sticking to sounds I hear in actual music scene frequently. Glad to see that conclusion is similar to all other sets of samples.

Quote from: C.R.Helmrich on
Edit: Kamedo, the Waiting samples in your and my set are identical and therefore duplicates, right? What about the Velvet(Realm) sample?
Yes, the sample "Waiting" is identical. I apparently found another flaw of FFmpeg on the second test, though
Velvet is a percussion sample, while VelvetRealm is a techno sample.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #13 –
Quote from: Serge Smirnoff on
Kamedo2, may I ask you to share these two binaries?
[blockquote]ffmpeg r.70351 with v7 patch applied
ffmpeg r.70351 with v9b patch applied [/blockquote]

listening-test.coresv.net/70351_with_patch.7z (I will remove it within week)
Note that v7, v9b are intermediate WIP patches that have never deployed in official FFmpeg. The improvement of v7 is only partly applied in the current FFmpeg.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #15 –
@kamedo2: You've already deleted the zip, so I can't test this by myself.

I found a file that sounds really bad with ffmpeg. I've posted it in this thread:

http://www.hydrogenaud.io/forums/index.php?showtopic=109757

Could you try how better (if any) it encodes with v9b ?

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #16 –
@[JAZ]: I listened to the sample you posted. Nice sound, and the sound illuminates the problem FFmpeg have. But I could not reproduce your findings, using FFmpeg version N-73515-g3526a12 unpatched and N-70351-g2b40416 v9b patched. I am wondering if you are using more than a month old FFmpeg, before improvements were committed.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #17 –
I indicated the version used in the uploads post :
The ffmpeg version is N-73818-g9ebe041:
avformat Lavf56.40.101
avcodec Lavc56.49.101 libvo_aacenc

So it is newer than the one you say.

I downloaded it directly from ffmpeg.org website.
Build date is  20/07/2015


Edit:
I just tried an older version that I have (7 june 2015, 72709) and shows the exact same problem.  I should mention that I run windows, and a 64bit compile.
Could it be that the patch is not applied with 64bit?


Edit 2:
Woops!  It looks like I was not using the same audiocodec than you.  As it is shown, i was using the "libvo_aacenc", not the "native" aac.

With your settings:  -c:a aac -strict experimental -b:a 128k  it shows:
  Metadata:
    encoder        : Lavf56.40.101
    Stream #0:0: Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder        : Lavc56.49.101 aac

And the dropouts dissapear. (It is not as good as apple's, but notably better than "visualOn")

Edit3:
I found the page for older releases and just tried 20150302 ( 70358 ) : http://ffmpeg.zeranoe.com/builds/win64/static/

I tried this release, and the 73818 and they sound the same to me ( i.e. vo_aacenc is bad on both, and native aac is "good" on both), for this sample.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #18 –
The bitrate vs MOS scatter graph.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #19 –
The conclusion, Opus > Nero > Lame, is radically different from SoundExpert's Opus, AAC, Vorbis, MP3 at 96 kbit/s listening test, Nero > Lame = Opus.

Although Opus 64kbps and Nero 64kbps matched Lame 96kbps on the Google MUSHRA listening test in 2011, on the SoundExpert test, Opus 96kbps behaves poorly, and Nero significantly beats Opus.

The biggest discrepancy comes from Glockenspiel and Harpsichord.



Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #21 –
From opus-codec.org:

Quote
•Version 1.1 released

5 December, 2013

After more than two years of development, we have released Opus 1.1. This includes:
◦ new analysis code and tuning that significantly improves encoding quality, especially for variable-bitrate (VBR),

This was done to improve tonal items by pumping bits into them IIRC.

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #22 –
Also for Lame there's a difference between --abr 98 and -V 6.9. At 96 kbps Lame ABR is said to be superior over VBR.
lame3995o -Q1.7 --lowpass 17

Personal Listening Test of MP3/Opus/AAC at 96kbps

Reply #24 –
Quote from: Serge Smirnoff on
Quote from: Kamedo2 on
Samples:
Total 74 samples.
40 samples from 2014 public listening test
25 samples from my corpus
9 samples from SoundExpert


May I ask you also to put somewhere for downloading the 25 samples from your corpus? In lossless.

https://www.hydrogenaud.io/forums/index.php?showtopic=98003
or
http://listening-test.coresv.net/25samples.lzma.7z
or
http://zak.s206.xrea.com/bitratetest/bitra...st_wav10-14.zip
http://zak.s206.xrea.com/bitratetest/bitra...st_wav15-19.zip
http://zak.s206.xrea.com/bitratetest/bitra...st_wav20-24.zip
http://zak.s206.xrea.com/bitratetest/bitra...st_wav25-29.zip
http://zak.s206.xrea.com/bitratetest/bitra...st_wav30-34.zip