Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal blind listening test – MultiCodec at ~192 VBR kbps (Read 50330 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

Personal blind listening test – MultiCodec at ~192 VBR kbps

Hello, Everyone!  :)

It's my personal blind test – MultiCodec at ~192 VBR kbps

All versions of encoders were the newest and stable
LAME MP3 3.100 -V2 (~192 kbps)
Aple LC-AAC 7.10.9.0 via QAAC TVBR q91  (~192 kbps)
xHE-AAC 1.0.7.0 -m 9 (~192 kbps)
Opus 1.3.1 -b 182 (yes, 182, not 192 kbps, that’s way Opus stays on par with all other encoders during bitrate verifitacion on a big number of albums)
Musepack (MPC Encoder 1.30.0 –-stable)   Q5.5 (~192 kbps)
Vorbis Aotuv “OGG” 6.03 -q6 (~192 kbps)

Sampling rate treatment for Opus codec (44.1/48kHz)
Encoding:Original .WAV 44.1 kHz → foobar2000’s dbpoweramp SSRC resampling 48 kHz (onfly feed)→ Opus file 48 kHz  
Decoding:Opus file 48 kHz → SSRC 44.1 kHz .WAV

Hardware
Source: Benchmark DAC1 with built-in headphone AMP HPA2 (24bits/44.1kHz mode)
Headpones: Sennheiser HD 800

Files
LINK

            RESULTS



            ~192 kbps
             Opus - 5.00 (fully transparent)
            Musepack - 4.95
            Apple LC-AAC - 4.94
            Vorbis Aotuv "OGG" - 4.90
            xHE-AAC - 4.86
            LAME MP3 - 4.46



Alternative graph of the results
https://i.ibb.co/vwWR8T7/Results-192-kbps.png


Conclusions
... empty spaces...

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #1
I'm curious, why you used ssrc resampler when opus use internal resampler? Is ssrc better quality?
Otherwise, excellent work! It's very nice to see opus being fully transparent at 192 kbps.
lame --abr 288 -f --lowpass 17 (+ mp3gain@92 dB)

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #2
My turn to say it, Igor: nice chart!  ;)
Listen to the music, not the media it's on.
União e reconstrução

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #3
I want to make a graph to my own listening test. Please tell me which software you have used to generate this graph. 

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #4
Great test. I'm really surprised with Musepack.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #5
I'm curious, why you used ssrc resampler when opus use internal resampler? Is ssrc better quality?
Otherwise, excellent work! It's very nice to see opus being fully transparent at 192 kbps.
I haven't any issue with native resampler in Opus in the past. The main reason to use SSRC here was to keep Opus output at 44.1 kHz. 

My turn to say it, Igor: nice chart!  ;)
Thank You! I didn't forget about another chart in poll 2020.  Will see it now.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #6
I want to make a graph to my own listening test. Please tell me which software you have used to generate this graph. 
It's Kamedo2's work https://listening-test.coresv.net/graphmaker5.htm

Great test. I'm really surprised with Musepack.
Uhm. In which way exactly?

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #7
I want to make a graph to my own listening test. Please tell me which software you have used to generate this graph. 
It's Kamedo2's work https://listening-test.coresv.net/graphmaker5.htm

Great test. I'm really surprised with Musepack.
Uhm. In which way exactly?

That it performed really well at 192Kbps. I always thought Musepack needed more bitrate to sound transparent.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #8
I was under  impression that I probably wasn't able to hear any artifact on Musepack at 192 kbps.
Until now.

Musepack has transparent transients but it’s very slightly noisy on high frequencies of tonality.
Its high frequency muffling which is perceived as very slightly dull + flat noise addition.

Anyway those are very minor issues which were detected during critical listening. I won't be able to hear them on some average session of just enjoying a music.  :)


Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #9
It should have been a hard test, Great work!

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #10
Anyway those are very minor issues which were detected during critical listening. I won't be able to hear them on some average session of just enjoying a music.  :)
 
 Specially, in your specific case, if we're talking about --quality 6 or up, as you used --quality 5.5 for your test, right?
Listen to the music, not the media it's on.
União e reconstrução

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #11
I reordered the encoders by score.



Code: [Select]
MP3	xHE-AAC	Vorbis	AAC-LC	Musepack	Opus
3.5 5.0 5.0 5.0 5.0 5.0
4.5 4.0 4.6 5.0 5.0 5.0
3.8 4.7 4.8 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.2 4.8 4.9 4.9 4.9 5.0
4.2 4.9 4.9 5.0 5.0 5.0
4.8 5.0 5.0 4.7 4.9 5.0
5.0 5.0 5.0 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.7 5.0 4.8 5.0 5.0 5.0
4.8 5.0 5.0 4.9 4.8 5.0
4.0 4.9 4.8 4.8 4.8 5.0
%feature 10 LAME 3.100 exhale 1.0.7 Aotuv 6.03 Apple CoreAudioToolbox.dll 7.10.9.0 MPC Encoder 1.30.0 --stable Opus 1.3.1
%feature 11 -V2 -m 9 -q6 TVBR q91 Q5.5 -b 182
%feature 12 Average:4.46 4.86 4.90 4.94 4.95 5.00
%samples 01 Castanets
%samples 02 Fatboy
%samples 03 EIG
%samples 04 Bachpsichord
%samples 05 Enola
%samples 06 Trumpet
%samples 07 applaud
%samples 08 Velvet
%samples 09 Linchpin
%samples 10 Angels_Fall_First_ringing
%samples 11 You look good to me
%samples 12 Berling Drug

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #12
I know it's a pointless and largely redundant observation, but if xHE-AAC didn't bomb on Fatboy and got a 5 like AAC-LC, Musepack and Opus, it would've just come third. Again, we could make similar "what ifs" observations about all the encoders. It's just that "killer sample" - from your personal listening test - is not handled well by xHE-AAC compared to how well it handled all the other audio samples.

I'm not familiar enough - nor can remember from tests years ago - what the "killer sample" property Fatboy possess. Anyone?

EDIT: thanks for the test and the clear presentation :-)

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #13
My thoughts:

  • As expected, all of those post-MP3 codecs offer very high fidelity on 192kbps.
  • Opus is indeed a mighty and stable codec. Even with his trained ear and twelve critical samples, Opus is transparent on 192k.
  • Musepack is unexpectedly performing well on 192kbps.
  • Apple AAC is nearly transparent in 192kbps, like in my 2011 personal test
  • xHE-AAC and AAC-LC performs equally well on high rates, except for the Fatboy. Fatboy scored 3.5 in my 96kbps test, and 4.0 in IgorC's 192kbps test.
  • MP3 192kbps occasionally drops below 4.0 on hard samples, which is consistent with my tests.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #14
It should have been a hard test, Great work!
Thank you, K.  :)

Specially, in your specific case, if we're talking about --quality 6 or up, as you used --quality 5.5 for your test, right?
Well, higher bitrate implies higher transparency for any encoder. 
I've used --quality 5.5 because it produced  ~192 kbps  and was  on par with other encoders.

I know it's a pointless and largely redundant observation...
I agree. But it's ok.

but if xHE-AAC didn't bomb on Fatboy and got a 5 like AAC-LC, Musepack and Opus, it would've just come third. Again, we could make similar "what ifs" observations about all the encoders. It's just that "killer sample" - from your personal listening test - is not handled well by xHE-AAC compared to how well it handled all the other audio samples.
I see your point. And Yes, we are here basically speculating.
I'll try to put it this way. "What if" Opus was being lucky on  those difficult samples?  ... bun on all of them?  
It's very unlikely that this is just a coincidence or "bad luck/good luck" for some encoder. 
And it's not just a Fatboy sample if we take a look at a  distribution of scores on another samples. 

I'm not familiar enough - nor can remember from tests years ago - what the "killer sample" property Fatboy possess. Anyone?
Sharp transients

EDIT: thanks for the test and the clear presentation :-)
Thank You  :)

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #15
  • Musepack is unexpectedly performing well on 192kbps.
 
 

As I understood on 192kbps average (as we all know, MPC only does pure VBR)  as Igor has actually used something slightly above "standard" (--quality 5.5.).

Edit: well, he replied just before I posted this, so there you go.
Listen to the music, not the media it's on.
União e reconstrução

 

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #16
@includemeout
I've calibrated bitrates on a large number of albums of different genres.

MPC Q5 - 170 kbps - too small
MPC Q6- 210 kbps - was already too large.

MPC Q5.5 hited ~ 192 kbps.

You can also see that MPC had used more bits on tested samples than LC-AAC/MP3/Opus/xHE-AAC.
So bitrate setting for Musepack was more than fair.  ;)

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #17
@jaybeee
Also exhale xHE-AAC was the only encoder which was tested with CVBR mode (constrained VBR) while all other competitors were tested in their VBR modes.
exhale only supports CVBR.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #18
@includemeout
I've calibrated bitrates on a large number of albums of different genres.

MPC Q5 - 170 kbps - too small
MPC Q6- 210 kbps - was already too large.

MPC Q5.5 hited ~ 192 kbps.

You can also see that MPC had used more bits on tested samples than LC-AAC/MP3/Opus/xHE-AAC.
So bitrate setting for Musepack was more than fair.  ;)
 
 Thanks. I got that. It's just that I wanted to make sure it was business as usual with MPC (well, in fact higher than "standard", but only slightly) and not some unheard of performance as I may have read into Kamedo's post, but I'll wait for his reply.
Listen to the music, not the media it's on.
União e reconstrução

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #19
No love for faac/faad2? It was the fastest encoder/decoder for me but I didn't like how it sounded even at the highest quality. I hoped to know other people's perception.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #20
No love for faac/faad2? It was the fastest encoder/decoder for me but I didn't like how it sounded even at the highest quality. I hoped to know other people's perception.
I think it might score about the same as MP3

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #21
Thanks very much, Igor, for this high-bit-rate test! It is very helpful for me, as the developer of exhale, since my hearing is not good enough anymore for testing such high rates myself. My 2 cents:

First cent
but if xHE-AAC didn't bomb on Fatboy and got a 5 like AAC-LC, Musepack and Opus, it would've just come third. Again, we could make similar "what ifs" observations about all the encoders. It's just that "killer sample" - from your personal listening test - is not handled well by xHE-AAC compared to how well it handled all the other audio samples.
I see your point. And Yes, we are here basically speculating.
I'll try to put it this way. "What if" Opus was being lucky on  those difficult samples?  ... bun on all of them?  
IMHO these are very good observations and discussions, and both issues (one lower-quality score on exhale, all-identical scores on Opus) are the two extremes of the same issue (outliers), which are addressed by proper statistical analysis. For those not familiar with such things:

When your statistical sample size is small (as is the case here, 1 listener times 12 samples), it is important to consider possible effects caused by the small sample size, in order to avoid drawing conclusions which are not, let's say, statistically solid. Luckily, it's very easy to do a "robust" statistical analysis: there's a publicly available tool called Friedman by ff123 which e.g. Kamedo2 regularly uses on his personal tests. I fed this tool with a text file containing the first 13 lines of the text box that Kamedo2 posted yesterday:
Code: [Select]
MP3	xHE-AAC	Vorbis	AAC-LC	Musepack	Opus
3.5 5.0 5.0 5.0 5.0 5.0
4.5 4.0 4.6 5.0 5.0 5.0
3.8 4.7 4.8 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.2 4.8 4.9 4.9 4.9 5.0
4.2 4.9 4.9 5.0 5.0 5.0
4.8 5.0 5.0 4.7 4.9 5.0
5.0 5.0 5.0 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.7 5.0 4.8 5.0 5.0 5.0
4.8 5.0 5.0 4.9 4.8 5.0
4.0 4.9 4.8 4.8 4.8 5.0
No matter which option of Friedman.exe I choose (blocked ANOVA, Friedman/Fischer, Tukey's HSD), I always get this report:

AAC-LC is better than MP3
Musepack is better than MP3
Opus is better than MP3
Vorbis is better than MP3
xHE-AAC is better than MP3

Note that it does not say that Opus is better than xHE-AAC or Vorbis, and Kamedo2 privately confirmed to me that I'm using the tool correctly. That's what I mean with "robust statistical analysis". Essentially, Friedman.exe's comments on your discussion could be phrased as: yes, Opus may be "lucky" on at least one sample, and exhale may be "unlucky" on at least one sample (though it can't say which sample). It also means that the plots should be taken with a grain of salt, which is often the case. As Garf once put it on this forum: "Basically, the graphics suck, but they look cute" ;)

Usually, when you add more samples (like Guruboolez and Kamedo2 did) or listeners (as was done in HA's recent public tests), those issues tend to disappear. Nonetheless, exhale's quality on Fatboy could be a bit higher even to my ears, which brings me to my

Second cent
Indeed, Igor, the reason is an excessive chain of transients on the first half of Fatboy. A few weeks ago I had a possible solution for this, but at the lower bit-rates it degraded the audio quality on some other samples, so I decided not to follow up on that issue. And, as Kamedo2 mentioned, at 96 kbit/s, where the overall audio quality is lower, the score on Fatboy doesn't degrade much more (3.5), see his personal test.

So I could try to "fix" Fatboy only for CVBR mode 8 or 9, but are people really using these modes much? This would be a fix specific to the first few seconds of that single sample, I don't know any other sample where this issue occurs. If so, I could try to come up with a "test.m4a" file to check if the fix works on the weekend.

Chris
If I don't reply to your reply, it means I agree with you.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #22
@C.R.Helmrich & @IgorC: thank you for your comments and thoughts on my observation, which I gladly now agree wasn't a redundant one  :))

Clearly in the grand scheme of things such killer samples aren't that common, esp the Fatboy one, and so you're right to wonder whether it's worth trying to fix it, which could negatively impact other audio samples. Of course, these samples are - as evidenced here - very useful for listening tests though.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #23
Very Interesting Igor, thank you for your efforts.

Re: Personal blind listening test – MultiCodec at ~192 VBR kbps

Reply #24
Good effort.
I 'm wondering if the results will be the same if the test is realized with hifi speakers. There is more space de analyze the sounds.

And what about stereo separation in 2 planes.
What codec preserves  most the position of instrument between right and left and from up to down.