Hydrogenaudio Forums

Hydrogenaudio Forum => Listening Tests => Topic started by: IgorC on 2020-10-07 05:45:36

Title: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 05:45:36
Hello, Everyone!  :)

It's my personal blind test – MultiCodec at ~192 VBR kbps

All versions of encoders were the newest and stable
LAME MP3 3.100 -V2 (~192 kbps)
Aple LC-AAC 7.10.9.0 via QAAC TVBR q91  (~192 kbps)
xHE-AAC 1.0.7.0 -m 9 (~192 kbps)
Opus 1.3.1 -b 182 (yes, 182, not 192 kbps, that’s way Opus stays on par with all other encoders during bitrate verifitacion on a big number of albums)
Musepack (MPC Encoder 1.30.0 –-stable)   Q5.5 (~192 kbps)
Vorbis Aotuv “OGG” 6.03 -q6 (~192 kbps)

Sampling rate treatment for Opus codec (44.1/48kHz)
Encoding:Original .WAV 44.1 kHz → foobar2000’s dbpoweramp SSRC resampling 48 kHz (onfly feed)→ Opus file 48 kHz  
Decoding:Opus file 48 kHz → SSRC 44.1 kHz .WAV

Hardware
Source: Benchmark DAC1 with built-in headphone AMP HPA2 (24bits/44.1kHz mode)
Headpones: Sennheiser HD 800

Files
LINK (https://drive.google.com/file/d/1rXORkWWacwQV9jjxPY0GjPY-Jx9QYfBx/view?usp=sharing)

            RESULTS

(https://i.ibb.co/6WtTscQ/Results-192-kbps-Closeup.png)


            ~192 kbps
             Opus - 5.00 (fully transparent)
            Musepack - 4.95
            Apple LC-AAC - 4.94
            Vorbis Aotuv "OGG" - 4.90
            xHE-AAC - 4.86
            LAME MP3 - 4.46



Alternative graph of the results
https://i.ibb.co/vwWR8T7/Results-192-kbps.png


Conclusions
... empty spaces...
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: synclagz on 2020-10-07 13:31:57
I'm curious, why you used ssrc resampler when opus use internal resampler? Is ssrc better quality?
Otherwise, excellent work! It's very nice to see opus being fully transparent at 192 kbps.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: includemeout on 2020-10-07 14:23:26
My turn to say it, Igor: nice chart!  ;)
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: MiGamer5 on 2020-10-07 17:05:38
I want to make a graph to my own listening test. Please tell me which software you have used to generate this graph. 
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: LithosZA on 2020-10-07 17:10:06
Great test. I'm really surprised with Musepack.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 17:11:12
I'm curious, why you used ssrc resampler when opus use internal resampler? Is ssrc better quality?
Otherwise, excellent work! It's very nice to see opus being fully transparent at 192 kbps.
I haven't any issue with native resampler in Opus in the past. The main reason to use SSRC here was to keep Opus output at 44.1 kHz. 

My turn to say it, Igor: nice chart!  ;)
Thank You! I didn't forget about another chart in poll 2020.  Will see it now.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 17:15:00
I want to make a graph to my own listening test. Please tell me which software you have used to generate this graph. 
It's Kamedo2's work https://listening-test.coresv.net/graphmaker5.htm

Great test. I'm really surprised with Musepack.
Uhm. In which way exactly?
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: LithosZA on 2020-10-07 17:19:28
I want to make a graph to my own listening test. Please tell me which software you have used to generate this graph. 
It's Kamedo2's work https://listening-test.coresv.net/graphmaker5.htm

Great test. I'm really surprised with Musepack.
Uhm. In which way exactly?

That it performed really well at 192Kbps. I always thought Musepack needed more bitrate to sound transparent.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 17:30:16
I was under  impression that I probably wasn't able to hear any artifact on Musepack at 192 kbps.
Until now.

Musepack has transparent transients but it’s very slightly noisy on high frequencies of tonality.
Its high frequency muffling which is perceived as very slightly dull + flat noise addition.

Anyway those are very minor issues which were detected during critical listening. I won't be able to hear them on some average session of just enjoying a music.  :)

Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: Kamedo2 on 2020-10-07 19:05:56
It should have been a hard test, Great work!
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: includemeout on 2020-10-07 19:10:46
Anyway those are very minor issues which were detected during critical listening. I won't be able to hear them on some average session of just enjoying a music.  :)
 
 Specially, in your specific case, if we're talking about --quality 6 or up, as you used --quality 5.5 for your test, right?
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: Kamedo2 on 2020-10-07 19:14:19
I reordered the encoders by score.

(https://listening-test.coresv.net/img2/IgorC_192kbps1enc.png)
(https://listening-test.coresv.net/img2/IgorC_192kbps2enc.png)
Code: [Select]
MP3	xHE-AAC	Vorbis	AAC-LC	Musepack	Opus
3.5 5.0 5.0 5.0 5.0 5.0
4.5 4.0 4.6 5.0 5.0 5.0
3.8 4.7 4.8 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.2 4.8 4.9 4.9 4.9 5.0
4.2 4.9 4.9 5.0 5.0 5.0
4.8 5.0 5.0 4.7 4.9 5.0
5.0 5.0 5.0 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.7 5.0 4.8 5.0 5.0 5.0
4.8 5.0 5.0 4.9 4.8 5.0
4.0 4.9 4.8 4.8 4.8 5.0
%feature 10 LAME 3.100 exhale 1.0.7 Aotuv 6.03 Apple CoreAudioToolbox.dll 7.10.9.0 MPC Encoder 1.30.0 --stable Opus 1.3.1
%feature 11 -V2 -m 9 -q6 TVBR q91 Q5.5 -b 182
%feature 12 Average:4.46 4.86 4.90 4.94 4.95 5.00
%samples 01 Castanets
%samples 02 Fatboy
%samples 03 EIG
%samples 04 Bachpsichord
%samples 05 Enola
%samples 06 Trumpet
%samples 07 applaud
%samples 08 Velvet
%samples 09 Linchpin
%samples 10 Angels_Fall_First_ringing
%samples 11 You look good to me
%samples 12 Berling Drug
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: jaybeee on 2020-10-07 19:35:20
I know it's a pointless and largely redundant observation, but if xHE-AAC didn't bomb on Fatboy and got a 5 like AAC-LC, Musepack and Opus, it would've just come third. Again, we could make similar "what ifs" observations about all the encoders. It's just that "killer sample" - from your personal listening test - is not handled well by xHE-AAC compared to how well it handled all the other audio samples.

I'm not familiar enough - nor can remember from tests years ago - what the "killer sample" property Fatboy possess. Anyone?

EDIT: thanks for the test and the clear presentation :-)
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: Kamedo2 on 2020-10-07 19:51:43
My thoughts:

Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 21:52:35
It should have been a hard test, Great work!
Thank you, K.  :)

Specially, in your specific case, if we're talking about --quality 6 or up, as you used --quality 5.5 for your test, right?
Well, higher bitrate implies higher transparency for any encoder. 
I've used --quality 5.5 because it produced  ~192 kbps  and was  on par with other encoders.

I know it's a pointless and largely redundant observation...
I agree. But it's ok.

but if xHE-AAC didn't bomb on Fatboy and got a 5 like AAC-LC, Musepack and Opus, it would've just come third. Again, we could make similar "what ifs" observations about all the encoders. It's just that "killer sample" - from your personal listening test - is not handled well by xHE-AAC compared to how well it handled all the other audio samples.
I see your point. And Yes, we are here basically speculating.
I'll try to put it this way. "What if" Opus was being lucky on  those difficult samples?  ... bun on all of them?  
It's very unlikely that this is just a coincidence or "bad luck/good luck" for some encoder. 
And it's not just a Fatboy sample if we take a look at a  distribution of scores on another samples. 

I'm not familiar enough - nor can remember from tests years ago - what the "killer sample" property Fatboy possess. Anyone?
Sharp transients

EDIT: thanks for the test and the clear presentation :-)
Thank You  :)
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: includemeout on 2020-10-07 21:55:00
  • Musepack is unexpectedly performing well on 192kbps.
 
 

As I understood on 192kbps average (as we all know, MPC only does pure VBR)  as Igor has actually used something slightly above "standard" (--quality 5.5.).

Edit: well, he replied just before I posted this, so there you go.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 22:06:36
@includemeout
I've calibrated bitrates on a large number of albums of different genres.

MPC Q5 - 170 kbps - too small
MPC Q6- 210 kbps - was already too large.

MPC Q5.5 hited ~ 192 kbps.

You can also see that MPC had used more bits on tested samples than LC-AAC/MP3/Opus/xHE-AAC.
So bitrate setting for Musepack was more than fair.  ;)
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-07 22:10:01
@jaybeee
Also exhale xHE-AAC was the only encoder which was tested with CVBR mode (constrained VBR) while all other competitors were tested in their VBR modes.
exhale only supports CVBR.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: includemeout on 2020-10-07 22:15:58
@includemeout
I've calibrated bitrates on a large number of albums of different genres.

MPC Q5 - 170 kbps - too small
MPC Q6- 210 kbps - was already too large.

MPC Q5.5 hited ~ 192 kbps.

You can also see that MPC had used more bits on tested samples than LC-AAC/MP3/Opus/xHE-AAC.
So bitrate setting for Musepack was more than fair.  ;)
 
 Thanks. I got that. It's just that I wanted to make sure it was business as usual with MPC (well, in fact higher than "standard", but only slightly) and not some unheard of performance as I may have read into Kamedo's post, but I'll wait for his reply.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: hardrain on 2020-10-08 07:51:31
No love for faac/faad2? It was the fastest encoder/decoder for me but I didn't like how it sounded even at the highest quality. I hoped to know other people's perception.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: LithosZA on 2020-10-08 10:09:39
No love for faac/faad2? It was the fastest encoder/decoder for me but I didn't like how it sounded even at the highest quality. I hoped to know other people's perception.
I think it might score about the same as MP3
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: C.R.Helmrich on 2020-10-08 11:40:26
Thanks very much, Igor, for this high-bit-rate test! It is very helpful for me, as the developer of exhale, since my hearing is not good enough anymore for testing such high rates myself. My 2 cents:

First cent
but if xHE-AAC didn't bomb on Fatboy and got a 5 like AAC-LC, Musepack and Opus, it would've just come third. Again, we could make similar "what ifs" observations about all the encoders. It's just that "killer sample" - from your personal listening test - is not handled well by xHE-AAC compared to how well it handled all the other audio samples.
I see your point. And Yes, we are here basically speculating.
I'll try to put it this way. "What if" Opus was being lucky on  those difficult samples?  ... bun on all of them?  
IMHO these are very good observations and discussions, and both issues (one lower-quality score on exhale, all-identical scores on Opus) are the two extremes of the same issue (outliers), which are addressed by proper statistical analysis. For those not familiar with such things:

When your statistical sample size is small (as is the case here, 1 listener times 12 samples), it is important to consider possible effects caused by the small sample size, in order to avoid drawing conclusions which are not, let's say, statistically solid. Luckily, it's very easy to do a "robust" statistical analysis: there's a publicly available tool called Friedman by ff123 (https://www.rarewares.org/others.php) which e.g. Kamedo2 regularly uses on his personal tests. I fed this tool with a text file containing the first 13 lines of the text box that Kamedo2 posted yesterday:
Code: [Select]
MP3	xHE-AAC	Vorbis	AAC-LC	Musepack	Opus
3.5 5.0 5.0 5.0 5.0 5.0
4.5 4.0 4.6 5.0 5.0 5.0
3.8 4.7 4.8 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.2 4.8 4.9 4.9 4.9 5.0
4.2 4.9 4.9 5.0 5.0 5.0
4.8 5.0 5.0 4.7 4.9 5.0
5.0 5.0 5.0 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
4.7 5.0 4.8 5.0 5.0 5.0
4.8 5.0 5.0 4.9 4.8 5.0
4.0 4.9 4.8 4.8 4.8 5.0
No matter which option of Friedman.exe I choose (blocked ANOVA, Friedman/Fischer, Tukey's HSD), I always get this report:

AAC-LC is better than MP3
Musepack is better than MP3
Opus is better than MP3
Vorbis is better than MP3
xHE-AAC is better than MP3

Note that it does not say that Opus is better than xHE-AAC or Vorbis, and Kamedo2 privately confirmed to me that I'm using the tool correctly. That's what I mean with "robust statistical analysis". Essentially, Friedman.exe's comments on your discussion could be phrased as: yes, Opus may be "lucky" on at least one sample, and exhale may be "unlucky" on at least one sample (though it can't say which sample). It also means that the plots should be taken with a grain of salt, which is often the case. As Garf once put it on this forum: "Basically, the graphics suck, but they look cute (https://hydrogenaud.io/index.php?topic=90403.msg767004#msg767004)" ;)

Usually, when you add more samples (like Guruboolez and Kamedo2 did) or listeners (as was done in HA's recent public tests), those issues tend to disappear. Nonetheless, exhale's quality on Fatboy could be a bit higher even to my ears, which brings me to my

Second cent
Indeed, Igor, the reason is an excessive chain of transients on the first half of Fatboy. A few weeks ago I had a possible solution for this, but at the lower bit-rates it degraded the audio quality on some other samples, so I decided not to follow up on that issue. And, as Kamedo2 mentioned, at 96 kbit/s, where the overall audio quality is lower, the score on Fatboy doesn't degrade much more (3.5), see his personal test (https://hydrogenaud.io/index.php?topic=119861).

So I could try to "fix" Fatboy only for CVBR mode 8 or 9, but are people really using these modes much? This would be a fix specific to the first few seconds of that single sample, I don't know any other sample where this issue occurs. If so, I could try to come up with a "test.m4a" file to check if the fix works on the weekend.

Chris
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: jaybeee on 2020-10-08 13:41:38
@C.R.Helmrich & @IgorC: thank you for your comments and thoughts on my observation, which I gladly now agree wasn't a redundant one  :))

Clearly in the grand scheme of things such killer samples aren't that common, esp the Fatboy one, and so you're right to wonder whether it's worth trying to fix it, which could negatively impact other audio samples. Of course, these samples are - as evidenced here - very useful for listening tests though.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: lozenge on 2020-10-08 22:50:55
Very Interesting Igor, thank you for your efforts.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: salpro on 2020-10-08 23:05:52
Good effort.
I 'm wondering if the results will be the same if the test is realized with hifi speakers. There is more space de analyze the sounds.

And what about stereo separation in 2 planes.
What codec preserves  most the position of instrument between right and left and from up to down.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: synclagz on 2020-10-09 09:33:27

Musepack has transparent transients but it’s very slightly noisy on high frequencies of tonality.
Its high frequency muffling which is perceived as very slightly dull + flat noise addition.


Do you think that using lowpass filter switch (--bw 16500 or 17200) would help Musepack to achieve better results?
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-09 14:46:15
@C.R.Helmrich
I'm agree with your observations. All post-MP3 formats are generally on par on such bitrate.

Clearly in the grand scheme of things such killer samples aren't that common, esp the Fatboy one, ...
Transients aren't that uncommon. Electronic music is full of them.

Also You make accent only on Fatboy sample and I've alredy mentioned to You that it wasn't the only sample where exhale had an issue.
One of another samples where exhale wasn't transparent is EIG ... which also has transients.

I 'm wondering if the results will be the same if the test is realized with hifi speakers...
Everything is transparent

Do you think that using lowpass filter switch (--bw 16500 or 17200) would help Musepack to achieve better results?
I don't know and I have no plans to dig into that.
As for me, only recommended default settings. Any deviation from default can work for one individual but can break the things for others. Not my type of fun.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: fabiorug on 2020-10-09 20:41:46
 Does anyone has tried opusrug? Is it competitive with some rock, melodic music? How does it score with fatboy or enolaguy? Do you think change the tonality target to 1.2f default could help with pads at low bitrate such as 32 kbps (Romanian music) while reducing the accuracy only a bit?
Anyone interested to try it in Linux?

So far it doesn't beat quality records with pop and it isn't miracles like Jpeg XL, Nero AAC ABR two-pass 69 kbps can sound good on unreleased pop. Also nobody has produced a bitstream of it in Linux and as I heard developers aren't interested because I didn't compile myself and the program should be up to master at least or use opusfile latest, I don't think j m valin or some other opus contributors have read my thread.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-10 20:46:11
Thanks very much, Igor, for this high-bit-rate test! It is very helpful for me, as the developer of exhale, since my hearing is not good enough anymore for testing such high rates myself.
You're always welcome, Chris.  :)
I didn't answer soon because was recovering from covid.  It's all ok now.

Second cent
Indeed, Igor, the reason is an excessive chain of transients on the first half of Fatboy. A few weeks ago I had a possible solution for this, but at the lower bit-rates it degraded the audio quality on some other samples, so I decided not to follow up on that issue. And, as Kamedo2 mentioned, at 96 kbit/s, where the overall audio quality is lower, the score on Fatboy doesn't degrade much more (3.5), see his personal test (https://hydrogenaud.io/index.php?topic=119861).
I see.  As from exhale's topic it's pretty clear that now its priority is lower bitrates <= 96 kbps.  And I can't be more agree with that. There is no reason to work on xHE-AAC and bitrates higher than 96 kbps when LC-AAC with its expired patents does excellent on such high bitrates. Last time I've tried exhale 1.0.7.0 @ 96 kbps and similar bitrates it was the best encoder and performed on par with Opus.

The main targent of this test was "MP3 vs post-MP3 codecs at high bitrate". I suspected but wasn't sure that all post-MP3 encoders were going to perform on par. Detecting where xHE-AAC has some issues was a by-product. Exhale here was for a complete picture, as just another post-MP3 encoder.

Take care, guys
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-10-11 14:33:17
Musepack can go above 320kbps even at --standard, I have some tracks reaching 390 ~ 800kbps. Yet with AAC/Vorbis on tracks like that don't do that despite having no frame limit like MP3/Opus. Yet on Noise/synth focused samples transform codecs really hate it, There some samples i have where the either artifact or have horrid bitrate bloat.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: C.R.Helmrich on 2020-10-11 17:40:10
I didn't answer soon because was recovering from covid.  It's all ok now.
Ouch! I'm glad you recovered!
There is no reason to work on xHE-AAC and bitrates higher than 96 kbps ... Last time I've tried exhale 1.0.7.0 @ 96 kbps and similar bitrates it was the best encoder and performed on par with Opus.
I'm glad to hear that. Still, your test sparked my motivation to look at exhale's handling of transients once more. Attached an experiment which, hopefully, improves the Fatboy sample a bit. I can hardly hear a difference. Could you compare this against the encoding in your test if you have time, please?

Chris
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-10-11 18:38:43
Chris,

Yes,  now it's much better.

Code: [Select]
ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:

1R = C:\Audio\15_samples\Personal Test 192 kbps\exhale m9 192 kbps\fatboy_exhale_cvbr9_exp17-brmRS2.wav
2R = C:\Audio\15_samples\Personal Test 192 kbps\exhale m9 192 kbps\13 fatboy_30sec.wav

---------------------------------------
General Comments:

---------------------------------------
1R File: C:\Audio\15_samples\Personal Test 192 kbps\exhale m9 192 kbps\fatboy_exhale_cvbr9_exp17-brmRS2.wav
1R Rating: 4.6
1R Comment:
---------------------------------------
2R File: C:\Audio\15_samples\Personal Test 192 kbps\exhale m9 192 kbps\13 fatboy_30sec.wav
2R Rating: 4.0
2R Comment: Some  ghost speech in R channel.
---------------------------------------
ABX Results:
C:\Audio\15_samples\Personal Test 192 kbps\exhale m9 192 kbps\fatboy_exhale_cvbr9_exp17-brmRS2.wav vs C:\Audio\15_samples\Personal Test 192 kbps\exhale m9 192 kbps\13 fatboy_30sec.wav
    5 out of 5, pval = 0.031
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: C.R.Helmrich on 2020-10-11 22:30:21
Great, thanks a lot! That would, hypothetically, result in an average score of almost 4.91 on your test set, assuming of course that no other samples are degraded in audio quality. I'll check if that's the case and then add this fix to the upcoming exhale 1.0.8 release.

Chris
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-10-14 07:17:36
Has anyone here tested MPC at Q4.5 which is 160kbps VBR?, Would be fun too see how it fairs to AAC/Ogg/Opus there. No idea why people are perplexed MPC is transparent within 130 ~ 200kbps despite being pretty much modern MP2. It already uses PNS when under 128kbps & i wonder if that helping it a bit even at 200kbps?.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: guruboolez on 2020-10-16 14:54:46
I'm back on HA.org
I'm truly impressed by this test Igor: I know how hard it is to get usable results at such high bitrate. Well done!
Result is not a real surprise to me: MP3 struggles with pre-echo/killer samples and increasing the bitrate doesn't always help.
Post-MP3 formats are therefore better but not equally. However it's very difficult to show on a listening test.

Still, this test shows that an alternative to MP3 is preferable even near 200 kbps.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: Pabl on 2020-11-01 10:32:29
AAC/OPUS 192kbps, really? ??? This quality is sufficient for this equipment? :o

Quote
Source: Benchmark DAC1 with built-in headphone AMP HPA2 (24bits/44.1kHz mode)
Headpones: Sennheiser HD 800
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-11-01 19:04:20
Result is not a real surprise to me: MP3 struggles with pre-echo/killer samples and increasing the bitrate doesn't always help.
MP3 fails to achieve transparency on a wide kind of signals as transients, tonal etc.

MP3's blocksize range is too limited (https://en.wikipedia.org/wiki/Advanced_Audio_Coding#AAC's_improvements_over_MP3):
Quote
higher coding efficiency for stationary signals (AAC uses a blocksize of 1024 or 960 samples, allowing more efficient coding than MP3's 576 sample blocks);
higher coding accuracy for transient signals (AAC uses a blocksize of 128 or 120 samples, allowing more accurate coding than MP3's 192 sample blocks);

Increasing bitrate on MP3 to ~256-320 kbps doesn't help much to reach quality of  post-MP3 codecs at 192 kbps.



AAC/OPUS 192kbps, really? ??? This quality is sufficient for this equipment? :o
That's what test shows.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: Pabl on 2020-11-02 16:14:34
AAC/OPUS 192kbps, really? ??? This quality is sufficient for this equipment? :o
That's what test shows.
This applies only to these samples or all music in average?
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: IgorC on 2020-11-02 22:24:22
The test was performed on samples which are harder to encode than an average music.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-11-05 08:46:03
So MP3 unfit for any music with pre echo/sharp attacks?, at any bit rate. Really wonder how MP3 was green lit with so many issues that even Lame encoder can't fix. Since so much current music uses electronic elements with sounds on par with Eig/fatboy samples. Could explain the upsurge with AAC, Opus, Vorbis for the lossy camp.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: guruboolez on 2020-11-05 10:17:02
Really wonder how MP3 was green lit with so many issues that even Lame encoder can't fix.
At the beginning of hydrogenaudio in 2001 MP3 was—along with WMA—the only suitable lossy format for mobile use (digital audio player, CD-MP3…). Quality issues were well-known here. The audio stayed nevertheless very popular: wide-compatibility, and transparent for most users and even many advanced ones were not horrified by slight smearing. That's why MP3 is still widely used.

IgorC's test highlight MP3's Achilles heel but overall performance is far from being bad. An overal score of 4.46 means “very good with audible but not annoying differences”. The worst sample is ranked at 3.5, which means “between slightly annoying and not annoying”. And this is only 192 kbps. There are two stronger VBR modes (V1 and V0), and of course 320 kbps CBR. All these modes will lower the audibility of distortions and could even eradicate the slightiest artifacts heard at 192 kbps.

15 years ago I performed a very large listening test with 200 samples (https://hydrogenaud.io/index.php?topic=38792.0) (150 classical music + 50 various). The high anchor was also LAME -V2 (3.98 alpha). Results were very similar: 4.58 for classical music and 4.61 for other musical genres. For the small electronic/artifical samples group (5 samples only) MP3's score was lower than Ogg Vorbis 128 kbps. Again, there's nothing new…  ;)
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-11-05 11:35:11
The pre echo/fast attack artifacts I get on stuff like electronic/artifical samples still fail at V0, 320kbps. I moved to other codecs since even just generic crackly static sounds like vinyl, noise, walking on snow would make Lame shoot to 275 ~ 318kbps at V4 setting or have loud pops not in the lossless version.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: guruboolez on 2020-11-05 11:54:22
The pre echo/fast attack artifacts I get on stuff like electronic/artifical samples still fail at V0, 320kbps.

It could be, but they won't be as audible. Overall score will rise further even if full transparency is technically impossible to achieve with MP3.

Quote
I moved to other codecs since even just generic crackly static sounds like vinyl, noise, walking on snow would make Lame shoot to 275 ~ 318kbps at V4 setting
During very short moments maybe but overall bitrate is much lower. And 275 kbps frames with V4 are not identical to 275 kbps with V2. Bitrate is only a result: there are maths behind, and different quality settings between different VBR modes.
And it's a smart move from you :) The best MP3 encoder/setting has been surpassed by Vorbis, AAC, MPC for twenty years ago and more recently by OPUS and USAC. Unless specific reason (like a car with limited compatibility) I don't see any reason to stick with MP3.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-11-05 12:02:56
Yeah if your using <192kb/s their no point with MP3 unless your stuck with stuff that don't support AAC or Vorbis. The sharp rise on AAC & Opus seems to be from that Phones/DAP(android) now use them, MP3 seems to be the last thing people pick when going lossy now.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-11-08 12:24:34
That it performed really well at 192Kbps. I always thought Musepack needed more bitrate to sound transparent.
It does it still a subband codec like MP2, Some pure noise samples need higher setting to stop bit rate starving them hard.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: lozenge on 2020-11-20 15:30:05
MP3 seems to be the last thing people pick when going lossy now.

Among people who know maybe, but in general the internet still has way too many posts from people wanting to convert their Youtube downloads or similar from AAC  to MP3, and many that are still convinced that MP3 CBR 320 is "the best", which sometimes leads to the mention of FLAC , which confuses the op even  more.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ani_Jackal3 on 2020-11-20 19:20:32
MP3 seems to be the last thing people pick when going lossy now.

Among people who know maybe, but in general the internet still has way too many posts from people wanting to convert their Youtube downloads or similar from AAC  to MP3, and many that are still convinced that MP3 CBR 320 is "the best", which sometimes leads to the mention of FLAC , which confuses the op even  more.

Noticed that too many get very sore when told AAC/Vorbis can be transparent at 160kbps. Since they just go then why does spotify/Apple use 256 ~ 320kbps.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ManekiNeko on 2020-11-28 18:37:16
Would love to have seen Fraunhofer FDK AAC Codec Library v4.0.1 codec used in this test at Q10 VBR (average 192 kbit/s) and how good it is against Apple’s encoder at this high bit rate.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: guruboolez on 2020-11-28 19:30:00
Has something changed between FDK AAC 4.0.0 and 4.0.1? With 4.0.0 there's no Q10 VBR. There are 5 VBR mode for LC-AAC: the fourth one is ≃ 136 kbps and the next, fifth and last VBR mode is ≃ 236 kbps. Not very flexible and no ~192 kbps VBR mode with 4.0.0.
>source< (https://docs.google.com/spreadsheets/d/18lGNoBB0ZB2A4SGAw3_z-5L7gzl54MG3D_V7tf2jII8/edit?usp=sharing).
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ManekiNeko on 2020-11-28 23:01:55
Has something changed between FDK AAC 4.0.0 and 4.0.1? With 4.0.0 there's no Q10 VBR. There are 5 VBR mode for LC-AAC: the fourth one is ≃ 136 kbps and the next, fifth and last VBR mode is ≃ 236 kbps. Not very flexible and no ~192 kbps VBR mode with 4.0.0.
>source< (https://docs.google.com/spreadsheets/d/18lGNoBB0ZB2A4SGAw3_z-5L7gzl54MG3D_V7tf2jII8/edit?usp=sharing).


Yeah sorry, my mistake. I use Poikosoft’s converter exclusively these days with Apple’s LC AAC encoder. I remember seeing a VBR mode In their FDK AAC offering a bit rate range of 180-230 and a Google search led me to this page but I now see is a beta version: https://www.poikosoft.com/ezcd-fraunhofer-fdk-aac-encoder I’m away from home this weekend so I can’t check my PC.
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: ManekiNeko on 2020-12-02 11:29:26
Just a quick note that the updated FDK AAC settings/tunings in the software I use has been released. Good to see that they’ve been working with Exhale too. I’d love to give it a serious listening test myself but due to the pandemic I don’t currently have the equipment at this high bit rate. https://www.poikosoft.com/music-converter-version-history
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: pr0m3th3u5 on 2020-12-03 14:20:58
Has something changed between FDK AAC 4.0.0 and 4.0.1? With 4.0.0 there's no Q10 VBR. There are 5 VBR mode for LC-AAC: the fourth one is ≃ 136 kbps and the next, fifth and last VBR mode is ≃ 236 kbps. Not very flexible and no ~192 kbps VBR mode with 4.0.0.
>source< (https://docs.google.com/spreadsheets/d/18lGNoBB0ZB2A4SGAw3_z-5L7gzl54MG3D_V7tf2jII8/edit?usp=sharing).


I wonder what is the reason for the huge jump in bitrate between 4 & 5?!
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: AiZ on 2020-12-03 20:44:33
Hello,

I wonder what is the reason for the huge jump in bitrate between 4 & 5?!

It seems to be designed like this,
Code: [Select]
libAACenc/include/aacenc_lib.h

-----------------------------------------------
 VBR_MODE | Approx. Bitrate in kbps for stereo
          |     AAC-LC    |      AAC-ELD
----------+---------------+--------------------
    VBR_1 | 32 (HE-AACv2) |         48
    VBR_2 | 72 (HE-AACv1) |         56
    VBR_3 |      112      |         72
    VBR_4 |      148      |        148
    VBR_5 |      228      |        224
--------------------------------------------
perhaps to accommodate the (superfluous) bandwidth.
Code: [Select]
libAACenc/src/bandwidth.cpp

static const BANDWIDTH_TAB_VBR bandWidthTableVBR[] = {
    {AACENC_BR_MODE_CBR, 0, 0},
    {AACENC_BR_MODE_VBR_1, 13000, 13000},
    {AACENC_BR_MODE_VBR_2, 13000, 13000},
    {AACENC_BR_MODE_VBR_3, 15750, 15750},
    {AACENC_BR_MODE_VBR_4, 16500, 16500},
    {AACENC_BR_MODE_VBR_5, 19293, 19293},
    {AACENC_BR_MODE_SFR, 0, 0},
    {AACENC_BR_MODE_FF, 0, 0}

    AiZ
Title: Re: Personal blind listening test – MultiCodec at ~192 VBR kbps
Post by: pr0m3th3u5 on 2020-12-06 09:55:19
Hello,

I wonder what is the reason for the huge jump in bitrate between 4 & 5?!

It seems to be designed like this,
Code: [Select]
libAACenc/include/aacenc_lib.h

-----------------------------------------------
 VBR_MODE | Approx. Bitrate in kbps for stereo
          |     AAC-LC    |      AAC-ELD
----------+---------------+--------------------
    VBR_1 | 32 (HE-AACv2) |         48
    VBR_2 | 72 (HE-AACv1) |         56
    VBR_3 |      112      |         72
    VBR_4 |      148      |        148
    VBR_5 |      228      |        224
--------------------------------------------
perhaps to accommodate the (superfluous) bandwidth.
Code: [Select]
libAACenc/src/bandwidth.cpp

static const BANDWIDTH_TAB_VBR bandWidthTableVBR[] = {
    {AACENC_BR_MODE_CBR, 0, 0},
    {AACENC_BR_MODE_VBR_1, 13000, 13000},
    {AACENC_BR_MODE_VBR_2, 13000, 13000},
    {AACENC_BR_MODE_VBR_3, 15750, 15750},
    {AACENC_BR_MODE_VBR_4, 16500, 16500},
    {AACENC_BR_MODE_VBR_5, 19293, 19293},
    {AACENC_BR_MODE_SFR, 0, 0},
    {AACENC_BR_MODE_FF, 0, 0}

    AiZ

I read the code but I still don't understand what does "to accommodate the (superfluous) bandwidth" mean?
SimplePortal 1.0.0 RC1 © 2008-2021