HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: IgorC on 2021-09-26 08:01:20

Title: Personal test – MP3 vs AAC (high rates, VBR)
Post by: IgorC on 2021-09-26 08:01:20
Hi,

All versions of encoders were the newest and stable
LAME MP3 3.100 -V2/V0/320kbps (~192 kbps/~260kbps/320 kbps)
Apple LC-AAC 7.10.9.0 via QAAC TVBR q91  (~192 kbps)

Files
LINK (https://drive.google.com/file/d/1FmewrA02W4CJkVXYafY0TNMTJHKzZGWc/view?usp=sharing)

RESULTS
(https://i.ibb.co/wh756X3/Graph01.png)
(https://i.ibb.co/wYRnG5r/Graph02.png)
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: Kamedo2 on 2021-09-26 08:31:48
IgorC,

Great test! The Apple AAC seems to be a very stable codec at high rates.
Also, you have tested LAME CBR 320kbps and it is a record high rate in an ABC-HR listening test of MP3.
Interesting that V0 seems somewhat safer than the CBR320k.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: shadowking on 2021-09-26 08:33:17
Excellent. It shows that Lame V2 is not be all end setting. Quality
will go up @ V1 and higher.

Further more, Mp3 is still competitive @ 192k (V2). People that
don't want other formats can be happy using mp3 at 160.. 224k, and
up to 256..320k
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: Kamedo2 on 2021-09-26 09:05:32
A bit different visualization.

(https://listening-test.coresv.net/img2/IgorC-AACMP3-en2x.png)
(https://listening-test.coresv.net/img2/IgorC-AACMP3-tracks-en2x.png)

Code: [Select]
Apple AAC-LC 192k	LAME CBR320k	LAME V0	LAME V2
%feature 8 AAC MP3 MP3 MP3
5.0 4.7 4.9 3.8
4.6 3.5 4.5 4.4
5.0 4.3 4.3 3.4
5.0 4.9 4.9 4.8
5.0 4.8 4.9 4.5
4.8 4.7 4.9 4.4
5.0 5.0 4.9 4.9
5.0 5.0 5.0 4.8
4.8 4.9 5.0 4.9
5.0 4.9 4.9 4.9
5.0 4.9 4.9 4.8
4.9 5.0 4.8 4.3
%samples 01 castanets
%samples 13 fatboy_30sec
%samples 03 eig
%samples 04 Bachpsichord
%samples 05 Enola
%samples 06 trumpet
%samples 07 applaud
%samples 08 velvet
%samples 09 Linchpin
%samples 10 Angels_Fall_First_ringing
%samples 11 You look good to me
%samples Berling Drug
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: IgorC on 2021-09-26 19:45:19
IgorC,

Great test! The Apple AAC seems to be a very stable codec at high rates.
:) 
Yes, Apple AAC@192kbps crashes everything that MP3 can offer at ~260-320 kbps.

That's +64-128kbps bitrate difference.

Interesting that V0 seems somewhat safer than the CBR320k.
CBR320k has smaller amount of short frames. This explains why it performs worse on transient signals.

A bit different visualization.
Much better.  :)  I was sleepy yesterday and wanted to finish it.

Further more, Mp3 is still competitive @ 192k (V2). People that
don't want other formats can be happy using mp3 at 160.. 224k, and
up to 256..320k
Yes, I will not hesitate to listen to V0. Though Apple AAC still provides a great bitrate reduction and actually handles critical samples better.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: synclagz on 2021-09-26 20:48:28
@IgorC
This test (and your other multicodec test at 192k) finally shows how much bitrate is needed to achieve very high quality at reasonable bitrate. Excellent work!

Are you familiar with this sample:
https://hydrogenaud.io/index.php?topic=120193.0

Do you think we have to worry about this sample and similar ones for 192-256k which is otherwise more than enough for quality listening which your tests confirm?
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: Kamedo2 on 2021-09-26 21:33:54
I combined the result with the IgorC's past result (https://hydrogenaud.io/index.php?topic=120007).

In the results below, the xHE-AAC encoder, exhale, have undergone many improvements and updates since then.
The other codecs, LAME, Apple, Vorbis, Musepack, and Opus are the latest as of 2021 September.
As expected of modern codecs at 192kbps and higher, all of them offer very high fidelity.

(https://listening-test.coresv.net/img2/IgorC-AACMP3comp-en2x.png)
(https://listening-test.coresv.net/img2/IgorC-AACMP3comp-tracks-en2x.png)

Code: [Select]
MP3	MP3	MP3	AAC-LC	MP3	xHE-AAC	Vorbis	AAC-LC	Musepack	Opus
4.7 4.9 3.8 5.0 3.5 5.0 5.0 5.0 5.0 5.0
3.5 4.5 4.4 4.6 4.5 4.0 4.6 5.0 5.0 5.0
4.3 4.3 3.4 5.0 3.8 4.7 4.8 5.0 5.0 5.0
4.9 4.9 4.8 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.8 4.9 4.5 5.0 4.2 4.8 4.9 4.9 4.9 5.0
4.7 4.9 4.4 4.8 4.2 4.9 4.9 5.0 5.0 5.0
5.0 4.9 4.9 5.0 4.8 5.0 5.0 4.7 4.9 5.0
5.0 5.0 4.8 5.0 5.0 5.0 5.0 5.0 5.0 5.0
4.9 5.0 4.9 4.8 5.0 5.0 5.0 5.0 5.0 5.0
4.9 4.9 4.9 5.0 4.7 5.0 4.8 5.0 5.0 5.0
4.9 4.9 4.8 5.0 4.8 5.0 5.0 4.9 4.8 5.0
5.0 4.8 4.3 4.9 4.0 4.9 4.8 4.8 4.8 5.0
%feature 6 320kbps 260kbps ≒192kbps ≒192kbps ≒192kbps ≒192kbps ≒192kbps ≒192kbps ≒192kbps ≒192kbps
%feature 7 2021 Sep. 2021 Sep. 2021 Sep. 2021 Sep. 2020 Oct. 2020 Oct. 2020 Oct. 2020 Oct. 2020 Oct. 2020 Oct.
%feature 10 LAME 3.100 LAME 3.100 LAME 3.100 Apple CoreAudioToolbox.dll 7.10.9.0 LAME 3.100 exhale 1.0.7 Aotuv 6.03 Apple CoreAudioToolbox.dll 7.10.9.0 MPC Encoder 1.30.0 --stable Opus 1.3.1
%feature 11 -b 320 -V0 -V2 TVBR q91 -V2 -m 9 -q6 TVBR q91 Q5.5 -b 182
%feature 12 Average:4.72 4.82 4.49 4.92 4.46 4.86 4.90 4.94 4.95 5.00

%samples 01 castanets
%samples 13 fatboy_30sec
%samples 03 eig
%samples 04 Bachpsichord
%samples 05 Enola
%samples 06 trumpet
%samples 07 applaud
%samples 08 velvet
%samples 09 Linchpin
%samples 10 Angels_Fall_First_ringing
%samples 11 You look good to me
%samples Berling Drug
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: Markuza97 on 2021-09-26 22:42:03
Great test Igor!

I always thought V0 would be slightly better than q91.
I was wrong.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: shadowking on 2021-09-27 11:12:51
So when you look at the combined results as Kamedo2 posted,

This recent 256 / 320k  'normal' isn't based on any rational.  You will find many posts on the net
saying the min 256k is needed for 'hifi'. Including Cnet and other 'credible' sources. Many posters say
just go with 320cbr as storage is cheap. Well its x 1.67 times larger than 192 , thats a LOT esp for portables.
Most of them can't tell apart -b320 vs  -b192 or even 160.  
MPC / Lame were tuned to 160..200 vbr.  Before that CBR excellence point was around 192. All this for good reasons.
You can do almost anything today around the same bitrates of 2002.  So it looks like nothing has been achieved since
then but gazillion of codecs, bitrates and settings. A lot of extra complexity , Yet its back to square one; 128 / 160 / 192  etc

Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: IgorC on 2021-09-27 22:08:42
@IgorC
This test (and your other multicodec test at 192k) finally shows how much bitrate is needed to achieve very high quality at reasonable bitrate. Excellent work!
...

Great test Igor!

I always thought V0 would be slightly better than q91.
I was wrong.
Thank You, guys!

@IgorC
...
Are you familiar with this sample:
https://hydrogenaud.io/index.php?topic=120193.0
...
It's sample with sharp transients. Three first samples from my test are also transients. So this kind of signal is well presented in this test.
Btw interesting sample.  Saved. Thank You.

@IgorC
Do you think we have to worry ...
You problably will need to answer this for yourself as transparency level is personal.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: Kamedo2 on 2021-10-01 16:58:35
Is VBR -V0 better than the CBR320k on average?

Somewhat likely, but since the p-value is 0.2411, we cannot be very certain.
If he tests another 12 tracks , 24 tracks, 48 tracks, and many many tracks, he may find that -V0 can be 0.30 better, or 0.08 worse, or anywhere in between.

Code: [Select]
> cbr = c(4.7, 3.5, 4.3, 4.9, 4.8, 4.7, 5, 5, 4.9, 4.9, 4.9, 5)
> vbr = c(4.9, 4.5, 4.3, 4.9, 4.9, 4.9, 4.9, 5, 5, 4.9, 4.9, 4.8)
> t.test(cbr, vbr, paired=TRUE)

        Paired t-test

data:  cbr and vbr
t = -1.239, df = 11, p-value = 0.2411
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.30078053  0.08411387
sample estimates:
mean of the differences
             -0.1083333

> R.version
               _                          
platform       x86_64-w64-mingw32         
arch           x86_64                     
os             mingw32                    
system         x86_64, mingw32            
status                                    
major          4                          
minor          0.4                        
year           2021                       
month          02                         
day            15                         
svn rev        80002                      
language       R                          
version.string R version 4.0.4 (2021-02-15)
nickname       Lost Library Book          
>
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: IgorC on 2021-10-02 14:59:45
Kamedo2, I've read your twitter. There is some mention explaining why v0 could potentially do better than cbr320.
https://hydrogenaud.io/index.php?topic=85135.msg864674#msg864674

Translated from https://twitter.com/kamedo2/status/1442042506464481287
"The reason why V0 performs better than CBR320k is that it may be because VBR uses bit reservoir very aggressively after LAME 3.99 (hence, the local bitrate can greatly exceed 320kbps). I was taught at HA before."

Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: Kamedo2 on 2021-10-02 15:58:51
Kamedo2, I've read your twitter. There is some mention explaining why v0 could potentially do better than cbr320.
https://hydrogenaud.io/index.php?topic=85135.msg864674#msg864674

Translated from https://twitter.com/kamedo2/status/1442042506464481287
"The reason why V0 performs better than CBR320k is that it may be because VBR uses bit reservoir very aggressively after LAME 3.99 (hence, the local bitrate can greatly exceed 320kbps). I was taught at HA before."

Yeah, nu774 seems right. The improvement is 10 years old now.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: IgorC on 2021-10-02 22:25:18
Yeah, nu774 seems right. The improvement is 10 years old now.
Considering the age of MP3, the improvement is refreshingly recent and new.   :P  
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: zoog on 2021-10-06 13:31:55
Hi.
It's quite important to know what equipment was in these test. Funnily i noticed the same (mp3@320 is worse mp3@~240) about 15 years ago with lame 3.93 and 3.97.
And why aac is only 192, q96, to compete on filesize plane?
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: IgorC on 2021-10-07 00:25:28
Hi.
It's quite important to know what equipment was in these test
Hello,
The equipment was the same as in this test https://hydrogenaud.io/index.php?topic=120007.0

And why aac is only 192, q96, to compete on filesize plane?
Given that AAC scored very high  (4.93) at 192 kbps, it's pretty clear what to expect at higher bitrates.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: DARcode on 2021-10-07 00:42:40
Very very interesting info, many thanks for sharing IgorC.
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: pr0m3th3u5 on 2021-10-09 21:30:09
Interesting how Apple's AAC is in a league of it's own. For me this settles it. No reason to use MP3 if you can use AAC. (Always thought at higher bitrates it doesn't matter.)

I wonder how FDK and QAAC would fair at higher bitrates like 192kbps.  :D
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: clintb on 2021-10-11 17:34:45
I wonder how FDK and QAAC would fair at higher bitrates like 192kbps.  :D
QAAC is Apple AAC. Well, not *is*, but QAAC is how you get access to Apple AAC on Windows outside of iTunes. Have you not used it before?

https://github.com/nu774/qaac
Title: Re: Personal test – MP3 vs AAC (high rates, VBR)
Post by: pr0m3th3u5 on 2021-10-11 21:22:13
Yep, I used it before.

I should've used QAAC both the times.