Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding (Read 1401 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Abstract:
Personal blind sound quality comparison of the Bluetooth codec, AAC and LC3, at 160 kbps, at 44.1 kHz(except for liblc3).
As of 2022, Bluetooth earphones and speakers often support AAC codec.
LC3 is a new codec, which will be mandatory in the upcoming LE Audio in Bluetooth 5.2.
Two AAC encoders were tested; one engine is CoreAudio, used by the Apple, and the other engine, FDK-AAC is used by the Android.
Two LC3 encoders were tested; the latest LC3plus encoder available from ESTI, and google / liblc3, which is actively developed by Antoine SOULIER as of June 2022.
The original lossless tracks were first encoded by Opus 128 kbps, decoded, and then re-encoded by the AAC and LC3. This is meant to replicate how online streaming services are used with Bluetooth.
MP3 encoded with LAME CBR was also tested, as anchors. MP3 was directly encoded from the original, not from Opus.

Encoders:
Opus 1.3 via opus-tools-0.2
CoreAudioToolbox 7.10.9.0, via qaac 2.73 (x64 version).
FDK-AAC 2.0.2, via ffmpeg version N-102573-g9d4c018497.
LC3plus Floating Point Software V1.6.3ETSI, ETSI TS 103 634 V1.3.1.
google / liblc3, Apr 20, 2022 version (older version, as of June 2022).

Opus settings:
opus-tools-0.2-opus-1.3\opusenc --bitrate 128 in.wav temp.opus
opus-tools-0.2-opus-1.3\opusdec --float --quiet temp.opus temp.wav
qaac_2.73\x64\refalac64 temp.wav --rate 48000 -D -b 16 -o temp.48kHz.wav
qaac_2.73\x64\refalac64 temp.wav --rate 44100 -D -b 16 -o temp.44kHz.wav

Opus ⇒ AAC Settings:
qaac_2.73\x64\qaac64 --cbr 160 -o out.mp4 temp.44kHz.wav
ffmpeg102573 -y -i temp.44kHz.wav -c:a libfdk_aac -b:a 160k out.mp4

Opus ⇒ LC3 Settings:
LC3plus -E -q -v temp.44kHz.wav out.lc3 160000
liblc3\bin\elc3 -b 160000 temp.48kHz.wav out.lc3

MP3 Settings:
lame3.100.1-x64\lame -b 128 in.wav out.mp3
lame3.100.1-x64\lame -b 64 in.wav out.mp3

Sample tracks:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 diverse music and speech sound samples.

Hardware:
Sony PSP-3000 + RP-HJE150.

Results (AAC and LC3 only):

Results (including MP3):



Conclusions & Observations:
  • AAC offered very high fidelity at 160 kbps. Most scores were better than 4.0 (Perceptible, but not annoying) in this particular situation. So were Opus.
  • LC3 encoder, provided by ESTI, did not offered fidelity comparable to the commonly used AAC codec at 160k in this particular situation.
  • LC3 encoder, developed by Google, did not offered fidelity comparable to the commonly used AAC codec at 160k in this particular situation.
  • It's not clear which LC3 encoder was better from this test alone.
  • An original track first encoded by Opus 128kbps, decoded, then re-encoded by LC3plus is still better than the LAME MP3 CBR 128kbps encoded once.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              161         132.90
Testers (blocks)    26           7.81
Codecs eval'd        5         114.05   22.81   268.52  0.00E+000
Error              130          11.04    0.08
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.157

Means:

AAC(FDK) AAC(Appl LC3plus  liblc3   LAMECBR1 LAMECBR6
  4.36     4.35     3.99     3.89     3.56     1.90  

---------------------------- p-value Matrix ---------------------------

         AAC(Appl LC3plus  liblc3   LAMECBR1 LAMECBR6
AAC(FDK) 0.926    0.000*   0.000*   0.000*   0.000*  
AAC(Appl          0.000*   0.000*   0.000*   0.000*  
LC3plus                    0.227    0.000*   0.000*  
liblc3                              0.000*   0.000*  
LAMECBR1                                     0.000*  
-----------------------------------------------------------------------

AAC(FDK) is better than LC3plus, liblc3, LAMECBR128kbps, LAMECBR64kbps
AAC(Apple) is better than LC3plus, liblc3, LAMECBR128kbps, LAMECBR64kbps
LC3plus is better than LAMECBR128kbps, LAMECBR64kbps
liblc3 is better than LAMECBR128kbps, LAMECBR64kbps
LAMECBR128kbps is better than LAMECBR64kbps


FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Friedman Analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
Fisher's protected LSD for rank sums:  26.945

Ranksums:

AAC(Appl AAC(FDK) LC3plus  liblc3   LAMECBR1 LAMECBR6
143.00   141.00    98.00    87.00    71.00    27.00  

---------------------------- p-value Matrix ---------------------------

         AAC(FDK) LC3plus  liblc3   LAMECBR1 LAMECBR6
AAC(Appl 0.884    0.001*   0.000*   0.000*   0.000*  
AAC(FDK)          0.002*   0.000*   0.000*   0.000*  
LC3plus                    0.424    0.050*   0.000*  
liblc3                              0.244    0.000*  
LAMECBR1                                     0.001*  
-----------------------------------------------------------------------

AAC(Apple) is better than LC3plus, liblc3, LAMECBR128kbps, LAMECBR64kbps
AAC(FDK) is better than LC3plus, liblc3, LAMECBR128kbps, LAMECBR64kbps
LC3plus is better than LAMECBR128kbps, LAMECBR64kbps
liblc3 is better than LAMECBR64kbps
LAMECBR128kbps is better than LAMECBR64kbps

Raw data:
Code: [Select]
AAC (Apple)	AAC (FDK)	LC3plus	liblc3	LAME 128kbps	LAME 64kbps
%feature 5 Opus ⇒ AAC Opus ⇒ AAC Opus ⇒ LC3 Opus ⇒ LC3
%feature 7 MP3 MP3
%feature 10 CoreAudioToolbox 7.10.9.0, via qaac 2.72 FDK-AAC 2.0.2 V1.6.3 ETSI (floating point ver.) liblc3 Apr 20, 2022 LAME 3.100.1-x64 LAME 3.100.1-x64
%feature 11 --cbr 160 -c:a libfdk_aac -b:a 160k 160000 -b 160000 -b 128 -b 64
%feature 12 from Opus 128kbps from Opus 128kbps from Opus 128kbps from Opus 128kbps from original from original
%genre Kamedo2's 15 sample
4.300 4.400 4.100 3.700 3.200 1.800
4.700 4.500 4.200 3.700 4.100 1.700
4.500 4.300 3.900 4.100 4.400 1.600
4.500 4.400 4.100 3.900 3.700 2.100
3.800 4.100 3.500 3.600 2.500 1.700
4.300 4.200 3.700 3.400 3.800 1.700
4.400 4.500 4.100 3.900 2.900 2.400
4.400 4.500 3.600 4.100 3.900 1.700
3.900 3.800 4.300 4.400 4.100 2.700
4.500 4.400 3.900 3.500 3.600 1.800
4.400 4.300 4.100 4.200 3.100 1.600
4.200 4.100 3.900 3.700 3.300 2.200
4.400 4.500 3.800 3.700 3.600 1.700
4.200 4.400 3.900 3.800 3.600 1.600
4.400 4.300 3.600 3.700 4.100 1.800
%genre IgorC's 12 sample
4.400 4.500 4.100 4.000 2.700 1.800
4.100 4.400 4.600 4.500 3.600 1.600
4.600 4.500 4.100 4.200 3.200 1.700
4.300 4.400 3.900 3.800 3.600 1.900
4.500 4.200 4.400 3.800 3.600 1.900
4.300 4.200 3.800 3.700 3.200 2.100
3.900 3.700 3.400 3.500 2.800 1.600
4.100 4.400 3.900 4.300 2.600 1.900
4.500 4.600 3.900 3.700 4.100 2.100
4.200 4.300 3.800 3.500 4.100 2.200
5.000 4.800 4.700 4.400 4.500 2.300
4.700 5.000 4.400 4.300 4.100 2.200

%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella

%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech

Bitrates:
All encoders were tested with CBR settings, therefore every tracks of AAC and LC3 is very close to 160kbps.


Code: [Select]
%bitrate
AAC (Apple) AAC (FDK) LC3plus liblc3 LAME CBR 128kbps LAME CBR 64kbps
162387 161978 161014 161659 128285 64154
162562 162020 160979 161667 128230 64196
162879 162365 160984 161655 128372 64266
162679 162084 161015 161646 128319 64214
162749 162340 160981 161652 128368 64267
162989 162338 160987 161719 128440 64278
162974 162222 161016 161679 128458 64286
162331 162011 160989 161658 128271 64202
163932 163180 161141 161773 128827 64534
162628 162092 160980 161683 128295 64199
162327 161897 161024 161667 128244 64205
163037 162391 161182 161834 128553 64319
162736 162244 161050 161634 128353 64260
162391 162100 160983 161628 128304 64185
162832 162343 161019 161702 128470 64276
165547 163738 161116 161762 129023 64827
162311 161976 161009 161624 128336 64161
163482 162617 161070 161709 128613 64406
162476 162136 160979 161669 128396 64204
162317 162027 161015 161648 128300 64203
164590 163095 161051 161759 128666 64466
165505 163638 161164 161773 128851 64589
164116 162966 161075 161768 128639 64396
163213 162632 161036 161696 128588 64332
162477 162004 161025 161649 128264 64195
162842 162243 160988 161630 128432 64282
163657 162689 161047 161716 128663 64396

Other tests:


Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #1
Thanks for the test!

Extrapolating a little bit, LC3 looks to be similar in audio quality to MP3, possibly worse. It might need 200+ kbps for high quality. (And a stable 200+ kbps Bluetooth connection is not guaranteed even when just moving around the room, IME.)
It didn't do great on the few vocal/speech samples, either.
I hope that at least it has really low latency.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #2
fix: LC3, the tested version of google / liblc3 is 67dcc7c, Apr 19, 2022.
6 commits on Apr 20, 2022 were not reflected in the tested binary.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #3
Great test Kamedo.
I appreciate all the hard work you are putting into these tests.

I have one question for you. What is your preferred encoder for AAC-LC at around 144 or 160 kbit/s?
(From lossless sources. No re-encoding.)
gold plated toslink fan

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #4
What is your preferred encoder for AAC-LC at around 144 or 160 kbit/s?
The Apple's CoreAudio engine. You can use it via qaac on Windows, and afconvert on Mac. This engine is mature and very stable.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #5
Very interesting, it looks like CoreAudio is no longer the definitively superior AAC encoder. I believe it hasn't been updated since 2010, so FDK-AAC (which is Android's AAC encoder, is it not?) has had plenty of time to catch up.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #6
No "quality" difference.
There is two ways of development:
1. refresh patent license.
2. avoid patent royalty.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #7
Very interesting, it looks like CoreAudio is no longer the definitively superior AAC encoder.
This was a Bluetooth codec test, so CoreAudio was forced to be CBR, while FDK-AAC is CBR on default.
Probably CoreAudio will be even closer to original on CVBR. That's the setting CoreAudio has been extensively used and tested.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #8
I'd ask why this was being done on a PSP? A really nice headphone amp and DAC are only £100. Do you have measurements for the headphone amp and DAC in the PSP?

This seems like a huge oversight to me. All it says is that when I pay x on a PSP, I can't tell the difference.

Will HA ever understand the rest of the chain is important?

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #9
I'd ask why this was being done on a PSP? A really nice headphone amp and DAC are only £100. Do you have measurements for the headphone amp and DAC in the PSP?

This seems like a huge oversight to me. All it says is that when I pay x on a PSP, I can't tell the difference.

Will HA ever understand the rest of the chain is important?
I wondered the same thing, considering how old the PSP is, but that doesn't make me doubt the results. If the DAC is so important, feel free to provide ABX results demonstrating that you can hear a difference between different DACs when all other factors are equal. The PSP isn't nearly old enough to be from the era when D2A and A2D conversions were dicey. The burden of proof lies with anyone who claims the PSP's DAC is lacking.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #10
Here there are significant differences on equal hardware, so the playback chain is good enough for those to be clear.
There is no plausible reason that cheap hardware should create differences.

Had the results been no difference consistently, it could have been due to one or a combination of more of the following
codecs being equally good to most users
codecs being indistinguishable to this particular user who doesn't have the best ears
playback chain being so bad that it erases all audible differences
... and one would have wished for some other user on some other hardware to try to replicate the study, right?

(Which of course isn't a bad thing in any case - replication, I mean.)
Last two months' worth of foobar2000.org ad revenue has been donated to support war refugees from Ukraine: https://www.foobar2000.org/

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #11
At 100-200kbps these codecs only have no more than 30dB of SNR, except for simple synthetic signals like sine waves and such, DAC quality is irrelevant here. Poor combination of headphone and amp could be an issue though, if someone delibrately wants to do so.

https://hydrogenaud.io/index.php?topic=39970.0
For the record I was able to ABX 320CBR mp3 with LAME 3.96.1 and 3.97b2 and the sample used was not considered as a killer sample by other members, and that test was done on an $128 sound blaster and $25 Sony earbud (not even IEM) in 2005.
[EDIT] The audio sample in the link above is dead, for those who want to try, download the attached flac.
[EDIT2] Strange. "Your file is too large. The maximum attachment size allowed is 0 KB." Can @Peter or @spoon help?

Lossy codecs are based on exploiting psychoacoustics, not about typical "SINAD" and such. This article should be a good starting point:
https://izotope-rx.livejournal.com/5760.html

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #12
At 100-200kbps these codecs only have no more than 30dB of SNR, except for simple synthetic signals like sine waves and such, DAC quality is irrelevant here.

Are the SNR ratio and THD percentage figures that informative on good codecs? After all, those are developed to make the "N" and the "D" less annoying, rather than good looking.
Last two months' worth of foobar2000.org ad revenue has been donated to support war refugees from Ukraine: https://www.foobar2000.org/

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #13
Are the SNR ratio and THD percentage figures that informative on good codecs? After all, those are developed to make the "N" and the "D" less annoying, rather than good looking.
My observation on several codecs is that opus often has poorest performance on simple tones. Sometimes can be improved by changing framesize as mentioned here:
https://hydrogenaud.io/index.php/topic,122093.msg1007920.html#msg1007920

LAME for example, when using the older versions with some debug switches, can be forced to use only long blocks to optimize tonal performance. A very detailed thread, in a sarcastic way:
https://hydrogenaud.io/index.php?topic=4457.0

Of course, measuring tonal performance alone is deceiving and many long time HA members know about these things very well. Things like pre-echo and stereo separation cannot be ignored too.

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #14
Interesting, thank you for the test!
Do we know for sure 160kbps is a representative bitrate to test LC3 in future Bluetooth LE Audio scenarios? I'm only asking because the codec it replaces (SBC) is typically running at 328kbps. Halving the bitrate makes sense, but I haven't seen any document about this yet.

Mod: I've found this chart. This does include 160kbps, but at the low end. Wonder what will be the typical bitrate.
X

 

Re: Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding

Reply #15
It looks like 160kbps is too low for the production quality. (3.89 for liblc3)