Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3. (Read 11841 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Abstract:
Personal blind sound quality comparison of the Bluetooth codec, AAC and LC3, at 192 kbps, and SBC at 345 kbps.
AAC is an optional codec in Bluetooth, but Bluetooth earphones and speakers usually support the AAC codec.
SBC is the mandatory codec in Bluetooth.
LC3 is a new codec, which became mandatory in the LE Audio in Bluetooth 5.2.
Lower bit rate settings were also tested.
The test was conducted on 48 kHz input on all codecs.

Encoders:
AAC: fdkaac-1.0.4 built against fdk-aac 2.0.2, x64, from rarewares, Feb 2023 version (not the latest version, as of May 2023).
SBC: FFmpeg N-109823-g385ec46424, Feb 2023 version (not the latest version, as of May 2023).
LC3: Google / liblc3 v1.0.1, Feb 2023 version (not the latest version, as of May 2023).

AAC Settings:
fdkaac -b 128000 -o out.mp4 in.48kHz.16bit2ch.wav
fdkaac -b 192000 -o out.mp4 in.48kHz.16bit2ch.wav

SBC Settings:
ffmpeg -y -i in.48kHz.16bit2ch.wav -c:a sbc -b:a 237k out.sbc
ffmpeg -y -i in.48kHz.16bit2ch.wav -c:a sbc -b:a 345k out.sbc

LC3 Settings:
elc3 -b 128000 in.48kHz.16bit2ch.wav out.lc3
elc3 -b 192000 in.48kHz.16bit2ch.wav out.lc3

Sample tracks:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 diverse music and speech sound samples.

Hardware:
Sony PSP-3000 + AKG K712.

Results (only the higher bitrate):


Results (including lower bitrates):



Conclusions & Observations:
  • AAC offered very high fidelity at 192 kbps. Most scores were better than 4.0 (Perceptible, but not annoying).
  • SBC also offered very high fidelity at 345 kbps. Most scores were better than 4.0 (Perceptible, but not annoying).
  • LC3, also offered very high fidelity at 192 kbps, with majority of scores better than 4.0. Still not as good as AAC or SBC.
  • It's not clear whether AAC at 192 kbps is better, or SBC at 345 kbps is better from this test alone.
  • Reducing the bitrate to about two-thirds significantly impaired sound quality on all Bluetooth codecs tested.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              161          69.27
Testers (blocks)    26           9.59
Codecs eval'd        5          37.25    7.45   43.18  0.00E+000
Error              130          22.43    0.17
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.224

Means:

FDK-AAC1 SBC345kb liblc319 FDK-AAC1 SBC237kb liblc312
  4.72     4.65     4.13     4.12     3.62     3.42  

---------------------------- p-value Matrix ---------------------------

         SBC345kb liblc319 FDK-AAC1 SBC237kb liblc312
FDK-AAC1 0.578    0.000*   0.000*   0.000*   0.000*  
SBC345kb          0.000*   0.000*   0.000*   0.000*  
liblc319                   0.961    0.000*   0.000*  
FDK-AAC1                            0.000*   0.000*  
SBC237kb                                     0.079   
-----------------------------------------------------------------------

FDK-AAC192kbps is better than liblc3192kbps, FDK-AAC128kbps, SBC237kbps, liblc3128kbps
SBC345kbps is better than liblc3192kbps, FDK-AAC128kbps, SBC237kbps, liblc3128kbps
liblc3192kbps is better than SBC237kbps, liblc3128kbps
FDK-AAC128kbps is better than SBC237kbps, liblc3128kbps

Raw data:
By mistake, 04 Bachpsichord and 05 Enola were tested twice. The two scores were averaged in below.
Code: [Select]
FDK-AAC128kbps	FDK-AAC192kbps	liblc3128kbps	liblc3192kbps	SBC237kbps	SBC345kbps	
%genre Kamedo2's 15 sample
3.700 4.500 3.200 4.200 3.500 5.000
4.800 4.700 3.300 4.100 3.100 4.500
3.900 5.000 3.100 3.500 3.700 5.000
4.100 5.000 3.500 4.400 3.700 5.000
3.900 5.000 3.300 4.500 4.200 5.000
4.200 5.000 3.300 3.900 2.800 4.400
4.900 5.000 3.400 4.500 3.700 4.700
4.200 5.000 3.400 4.100 3.100 4.300
3.100 3.800 3.500 4.100 3.300 4.200
3.900 5.000 3.100 3.800 2.700 5.000
3.800 4.500 3.500 4.200 4.300 4.400
3.700 5.000 3.400 4.100 3.500 4.600
4.300 5.000 3.400 3.900 3.100 4.500
4.200 4.800 3.300 4.100 2.700 5.000
5.000 4.700 2.900 3.700 2.600 3.800

%genre IgorC's 12 sample
4.400 4.700 3.400 3.900 4.600 5.000
4.200 4.500 3.900 4.400 5.000 5.000
4.300 4.500 3.500 3.900 4.800 5.000
4.000 4.800 3.200 4.150 2.800 4.450
4.000 4.750 3.250 4.200 3.450 4.000
4.200 4.300 3.200 3.800 2.500 3.900
3.800 4.100 3.400 4.300 5.000 4.400
3.700 5.000 3.600 4.200 3.900 5.000
3.400 4.100 3.700 4.200 4.300 4.500
4.300 4.600 3.200 4.200 3.900 5.000
5.000 5.000 5.000 5.000 3.700 5.000
4.300 5.000 3.400 4.100 3.800 5.000

%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella

%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech

Other tests:


Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #1
Correct me if I am wrong.
The default frame size / latency of LC3 is 10ms. And fdkaac default frame size around 55ms or possibly higher.

Which makes the results for LC3 pretty damn impressive. But are there any reason why 256kbps or 360kbps LC3 wasn't tested? I would imagine most bluetooth should be capable supporting those bitrate reliably.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #2
But are there any reason why 256kbps or 360kbps LC3 wasn't tested? I would imagine most bluetooth should be capable supporting those bitrate reliably.
Based on Bluetooth SIG's website, I had thought that LC3 at 192 kbps should be close to 5.0 and set it to 192 kbps. Based on this listening test result, using somewhere around 256 kbps sounds like a good idea.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #3
As for the bitrate, All codecs were tested with CBR settings, so the file-based bitrate very constant.

Code: [Select]
%bitrate
FDK-AAC128kbps FDK-AAC192kbps SBC237kbps SBC345kbps liblc3_128kbps liblc3_192kbps
130216 194306 240012 348018 129655 193679
130143 194211 240012 348018 129669 193701
130544 194658 240019 348028 129658 193684
130269 194365 240015 348022 129649 193671
130464 194528 240018 348026 129665 193693
130447 194538 240021 348030 129718 193773
130472 194591 240018 348026 129689 193729
130247 194287 240013 348019 129661 193689
131536 195818 240035 348051 129784 193867
130358 194439 240014 348021 129686 193725
130114 194223 240013 348018 129662 193691
130429 194528 240018 348026 129651 193673
130446 194495 240018 348027 129647 193667
130248 194322 240013 348019 129633 193647
130511 194600 240018 348026 129708 193758
131858 196051 240055 348080 129808 193900
130174 194253 240013 348018 129632 193646
130895 195069 240024 348035 129709 193758
130265 194356 240015 348021 129665 193695
130250 194253 240013 348019 129652 193675
131374 195543 240036 348052 129772 193849
131676 195902 240044 348064 129788 193872
131075 195226 240031 348045 129767 193844
130805 194946 240023 348034 129711 193762
130125 194202 240012 348018 129651 193674
130456 194562 240019 348028 129640 193656
130795 194951 240031 348044 129723 193778

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #4
From what I've learned myself:

1. I see LC3 technically being a dual-channel Opus (with different insignificant changes possibly, but it does a lot of things the same way I think);
2. You naturally get a lot worse result on tracks that do not use stereo extensively enough - because no joint stereo techniques are used in LC3, it's true stereo;
3. It still sounds impressive at 192 kbps, keeping in mind it's strict 96 kbps to every channel, 10 ms frame size and possibly a low-complexity mode comparing to what Opus does by default. And the main plus is that it's better than SBC at 238 kbps.
Opus VBR 256 + SoX

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #5
This is a paired comparison with my older test.

This listening test result was paired with my Nov 2020 test of AAC, SBC, aptX.
The AAC, SBC on the left side is the Nov 2020 test result, while the AAC, SBC on the right side is this May 2023 test result.


The closeup of the interesting section.


Score table. The score of the AAC and SBC is very similar, even though the two tests are about 2.5 years apart, the Nov 2020 test is 44.1kHz while the May 2023 test is 48kHz.
Again, the AAC, SBC, aptX on the left side is from the Nov 2020 test result, while the AAC, SBC, LC3 on the right side is from this May 2023 test result.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #6
According to the test that you did in 2022 ,  you where told that 80kbps/channel (160kbps stereo) is the minimum required bitrate, so testing it now at 128 is just wanting to make it look bad again.

In that thread it explains that this codec is meant for transmission and low latency, and in that scenario it is where it excels. Not in file-based scenarios compared to high delay codecs.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #7
True, low latency, but having LC3 at 128 kbps is still informative since combining these results with those here shows that LC3 provides roughly the same audio quality as mp3 (LAME) - a codec for file based use caes - at both 128 and 192 kbps.

Chris
If I don't reply to your reply, it means I agree with you.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #8
According to the test that you did in 2022 ,  you where told that 80kbps/channel (160kbps stereo) is the minimum required bitrate, so testing it now at 128 is just wanting to make it look bad again.
128 is meant to expose artifacts specific to the codec in an obvious way, and then proceed to 192, the rate we truly want to test.
Modern codecs at 192 tend to be so good that without a practice at lower rate, we may overlook the difference.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #9
According to the test that you did in 2022 ,  you where told that 80kbps/channel (160kbps stereo) is the minimum required bitrate, so testing it now at 128 is just wanting to make it look bad again.

In that thread it explains that this codec is meant for transmission and low latency, and in that scenario it is where it excels. Not in file-based scenarios compared to high delay codecs.


Assuming a 60 FPS video that's about almost a frame with a latency of 13.6 MS.  Most video is going to be at 30 or 25 FPS with most film at 24 FPS,  Not too bad for lip sync there with LC3.  Impressive.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #10
IMO it's important to note that FDK-AAC, and the AAC encoders actually used in the wild specifically for Bluetooth, can be different like night and day. 
I have an Android phone where if you choose AAC bluetooth codec (or, more accurately - forget to un-choose it, since it's the default where it's supported), it sounds abysmal, with obvious artifacts when there are e.g. busy electric guitars and a drum kit.
a fan of AutoEq + Meier Crossfeed

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #11
Are there significant audio quality differences between SBC encoders? For other audio formats, ffmpeg's native encoder implementations (AAC, Vorbis, Opus, WMA, ...) usually are not considered to equal the best respective non-ffmpeg encoders, especially if the format is dependent on robust psycho-acoustic modeling. For SBC, I'm not sure if there's much psycho-acoustic optimization in play.

For instance, another SBC encoder implementation appears to be this one from Broadcom, which is part of Android's Bluetooth stack: https://android.googlesource.com/platform/external/bluetooth/bluedroid/+/refs/heads/main/embdrv/sbc/encoder/srce/

edit: And Linux appears to use https://git.kernel.org/pub/scm/bluetooth/sbc.git/tree/src - this encoder seems to be packaged as "sbc-tools" in Debian.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #12
> For SBC, I'm not sure if there's much psycho-acoustic optimization in play.

AFAIK, there is a little room for choices based on psycho-acoustics.
In each slice, there's a number of frequency bands, which are encoded "independently", but their total size in bytes has a limit.
SBC encoder must choose which band gets how many bits, this ideally must be based on where the human listener will notice the distortion the least.
And for this choice I saw at least 2 distinct algorithms being mentioned (and only one widely used as the default).
There is probably some room for improvement in these algorithms to "squeeze the maximum quality" out of SBC, but I would guess it won't make a huge difference. There's also the problem that it's not practical to upgrade hardware SBC encoders anyway, so any work on improving a SBC encoder will not be delivered to some (arguably the most important) targets.
(Or does SBC encoding actually happen in special hardware? It's probably cheap enough to run on CPU in smartphones, too - but I don't know about that.)
a fan of AutoEq + Meier Crossfeed

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #13
IMO it's important to note that FDK-AAC, and the AAC encoders actually used in the wild specifically for Bluetooth, can be different like night and day. 
I have an Android phone where if you choose AAC bluetooth codec (or, more accurately - forget to un-choose it, since it's the default where it's supported), it sounds abysmal, with obvious artifacts when there are e.g. busy electric guitars and a drum kit.
IDK but other than ffmpeg's aac, maybe they use fdkaac with settings optimized for voice, or the soc's internal hw aac encoder. I'm pretty sure my android phone shows "/system/lib/libFraunhoferAAC.a" licence. I don't get why a manufaturer should adopt another codec when basically they have yet fdk one.

 

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #14
i think the bitrates chosen for AAC and especially for LC3 may be a little low. for example using the log console on macos, i can see that even my samsung earbuds always connect with 224 kbps to both my iphone and mac, and my bluetooth amp always connects with 256 kbps. (this is aac ofc since it's apple source devices)

this never changes unless i put the source a really unreasonable distance away from the receiver (which wouldn't really happen especially if you're using a phone to earbud connection) and i'd guess it's the same for android.

for the earbuds, aac probably transmits 2x224 kbps since tws gen 2 (where both devices get their data from the source, instead of one device sending data to the next) improves battery life so drastically on earbuds, i can't imagine modern earbuds would still use the older configuration - although i haven't really found a way to 100% confirm this.

but, in conclusion i think it's safe to assume that in any realistic scenario, LC3 will always at least use 224 kbps, more likely 256 kbps or even 320 kbps, since even in that mode it would transmit less data to earbuds than AAC does right now, due to the stereo signal being split in LC3.