Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3. (Read 8701 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Abstract:
Personal blind sound quality comparison of the Bluetooth codec, AAC and LC3, at 192 kbps, and SBC at 345 kbps.
AAC is an optional codec in Bluetooth, but Bluetooth earphones and speakers usually support the AAC codec.
SBC is the mandatory codec in Bluetooth.
LC3 is a new codec, which became mandatory in the LE Audio in Bluetooth 5.2.
Lower bit rate settings were also tested.
The test was conducted on 48 kHz input on all codecs.

Encoders:
AAC: fdkaac-1.0.4 built against fdk-aac 2.0.2, x64, from rarewares, Feb 2023 version (not the latest version, as of May 2023).
SBC: FFmpeg N-109823-g385ec46424, Feb 2023 version (not the latest version, as of May 2023).
LC3: Google / liblc3 v1.0.1, Feb 2023 version (not the latest version, as of May 2023).

AAC Settings:
fdkaac -b 128000 -o out.mp4 in.48kHz.16bit2ch.wav
fdkaac -b 192000 -o out.mp4 in.48kHz.16bit2ch.wav

SBC Settings:
ffmpeg -y -i in.48kHz.16bit2ch.wav -c:a sbc -b:a 237k out.sbc
ffmpeg -y -i in.48kHz.16bit2ch.wav -c:a sbc -b:a 345k out.sbc

LC3 Settings:
elc3 -b 128000 in.48kHz.16bit2ch.wav out.lc3
elc3 -b 192000 in.48kHz.16bit2ch.wav out.lc3

Sample tracks:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 diverse music and speech sound samples.

Hardware:
Sony PSP-3000 + AKG K712.

Results (only the higher bitrate):


Results (including lower bitrates):



Conclusions & Observations:
  • AAC offered very high fidelity at 192 kbps. Most scores were better than 4.0 (Perceptible, but not annoying).
  • SBC also offered very high fidelity at 345 kbps. Most scores were better than 4.0 (Perceptible, but not annoying).
  • LC3, also offered very high fidelity at 192 kbps, with majority of scores better than 4.0. Still not as good as AAC or SBC.
  • It's not clear whether AAC at 192 kbps is better, or SBC at 345 kbps is better from this test alone.
  • Reducing the bitrate to about two-thirds significantly impaired sound quality on all Bluetooth codecs tested.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              161          69.27
Testers (blocks)    26           9.59
Codecs eval'd        5          37.25    7.45   43.18  0.00E+000
Error              130          22.43    0.17
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.224

Means:

FDK-AAC1 SBC345kb liblc319 FDK-AAC1 SBC237kb liblc312 
  4.72     4.65     4.13     4.12     3.62     3.42   

---------------------------- p-value Matrix ---------------------------

         SBC345kb liblc319 FDK-AAC1 SBC237kb liblc312 
FDK-AAC1 0.578    0.000*   0.000*   0.000*   0.000*   
SBC345kb          0.000*   0.000*   0.000*   0.000*   
liblc319                   0.961    0.000*   0.000*   
FDK-AAC1                            0.000*   0.000*   
SBC237kb                                     0.079    
-----------------------------------------------------------------------

FDK-AAC192kbps is better than liblc3192kbps, FDK-AAC128kbps, SBC237kbps, liblc3128kbps
SBC345kbps is better than liblc3192kbps, FDK-AAC128kbps, SBC237kbps, liblc3128kbps
liblc3192kbps is better than SBC237kbps, liblc3128kbps
FDK-AAC128kbps is better than SBC237kbps, liblc3128kbps

Raw data:
By mistake, 04 Bachpsichord and 05 Enola were tested twice. The two scores were averaged in below.
Code: [Select]
FDK-AAC128kbps	FDK-AAC192kbps	liblc3128kbps	liblc3192kbps	SBC237kbps	SBC345kbps	
%genre Kamedo2's 15 sample
3.700	4.500	3.200	4.200	3.500	5.000	
4.800	4.700	3.300	4.100	3.100	4.500	
3.900	5.000	3.100	3.500	3.700	5.000	
4.100	5.000	3.500	4.400	3.700	5.000	
3.900	5.000	3.300	4.500	4.200	5.000	
4.200	5.000	3.300	3.900	2.800	4.400	
4.900	5.000	3.400	4.500	3.700	4.700	
4.200	5.000	3.400	4.100	3.100	4.300	
3.100	3.800	3.500	4.100	3.300	4.200	
3.900	5.000	3.100	3.800	2.700	5.000	
3.800	4.500	3.500	4.200	4.300	4.400	
3.700	5.000	3.400	4.100	3.500	4.600	
4.300	5.000	3.400	3.900	3.100	4.500	
4.200	4.800	3.300	4.100	2.700	5.000	
5.000	4.700	2.900	3.700	2.600	3.800	

%genre IgorC's 12 sample
4.400	4.700	3.400	3.900	4.600	5.000	
4.200	4.500	3.900	4.400	5.000	5.000	
4.300	4.500	3.500	3.900	4.800	5.000	
4.000	4.800	3.200	4.150	2.800	4.450	
4.000	4.750	3.250	4.200	3.450	4.000	
4.200	4.300	3.200	3.800	2.500	3.900	
3.800	4.100	3.400	4.300	5.000	4.400	
3.700	5.000	3.600	4.200	3.900	5.000	
3.400	4.100	3.700	4.200	4.300	4.500	
4.300	4.600	3.200	4.200	3.900	5.000	
5.000	5.000	5.000	5.000	3.700	5.000	
4.300	5.000	3.400	4.100	3.800	5.000	

%samples	41_30sec	Perc.
%samples	finalfantasy	Strings
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	Classic
%samples	mybloodrusts	Guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart to Heart	Pops
%samples	Tom's Diner	Acappella

%samples	01 castanets	inst.
%samples	02 fatboy_30sec	Techno
%samples	03 eig	Techno
%samples	04 Bachpsichord	inst.
%samples	05 Enola	Techno
%samples	06 trumpet	inst.
%samples	07 applaud	Live
%samples	08 velvet	perc.
%samples	09 Linchpin	Rock
%samples	10 spill_the_blood	guitar
%samples	11 female_speech	Speech
%samples	12 French_Ad	Speech

Other tests:

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #1 –
Correct me if I am wrong.
The default frame size / latency of LC3 is 10ms. And fdkaac default frame size around 55ms or possibly higher.

Which makes the results for LC3 pretty damn impressive. But are there any reason why 256kbps or 360kbps LC3 wasn't tested? I would imagine most bluetooth should be capable supporting those bitrate reliably.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #2 –
Quote from: iwod on
But are there any reason why 256kbps or 360kbps LC3 wasn't tested? I would imagine most bluetooth should be capable supporting those bitrate reliably.
Based on Bluetooth SIG's website, I had thought that LC3 at 192 kbps should be close to 5.0 and set it to 192 kbps. Based on this listening test result, using somewhere around 256 kbps sounds like a good idea.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #3 –
As for the bitrate, All codecs were tested with CBR settings, so the file-based bitrate very constant.

Code: [Select]
%bitrate
FDK-AAC128kbps	FDK-AAC192kbps	SBC237kbps	SBC345kbps	liblc3_128kbps	liblc3_192kbps
130216	194306	240012	348018	129655	193679
130143	194211	240012	348018	129669	193701
130544	194658	240019	348028	129658	193684
130269	194365	240015	348022	129649	193671
130464	194528	240018	348026	129665	193693
130447	194538	240021	348030	129718	193773
130472	194591	240018	348026	129689	193729
130247	194287	240013	348019	129661	193689
131536	195818	240035	348051	129784	193867
130358	194439	240014	348021	129686	193725
130114	194223	240013	348018	129662	193691
130429	194528	240018	348026	129651	193673
130446	194495	240018	348027	129647	193667
130248	194322	240013	348019	129633	193647
130511	194600	240018	348026	129708	193758
131858	196051	240055	348080	129808	193900
130174	194253	240013	348018	129632	193646
130895	195069	240024	348035	129709	193758
130265	194356	240015	348021	129665	193695
130250	194253	240013	348019	129652	193675
131374	195543	240036	348052	129772	193849
131676	195902	240044	348064	129788	193872
131075	195226	240031	348045	129767	193844
130805	194946	240023	348034	129711	193762
130125	194202	240012	348018	129651	193674
130456	194562	240019	348028	129640	193656
130795	194951	240031	348044	129723	193778

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #4 –
From what I've learned myself:

1. I see LC3 technically being a dual-channel Opus (with different insignificant changes possibly, but it does a lot of things the same way I think);
2. You naturally get a lot worse result on tracks that do not use stereo extensively enough - because no joint stereo techniques are used in LC3, it's true stereo;
3. It still sounds impressive at 192 kbps, keeping in mind it's strict 96 kbps to every channel, 10 ms frame size and possibly a low-complexity mode comparing to what Opus does by default. And the main plus is that it's better than SBC at 238 kbps.
Opus VBR 256 + SoX

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #5 –
This is a paired comparison with my older test.

This listening test result was paired with my Nov 2020 test of AAC, SBC, aptX.
The AAC, SBC on the left side is the Nov 2020 test result, while the AAC, SBC on the right side is this May 2023 test result.


The closeup of the interesting section.


Score table. The score of the AAC and SBC is very similar, even though the two tests are about 2.5 years apart, the Nov 2020 test is 44.1kHz while the May 2023 test is 48kHz.
Again, the AAC, SBC, aptX on the left side is from the Nov 2020 test result, while the AAC, SBC, LC3 on the right side is from this May 2023 test result.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #6 –
According to the test that you did in 2022 ,  you where told that 80kbps/channel (160kbps stereo) is the minimum required bitrate, so testing it now at 128 is just wanting to make it look bad again.

In that thread it explains that this codec is meant for transmission and low latency, and in that scenario it is where it excels. Not in file-based scenarios compared to high delay codecs.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #7 –
True, low latency, but having LC3 at 128 kbps is still informative since combining these results with those here shows that LC3 provides roughly the same audio quality as mp3 (LAME) - a codec for file based use caes - at both 128 and 192 kbps.

Chris
If I don't reply to your reply, it means I agree with you.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #8 –
Quote from: [JAZ] on
According to the test that you did in 2022 ,  you where told that 80kbps/channel (160kbps stereo) is the minimum required bitrate, so testing it now at 128 is just wanting to make it look bad again.
128 is meant to expose artifacts specific to the codec in an obvious way, and then proceed to 192, the rate we truly want to test.
Modern codecs at 192 tend to be so good that without a practice at lower rate, we may overlook the difference.
 

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #9 –
Quote from: [JAZ] on
According to the test that you did in 2022 ,  you where told that 80kbps/channel (160kbps stereo) is the minimum required bitrate, so testing it now at 128 is just wanting to make it look bad again.

In that thread it explains that this codec is meant for transmission and low latency, and in that scenario it is where it excels. Not in file-based scenarios compared to high delay codecs.


Assuming a 60 FPS video that's about almost a frame with a latency of 13.6 MS.  Most video is going to be at 30 or 25 FPS with most film at 24 FPS,  Not too bad for lip sync there with LC3.  Impressive.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #10 –
IMO it's important to note that FDK-AAC, and the AAC encoders actually used in the wild specifically for Bluetooth, can be different like night and day. 
I have an Android phone where if you choose AAC bluetooth codec (or, more accurately - forget to un-choose it, since it's the default where it's supported), it sounds abysmal, with obvious artifacts when there are e.g. busy electric guitars and a drum kit.
a fan of AutoEq + Meier Crossfeed

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #11 –
Are there significant audio quality differences between SBC encoders? For other audio formats, ffmpeg's native encoder implementations (AAC, Vorbis, Opus, WMA, ...) usually are not considered to equal the best respective non-ffmpeg encoders, especially if the format is dependent on robust psycho-acoustic modeling. For SBC, I'm not sure if there's much psycho-acoustic optimization in play.

For instance, another SBC encoder implementation appears to be this one from Broadcom, which is part of Android's Bluetooth stack: https://android.googlesource.com/platform/external/bluetooth/bluedroid/+/refs/heads/main/embdrv/sbc/encoder/srce/

edit: And Linux appears to use https://git.kernel.org/pub/scm/bluetooth/sbc.git/tree/src - this encoder seems to be packaged as "sbc-tools" in Debian.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #12 –
> For SBC, I'm not sure if there's much psycho-acoustic optimization in play.

AFAIK, there is a little room for choices based on psycho-acoustics.
In each slice, there's a number of frequency bands, which are encoded "independently", but their total size in bytes has a limit.
SBC encoder must choose which band gets how many bits, this ideally must be based on where the human listener will notice the distortion the least.
And for this choice I saw at least 2 distinct algorithms being mentioned (and only one widely used as the default).
There is probably some room for improvement in these algorithms to "squeeze the maximum quality" out of SBC, but I would guess it won't make a huge difference. There's also the problem that it's not practical to upgrade hardware SBC encoders anyway, so any work on improving a SBC encoder will not be delivered to some (arguably the most important) targets.
(Or does SBC encoding actually happen in special hardware? It's probably cheap enough to run on CPU in smartphones, too - but I don't know about that.)
a fan of AutoEq + Meier Crossfeed

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #13 –
Quote from: magicgoose on
IMO it's important to note that FDK-AAC, and the AAC encoders actually used in the wild specifically for Bluetooth, can be different like night and day. 
I have an Android phone where if you choose AAC bluetooth codec (or, more accurately - forget to un-choose it, since it's the default where it's supported), it sounds abysmal, with obvious artifacts when there are e.g. busy electric guitars and a drum kit.
IDK but other than ffmpeg's aac, maybe they use fdkaac with settings optimized for voice, or the soc's internal hw aac encoder. I'm pretty sure my android phone shows "/system/lib/libFraunhoferAAC.a" licence. I don't get why a manufaturer should adopt another codec when basically they have yet fdk one.

Re: Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.

Reply #14 –
i think the bitrates chosen for AAC and especially for LC3 may be a little low. for example using the log console on macos, i can see that even my samsung earbuds always connect with 224 kbps to both my iphone and mac, and my bluetooth amp always connects with 256 kbps. (this is aac ofc since it's apple source devices)

this never changes unless i put the source a really unreasonable distance away from the receiver (which wouldn't really happen especially if you're using a phone to earbud connection) and i'd guess it's the same for android.

for the earbuds, aac probably transmits 2x224 kbps since tws gen 2 (where both devices get their data from the source, instead of one device sending data to the next) improves battery life so drastically on earbuds, i can't imagine modern earbuds would still use the older configuration - although i haven't really found a way to 100% confirm this.

but, in conclusion i think it's safe to assume that in any realistic scenario, LC3 will always at least use 224 kbps, more likely 256 kbps or even 320 kbps, since even in that mode it would transmit less data to earbuds than AAC does right now, due to the stereo signal being split in LC3.