Personal Blind Listening Test of Bluetooth codecs

Topic: Personal Blind Listening Test of Bluetooth codecs (Read 40378 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Personal Blind Listening Test of Bluetooth codecs

2020-11-11 08:41:45

Abstract:
Personal blind sound quality comparison of common Bluetooth codecs (SBC, AAC, aptX).

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)

SBC Settings:
FFmpeg's native SBC encoders and decoders were used.
ffmpeg -i in.wav -c:a sbc -b:a 237k out.sbc
ffmpeg -i in.wav -c:a sbc -b:a 280k out.sbc
ffmpeg -i in.wav -c:a sbc -b:a 328k out.sbc (common Bluetooth setting)

AAC Settings:
Fraunhofer FDK AAC Codec Library for Android was used as the AAC-LC encoder.
ffmpeg -i in.wav -c:a libfdk_aac -b:a 128k out.mp4
ffmpeg -i in.wav -c:a libfdk_aac -b:a 192k out.mp4 (common Bluetooth setting)

aptX Settings:
FFmpeg's native aptX encoders and decoders were used. It is NOT aptX LL nor aptX HD.
ffmpeg -i in.wav -c:a aptx -ar 32000 out.aptX
ffmpeg -i in.wav -c:a aptx -ar 44100 out.aptX (common Bluetooth setting at 44.1kHz)

Samples:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 sound samples.

Hardwares:
Sony PSP-3000 + RP-HT560.

Results, only the common Bluetooth settings:

All Results:

Tracks list:

Conclusions & Observations:

All Bluetooth codecs tested, SBC, AAC, and aptX offer high fidelity on the common settings.
AAC offers the fidelity comparable to SBC and aptX, on 41%-45% reduced bitrate.
In this test, Fraunhofer FDK AAC Codec Library for Android was used to evaluate the fidelity of the AAC codec. Android users use this for the Bluetooth connection. Apple iPhone users may be able to boost the fidelity even more by using Apple's CoreAudio engine, judging from Guruboolez test, but it cannot be confirmed from this test alone.
SBC significantly drops its fidelity, when the rate was adjusted to mere 15% less than the 328kbps setting.
aptX is not made to dynamically adjust its bitrate. Using 32kHz sampling rate instead of 44.1kHz severely affects fidelity.

Anova analysis:

Code: [Select]

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 21
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              146          52.59
Testers (blocks)    20          10.48
Codecs eval'd        6          22.71    3.79   23.42  0.00E+000
Error              120          19.40    0.16
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.246

Means:

fdk192   sbc328   apt352   sbc280   fdk128   apt256   sbc237   
  4.68     4.60     4.47     4.15     4.08     3.82     3.52   

---------------------------- p-value Matrix ---------------------------

         sbc328   apt352   sbc280   fdk128   apt256   sbc237   
fdk192   0.515    0.087    0.000*   0.000*   0.000*   0.000*   
sbc328            0.285    0.000*   0.000*   0.000*   0.000*   
apt352                     0.013*   0.002*   0.000*   0.000*   
sbc280                              0.540    0.008*   0.000*   
fdk128                                       0.040*   0.000*   
apt256                                                0.019*   
-----------------------------------------------------------------------

fdk192 is better than sbc280, fdk128, apt256, sbc237
sbc328 is better than sbc280, fdk128, apt256, sbc237
apt352 is better than sbc280, fdk128, apt256, sbc237
sbc280 is better than apt256, sbc237
fdk128 is better than apt256, sbc237
apt256 is better than sbc237

Raw data:

Code: [Select]

sbc237k	sbc280k	sbc328k	aac128k	aac192k	aptx256k	aptx352k
%feature	7	SBC	SBC	SBC	AAC	AAC	aptX	aptX
%feature	10	ffmpeg N-99419-g4fceb2634e	ffmpeg N-99419-g4fceb2634e	ffmpeg N-99419-g4fceb2634e	FDK-AAC v2.0.1	FDK-AAC v2.0.1	ffmpeg N-99419-g4fceb2634e	ffmpeg N-99419-g4fceb2634e
%feature	11	-c:a sbc	-c:a sbc	-c:a sbc	-c:a libfdk_aac	-c:a libfdk_aac	-c:a aptx	-c:a aptx
%feature 12 237kbps 281kbps 331kbps 130kbps 194kbps 256kbps 353kbps
3.800	4.700	5.000	4.300	4.700	3.900	4.600	
3.100	3.800	4.600	4.600	5.000	3.600	4.400	
3.900	4.700	5.000	4.100	5.000	4.400	5.000	
3.600	5.000	5.000	4.300	5.000	4.500	5.000	
4.300	4.400	5.000	3.600	4.500	3.400	3.800	
3.200	3.900	4.500	4.100	4.600	4.200	4.400	
3.900	4.500	4.400	5.000	5.000	4.100	4.700	
3.200	4.100	4.600	4.400	5.000	5.000	5.000	
3.000	3.100	4.800	3.300	3.800	4.300	4.600	
2.900	3.300	4.300	4.100	4.700	3.500	4.600	
4.200	4.600	4.700	3.600	4.300	3.700	4.800	
2.900	3.900	4.500	3.500	4.400	3.700	4.200	
3.700	4.500	4.700	4.600	5.000	3.400	4.200	
3.100	3.800	4.300	3.900	5.000	3.400	4.200	
2.900	3.500	3.800	4.100	5.000	3.300	4.300	
4.100	4.400	5.000	3.700	4.500	3.900	4.800	
4.700	5.000	5.000	3.700	4.400	3.800	5.000	
4.300	4.800	4.700	3.800	4.200	3.200	4.500	
3.200	3.800	4.400	4.100	4.700	4.200	4.500	
3.100	4.200	4.400	4.500	5.000	3.300	3.500	
2.900	3.200	3.900	4.300	4.500	3.400	3.700	
3.700	4.400	4.300	3.200	4.100	3.800	4.200	
3.200	3.900	4.200	3.800	4.400	3.600	5.000	
3.800	5.000	4.700	3.400	3.900	4.300	4.500	
3.600	4.600	4.700	4.200	5.000	3.800	4.500	
3.800	4.400	5.000	5.000	5.000	5.000	5.000	
3.800	4.400	4.400	4.500	5.000	4.200	4.600	
%samples	41_30sec	Perc.
%samples	finalfantasy	Strings
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	Classic
%samples	mybloodrusts	Guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart to Heart	Pops
%samples	Tom's Diner	Acappella
%samples	01 castanets	inst.
%samples	02 fatboy_30sec	Techno
%samples	03 eig	Techno
%samples	04 Bachpsichord	inst.
%samples	05 Enola	Techno
%samples	06 trumpet	inst.
%samples	07 applaud	Live
%samples	08 velvet	perc.
%samples	09 Linchpin	Rock
%samples	10 spill_the_blood	guitar
%samples	11 female_speech	Speech
%samples	12 French_Ad	Speech

Bitrates:

Code: [Select]

%bitrate
237036 	281135 	330747 	130016 	194031 	256000 	352800 
237006 	281100 	330706 	130850 	195057 	256000 	352800 
237030 	281128 	330739 	131475 	195827 	256001 	352800 
236969 	281057 	330655 	131082 	195248 	256000 	352800 
237027 	281125 	330735 	130561 	194707 	256000 	352800 
237024 	281121 	330731 	129960 	194015 	256000 	352800 
236965 	281051 	330648 	131447 	195921 	255999 	352800 
237028 	281126 	330737 	130083 	194180 	256000 	352800 
237016 	281112 	330720 	130659 	194779 	256000 	352800 
237028 	281126 	330736 	129955 	194043 	256000 	352800 
237016 	281112 	330720 	129951 	194008 	256000 	352800 
237015 	281110 	330718 	130240 	194347 	256000 	352800 
237028 	281126 	330736 	129990 	194050 	256000 	352800 
236995 	281087 	330690 	130563 	194840 	256001 	352800 
237008 	281102 	330708 	130306 	194432 	256000 	352800 
237011 	281106 	330713 	130050 	194111 	256000 	352800 
237032 	281131 	330743 	130262 	194347 	256000 	352800 
237026 	281124 	330735 	130223 	194454 	256000 	352800 
237008 	281103 	330709 	130183 	194246 	256000 	352800 
237028 	281126 	330737 	129975 	194041 	256000 	352800 
236973 	281061 	330660 	131100 	195263 	256000 	352800 
237037 	281136 	330749 	130049 	194117 	256000 	352800 
237015 	281111 	330719 	129868 	193955 	256000 	352800 
237037 	281137 	330750 	130167 	194256 	256000 	352800 
237005 	281099 	330705 	130234 	194273 	256000 	352800 
237020 	281117 	330725 	130062 	194140 	256000 	352800 
237009 	281104 	330711 	130241 	194337 	256000 	352800

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #1 – 2020-11-11 09:39:20

Captivating! I was waiting for such comparison for years

I didn't know that SBC and aptX were available in FFMPEG. Great test, thank you very much for this huge effort!

I'm surprised by aptX lower efficiency. I always thought this format was introduced to increase quality. I don't know if FFMPEG's implementation is identical to what audio devices are doing but from this test it's clear that this format is more a selling feature than a real progress in audio quality (maybe a better latency?).

AAC is in a different league though. The most important benefit in my opinion is the perceptual quality at 128 kbps which remains high. Funny though: FDK-AAC-128 ends here at 4.06—and in my own test it ends at 4.04

All formats at full-bitrate are excellent though. SBC subband format seems to handle very well preecho (castanets, eig, applauds…) but is less at ease with tonal samples (trumpet, enola).

Well done!

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #2 – 2020-11-11 14:49:01

Damn, this is listening test season

Quote from: Kamedo2 on 2020-11-11 08:41:45

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)

While this kind of test is rare and indeed very interesting, we have to be careful not to draw any conclusions on Bluetooth codecs used in practical situations (meaning when used with phones, cars, etc.) because the encoders used are most likely different, and may exhibit different qualities/flaws.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #3 – 2020-11-11 22:11:05

Quote from: guruboolez on 2020-11-11 09:39:20

I'm surprised by aptX lower efficiency. I always thought this format was introduced to increase quality. I don't know if FFMPEG's implementation is identical to what audio devices are doing but from this test it's clear that this format is more a selling feature than a real progress in audio quality (maybe a better latency?).

Don't forget that aptX is also an ADPCM format, so its main selling point is probably that it's easier to encode and decode.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #4 – 2020-11-13 12:12:56

Quote from: dutch109 on 2020-11-11 14:49:01

Damn, this is listening test season

Quote from: Kamedo2 on 2020-11-11 08:41:45
Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)
While this kind of test is rare and indeed very interesting, we have to be careful not to draw any conclusions on Bluetooth codecs used in practical situations (meaning when used with phones, cars, etc.) because the encoders used are most likely different, and may exhibit different qualities/flaws.

AFAIK, the development of the SBC codec focuses on speed, not the quality. The quality of the SBC is left unchanged. It's safe to assume that devices are shipped with the same-old encoders.
https://www.openhub.net/p/sbc/commits/summary
http://git.videolan.org/?p=ffmpeg.git&a=search&h=HEAD&st=commit&s=sbc

I suspect that aptX cannot change its quality, as it is two layers of QMF + APCM.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #5 – 2020-11-14 00:20:11

As indicated above, the only way to effectively change the quality of aptX is to change its sample rate. Then you may be pushing the resulting quantization noise closer to being audible to you, depending on how high your hearing goes.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #6 – 2020-11-14 07:06:59

Is there any way that aptX implementations could reasonably differ audibly, like if an encoder used a less precise algorithm?

(I'm pretty sure the difference wouldn't be anywhere near as big as an early mp3 encoder and something vaguely competent, though)

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #7 – 2020-11-14 11:49:34

Wondering how close is ffmpeg's SBC encoder to the one used in Android (10). Even if the bitrate is the same, encoder is theoretically allowed to make different decisions about distributing bits between bands, and stuff like that.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #8 – 2020-11-19 09:50:55

The bitrate vs score graph.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #9 – 2020-12-12 22:20:54

It's so cool to come back here after maybe 10 (correction: 6) years and see several members' names that I actually remember from way back then.

Really just wanted to say thanks for a very interesting listening test. I've been reading up on Bluetooth codecs recently and this was a valuable addition to my knowledge.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #10 – 2020-12-16 10:09:18

Quote from: Kamedo2 on 2020-11-19 09:50:55

The bitrate vs score graph.

APTX codec not reencode mp3 files, therefore there aren't sound degradation with mp3 files.
I just find APTX LOW LATENCY codec, is able give best quality in common bluetooth codecs, but only with CSR chip - I don't know why but this chip make sound a little bit compressed but more balanced. I talking about playing with F2K wave and flac files.
I use that device - Avantree Aptx Low Latency Bluetooth Transmitter And Receiver 2-In-1, Wireless 3. - https://www.ebay.com/c/938285992

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #11 – 2020-12-17 02:21:29

Quote from: Priest_1 on 2020-12-16 10:09:18

APTX codec not reencode mp3 files, therefore there aren't sound degradation with mp3 files.

Yes it does, and yes there is. The aptX encoder is supposed to use a high enough bitrate that you won't hear it, but it's always there.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #12 – 2020-12-18 10:13:02

As I've said from examining the open source encoder, it's an ADPCM codec. It's supposed to be generally low latency and fairly high quality, you just need to encode at a high enough sample rate. It's constant bitrate, regardless of audio complexity. The only way to reduce the bitrate is to also reduce the frequency response.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #13 – 2020-12-19 21:50:10

I mean, in theory shouldn't any artifacts be along the lines of analog noise?

(But I'm a weirdo for whom SBC sounds "fine")

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #14 – 2020-12-20 01:02:31

For a codec like AptX, yes, they'll sound like analog noise. For SBC or AAC, they'll sound different.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #15 – 2020-12-20 19:03:40

Yes, that's what I meant, sorry.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #16 – 2021-02-02 20:56:20

Quote from: magicgoose on 2020-11-14 11:49:34

Wondering how close is ffmpeg's SBC encoder to the one used in Android (10). Even if the bitrate is the same, encoder is theoretically allowed to make different decisions about distributing bits between bands, and stuff like that.

It has been a while since I looked at the standard, but I think this is incorrect. Given a valid set of CODEC parameters (e.g. nrof_subbands, nrof_blocks, and bitpool), I believe the encoder is deterministic. Meaning that, if you give two different SBC encoders a specific audio file to encode, and the same set of parameters, you would get the same, bit-perfect output from both of them.

There are two ways to allocate bits to sub-bands, SNR and LOUDNESS, but this parameter is negotiated when the connection is initiated. The bit allocation of course changes with the input audio file, but there is no reason to think that, if two separate encoders, both adhering to the Standard, with the same encoder parameters, would allocate the bits to sub-bands differently.

Of course, not all encoders adhere to the Standard. All Bluetooth A2DP audio devices must support SBC. Further, the Standard states which encoder parameters must be supported by the SBC encoder. The Standard also provides some example "known good combinations" of these parameters. Many SBC encoders use (and test) only these example "known good combinations", despite advertising during connection initiation that that other combinations are supported. If you try to set up a connection with less-frequently-used, but supposedly supported, parameter combinations, real-world SBC encoders often just stop working until it is reset.

It is a shame, as better use of the flexibility built into SBC likely could have avoided all these later, proprietary CODECs. (SBC is open source.)

If you want to learn more, see my white-paper:
https://drive.google.com/file/d/1J40QF_gZfW7Lo35lc_pgpGi8MZEeWGVt/view?usp=sharing

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #17 – 2021-08-01 07:53:09

The original anova analysis was wrong; it used only the 21 tracks. It somehow ignored the last 5 tracks.
This is the correct anova analysis.
It won't change the final conclusions.

Code: [Select]

C:\Users\[removed]>friedman.exe -a bluetoothc.txt
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 27
Critical significance:  0.05
Significance of data: 2.22E-016 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              188          64.42
Testers (blocks)    26          13.67
Codecs eval'd        6          26.69    4.45   28.85  2.22E-016
Error              156          24.05    0.15
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.211

Means:

aac192k  sbc328k  aptx352k sbc280k  aac128k  aptx256k sbc237k
  4.66     4.59     4.50     4.22     4.06     3.89     3.55

---------------------------- p-value Matrix ---------------------------

         sbc328k  aptx352k sbc280k  aac128k  aptx256k sbc237k
aac192k  0.534    0.157    0.000*   0.000*   0.000*   0.000*
sbc328k           0.427    0.001*   0.000*   0.000*   0.000*
aptx352k                   0.008*   0.000*   0.000*   0.000*
sbc280k                             0.148    0.002*   0.000*
aac128k                                      0.098    0.000*
aptx256k                                              0.002*
-----------------------------------------------------------------------

aac192k is better than sbc280k, aac128k, aptx256k, sbc237k
sbc328k is better than sbc280k, aac128k, aptx256k, sbc237k
aptx352k is better than sbc280k, aac128k, aptx256k, sbc237k
sbc280k is better than aptx256k, sbc237k
aac128k is better than sbc237k
aptx256k is better than sbc237k

C:\Users\[removed]>

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #18 – 2021-09-09 10:22:50

Hey, thanks for the comparison! One thing that leaves me wondering though is this - most people (maybe not here but in general) don't listen to lossless tracks, they're probably streaming music that's either encoded as 128-320 kbit/s mp3 or 128-256 kbit/s aac. So what they would hear with their bluetooth device is actually twice encoded music, once by the streaming service and once by bluetooth. (AFAIK no devices actually passthrough aac without re-encoding it.) This might change the results a bit, but I have no idea in which way. It might be that re-encoding 128 kbit/s aac with much higher bitrates results in imperceptible differences, or it might be that there's so little data left for the encoder to see that it sounds much worse.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #19 – 2022-09-15 11:33:04

This would be a really great test if a lot of phones didn't use crappy hardware AAC encoders instead. Bluetooth AAC on Android is often unbelievably bad.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #20 – 2022-11-13 22:10:41

Quote

AAC Settings:
Fraunhofer FDK AAC Codec Library for Android was used as the AAC-LC encoder.
ffmpeg -i in.wav -c:a libfdk_aac -b:a 128k out.mp4
ffmpeg -i in.wav -c:a libfdk_aac -b:a 192k out.mp4 (common Bluetooth setting)

just wondering, ffmpeg aac encoding guide suggest to increase cutoff frequency for the libfdk_aac encoder from 14kHz to 18kHz, fdkaac wiki report 16120Hz as cutoff for 2+ channels CBR mode 72-96kpbs/ch and 17kHz for 96+kbps/ch.

how much those settings (plain -b:a 192k) will comply with how android phones and linux os (also apple devices) will encode their aac bt streams?

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #21 – 2022-11-14 12:21:36

Quote from: kenta_hirono on 2022-11-13 22:10:41

how much those settings (plain -b:a 192k) will comply with how android phones and linux os (also apple devices) will encode their aac bt streams?

Mostly they do not comply at all because most phones use hardware encoding instead of libfdk_aac. Which is a very sad thing.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #22 – 2022-11-14 15:04:53

Quote from: Eurobeat_fan on 2022-11-14 12:21:36

Quote from: kenta_hirono on 2022-11-13 22:10:41
how much those settings (plain -b:a 192k) will comply with how android phones and linux os (also apple devices) will encode their aac bt streams?
Mostly they do not comply at all because most phones use hardware encoding instead of libfdk_aac. Which is a very sad thing.

That's a point, but imho to compare the same codec it's better to use the same cutoff or to use a bt receiver with an actual phone (or a selection of the most diffused).
Also they can use an hw encoder for sbc and apt-x too.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #23 – 2022-11-14 15:11:11

Quote from: kenta_hirono on 2022-11-13 22:10:41

just wondering, ffmpeg aac encoding guide suggest to increase cutoff frequency for the libfdk_aac encoder from 14kHz to 18kHz

I don't know if the suggestion posted in the FFmpeg wiki is valid, or the default is better.

Quote from: kenta_hirono on 2022-11-13 22:10:41

fdkaac wiki report 16120Hz as cutoff for 2+ channels CBR mode 72-96kpbs/ch and 17kHz for 96+kbps/ch.

Confirmed. From the encoded AAC files used in this listening test and judged from the spectrogram, the cutoff of the ffmpeg -i in.wav -c:a libfdk_aac -b:a 128k out.mp4 was around 15.5 kHz, and the cutoff of the ffmpeg -i in.wav -c:a libfdk_aac -b:a 192k out.mp4 was around 17 kHz.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #24 – 2024-07-20 17:21:11

Could someone please re-share IgorC's samples?
The Google drive link is not working.

Notice