Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Blind Listening Test of Bluetooth codecs (Read 8250 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Blind Listening Test of Bluetooth codecs

Abstract:
Personal blind sound quality comparison of common Bluetooth codecs (SBC, AAC, aptX).

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)

SBC Settings:
FFmpeg's native SBC encoders and decoders were used.
ffmpeg -i in.wav -c:a sbc -b:a 237k out.sbc
ffmpeg -i in.wav -c:a sbc -b:a 280k out.sbc
ffmpeg -i in.wav -c:a sbc -b:a 328k out.sbc (common Bluetooth setting)

AAC Settings:
Fraunhofer FDK AAC Codec Library for Android was used as the AAC-LC encoder.
ffmpeg -i in.wav -c:a libfdk_aac -b:a 128k out.mp4
ffmpeg -i in.wav -c:a libfdk_aac -b:a 192k out.mp4 (common Bluetooth setting)

aptX Settings:
FFmpeg's native aptX encoders and decoders were used. It is NOT aptX LL nor aptX HD.
ffmpeg -i in.wav -c:a aptx -ar 32000 out.aptX
ffmpeg -i in.wav -c:a aptx -ar 44100 out.aptX (common Bluetooth setting at 44.1kHz)

Samples:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 sound samples.

Hardwares:
Sony PSP-3000 + RP-HT560.

Results, only the common Bluetooth settings:


All Results:


Tracks list:


Conclusions & Observations:
  • All Bluetooth codecs tested, SBC, AAC, and aptX offer high fidelity on the common settings.
  • AAC offers the fidelity comparable to SBC and aptX, on 41%-45% reduced bitrate.
  • In this test, Fraunhofer FDK AAC Codec Library for Android was used to evaluate the fidelity of the AAC codec. Android users use this for the Bluetooth connection. Apple iPhone users may be able to boost the fidelity even more by using Apple's CoreAudio engine, judging from Guruboolez test, but it cannot be confirmed from this test alone.
  • SBC significantly drops its fidelity, when the rate was adjusted to mere 15% less than the 328kbps setting.
  • aptX is not made to dynamically adjust its bitrate. Using 32kHz sampling rate instead of 44.1kHz severely affects fidelity.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 21
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              146          52.59
Testers (blocks)    20          10.48
Codecs eval'd        6          22.71    3.79   23.42  0.00E+000
Error              120          19.40    0.16
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.246

Means:

fdk192   sbc328   apt352   sbc280   fdk128   apt256   sbc237  
  4.68     4.60     4.47     4.15     4.08     3.82     3.52  

---------------------------- p-value Matrix ---------------------------

         sbc328   apt352   sbc280   fdk128   apt256   sbc237  
fdk192   0.515    0.087    0.000*   0.000*   0.000*   0.000*  
sbc328            0.285    0.000*   0.000*   0.000*   0.000*  
apt352                     0.013*   0.002*   0.000*   0.000*  
sbc280                              0.540    0.008*   0.000*  
fdk128                                       0.040*   0.000*  
apt256                                                0.019*  
-----------------------------------------------------------------------

fdk192 is better than sbc280, fdk128, apt256, sbc237
sbc328 is better than sbc280, fdk128, apt256, sbc237
apt352 is better than sbc280, fdk128, apt256, sbc237
sbc280 is better than apt256, sbc237
fdk128 is better than apt256, sbc237
apt256 is better than sbc237


Raw data:
Code: [Select]
sbc237k	sbc280k	sbc328k	aac128k	aac192k	aptx256k	aptx352k
%feature 7 SBC SBC SBC AAC AAC aptX aptX
%feature 10 ffmpeg N-99419-g4fceb2634e ffmpeg N-99419-g4fceb2634e ffmpeg N-99419-g4fceb2634e FDK-AAC v2.0.1 FDK-AAC v2.0.1 ffmpeg N-99419-g4fceb2634e ffmpeg N-99419-g4fceb2634e
%feature 11 -c:a sbc -c:a sbc -c:a sbc -c:a libfdk_aac -c:a libfdk_aac -c:a aptx -c:a aptx
%feature 12 237kbps 281kbps 331kbps 130kbps 194kbps 256kbps 353kbps
3.800 4.700 5.000 4.300 4.700 3.900 4.600
3.100 3.800 4.600 4.600 5.000 3.600 4.400
3.900 4.700 5.000 4.100 5.000 4.400 5.000
3.600 5.000 5.000 4.300 5.000 4.500 5.000
4.300 4.400 5.000 3.600 4.500 3.400 3.800
3.200 3.900 4.500 4.100 4.600 4.200 4.400
3.900 4.500 4.400 5.000 5.000 4.100 4.700
3.200 4.100 4.600 4.400 5.000 5.000 5.000
3.000 3.100 4.800 3.300 3.800 4.300 4.600
2.900 3.300 4.300 4.100 4.700 3.500 4.600
4.200 4.600 4.700 3.600 4.300 3.700 4.800
2.900 3.900 4.500 3.500 4.400 3.700 4.200
3.700 4.500 4.700 4.600 5.000 3.400 4.200
3.100 3.800 4.300 3.900 5.000 3.400 4.200
2.900 3.500 3.800 4.100 5.000 3.300 4.300
4.100 4.400 5.000 3.700 4.500 3.900 4.800
4.700 5.000 5.000 3.700 4.400 3.800 5.000
4.300 4.800 4.700 3.800 4.200 3.200 4.500
3.200 3.800 4.400 4.100 4.700 4.200 4.500
3.100 4.200 4.400 4.500 5.000 3.300 3.500
2.900 3.200 3.900 4.300 4.500 3.400 3.700
3.700 4.400 4.300 3.200 4.100 3.800 4.200
3.200 3.900 4.200 3.800 4.400 3.600 5.000
3.800 5.000 4.700 3.400 3.900 4.300 4.500
3.600 4.600 4.700 4.200 5.000 3.800 4.500
3.800 4.400 5.000 5.000 5.000 5.000 5.000
3.800 4.400 4.400 4.500 5.000 4.200 4.600
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella
%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech


Bitrates:
Code: [Select]
%bitrate
237036 281135 330747 130016 194031 256000 352800
237006 281100 330706 130850 195057 256000 352800
237030 281128 330739 131475 195827 256001 352800
236969 281057 330655 131082 195248 256000 352800
237027 281125 330735 130561 194707 256000 352800
237024 281121 330731 129960 194015 256000 352800
236965 281051 330648 131447 195921 255999 352800
237028 281126 330737 130083 194180 256000 352800
237016 281112 330720 130659 194779 256000 352800
237028 281126 330736 129955 194043 256000 352800
237016 281112 330720 129951 194008 256000 352800
237015 281110 330718 130240 194347 256000 352800
237028 281126 330736 129990 194050 256000 352800
236995 281087 330690 130563 194840 256001 352800
237008 281102 330708 130306 194432 256000 352800
237011 281106 330713 130050 194111 256000 352800
237032 281131 330743 130262 194347 256000 352800
237026 281124 330735 130223 194454 256000 352800
237008 281103 330709 130183 194246 256000 352800
237028 281126 330737 129975 194041 256000 352800
236973 281061 330660 131100 195263 256000 352800
237037 281136 330749 130049 194117 256000 352800
237015 281111 330719 129868 193955 256000 352800
237037 281137 330750 130167 194256 256000 352800
237005 281099 330705 130234 194273 256000 352800
237020 281117 330725 130062 194140 256000 352800
237009 281104 330711 130241 194337 256000 352800

 

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #1
Captivating! I was waiting for such comparison for years :) I didn't know that SBC and aptX were available in FFMPEG. Great test, thank you very much for this huge effort!

I'm surprised by aptX lower efficiency. I always thought this format was introduced to increase quality. I don't know if FFMPEG's implementation is identical to what audio devices are doing but from this test it's clear that this format is more a selling feature than a real progress in audio quality (maybe a better latency?).

AAC is in a different league though. The most important benefit in my opinion is the perceptual quality at 128 kbps which remains high. Funny though: FDK-AAC-128 ends here at 4.06—and in my own test it ends at 4.04  :)

All formats at full-bitrate are excellent though. SBC subband format seems to handle very well preecho (castanets, eig, applauds…) but is less at ease with tonal samples (trumpet, enola).

Well done!

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #2
Damn, this is listening test season :)

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)
While this kind of test is rare and indeed very interesting, we have to be careful not to draw any conclusions on Bluetooth codecs used in practical situations (meaning when used with phones, cars, etc.) because the encoders used are most likely different, and may exhibit different qualities/flaws.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #3
I'm surprised by aptX lower efficiency. I always thought this format was introduced to increase quality. I don't know if FFMPEG's implementation is identical to what audio devices are doing but from this test it's clear that this format is more a selling feature than a real progress in audio quality (maybe a better latency?).
Don't forget that aptX is also an ADPCM format, so its main selling point is probably that it's easier to encode and decode.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #4
Damn, this is listening test season :)

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)
While this kind of test is rare and indeed very interesting, we have to be careful not to draw any conclusions on Bluetooth codecs used in practical situations (meaning when used with phones, cars, etc.) because the encoders used are most likely different, and may exhibit different qualities/flaws.

AFAIK, the development of the SBC codec focuses on speed, not the quality. The quality of the SBC is left unchanged. It's safe to assume that devices are shipped with the same-old encoders.
https://www.openhub.net/p/sbc/commits/summary
http://git.videolan.org/?p=ffmpeg.git&a=search&h=HEAD&st=commit&s=sbc

I suspect that aptX cannot change its quality, as it is two layers of QMF + APCM.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #5
As indicated above, the only way to effectively change the quality of aptX is to change its sample rate. Then you may be pushing the resulting quantization noise closer to being audible to you, depending on how high your hearing goes.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #6
Is there any way that aptX implementations could reasonably differ audibly, like if an encoder used a less precise algorithm?

(I'm pretty sure the difference wouldn't be anywhere near as big as an early mp3 encoder and something vaguely competent, though)

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #7
Wondering how close is ffmpeg's SBC encoder to the one used in Android (10). Even if the bitrate is the same, encoder is theoretically allowed to make different decisions about distributing bits between bands, and stuff like that.
some ANC'd headphones + AutoEq-based impulse + Meier Crossfeed (30%)

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #8
The bitrate vs score graph.


Re: Personal Blind Listening Test of Bluetooth codecs

Reply #9
It's so cool to come back here after maybe 10 (correction: 6) years and see several members' names that I actually remember from way back then.  :D

Really just wanted to say thanks for a very interesting listening test. I've been reading up on Bluetooth codecs recently and this was a valuable addition to my knowledge.
davidnaylor.org

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #10
The bitrate vs score graph.



APTX codec not reencode mp3 files, therefore there aren't sound degradation with mp3 files.
I just find APTX LOW LATENCY codec, is able give best quality in common bluetooth codecs, but only with CSR chip - I don't know why but this chip make sound a little bit compressed but more balanced. I talking about playing with F2K wave and flac files.
I use that device - Avantree Aptx Low Latency Bluetooth Transmitter And Receiver 2-In-1, Wireless 3. - https://www.ebay.com/c/938285992
 

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #11
APTX codec not reencode mp3 files, therefore there aren't sound degradation with mp3 files.
Yes it does, and yes there is. The aptX encoder is supposed to use a high enough bitrate that you won't hear it, but it's always there.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #12
As I've said from examining the open source encoder, it's an ADPCM codec. It's supposed to be generally low latency and fairly high quality, you just need to encode at a high enough sample rate. It's constant bitrate, regardless of audio complexity. The only way to reduce the bitrate is to also reduce the frequency response.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #13
I mean, in theory shouldn't any artifacts be along the lines of analog noise?

(But I'm a weirdo for whom SBC sounds "fine")

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #14
For a codec like AptX, yes, they'll sound like analog noise. For SBC or AAC, they'll sound different.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #15
Yes, that's what I meant, sorry.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #16
Wondering how close is ffmpeg's SBC encoder to the one used in Android (10). Even if the bitrate is the same, encoder is theoretically allowed to make different decisions about distributing bits between bands, and stuff like that.

It has been a while since I looked at the standard, but I think this is incorrect.  Given a valid set of CODEC parameters (e.g. nrof_subbands, nrof_blocks, and bitpool), I believe the encoder is deterministic.  Meaning that, if you give two different SBC encoders a specific audio file to encode, and the same set of parameters, you would get the same, bit-perfect output from both of them.

There are two ways to allocate bits to sub-bands, SNR and LOUDNESS, but this parameter is negotiated when the connection is initiated.  The bit allocation of course changes with the input audio file, but there is no reason to think that, if two separate encoders, both adhering to the Standard, with the same encoder parameters, would allocate the bits to sub-bands differently.

Of course, not all encoders adhere to the Standard.  All Bluetooth A2DP audio devices must support SBC.  Further, the Standard states which encoder parameters must be supported by the SBC encoder.  The Standard also provides some example "known good combinations" of these parameters.  Many SBC encoders use (and test)  only these example "known good combinations", despite advertising during connection initiation that that other combinations are supported. If you try to set up a connection with less-frequently-used, but supposedly supported, parameter combinations, real-world SBC encoders often just stop working until it is reset.

It is a shame, as better use of the flexibility built into SBC likely could have avoided all these later, proprietary CODECs.  (SBC is open source.)

If you want to learn more, see my white-paper:
https://drive.google.com/file/d/1J40QF_gZfW7Lo35lc_pgpGi8MZEeWGVt/view?usp=sharing


 
SimplePortal 1.0.0 RC1 © 2008-2021