Skip to main content

Notice

If you are using a Hotmail or Outlook email address, please change it now, as Microsoft is rejecting all email from our service outright.
Topic: Personal Blind Listening Test of Bluetooth codecs (Read 1040 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Blind Listening Test of Bluetooth codecs

Abstract:
Personal blind sound quality comparison of common Bluetooth codecs (SBC, AAC, aptX).

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)

SBC Settings:
FFmpeg's native SBC encoders and decoders were used.
ffmpeg -i in.wav -c:a sbc -b:a 237k out.sbc
ffmpeg -i in.wav -c:a sbc -b:a 280k out.sbc
ffmpeg -i in.wav -c:a sbc -b:a 328k out.sbc (common Bluetooth setting)

AAC Settings:
Fraunhofer FDK AAC Codec Library for Android was used as the AAC-LC encoder.
ffmpeg -i in.wav -c:a libfdk_aac -b:a 128k out.mp4
ffmpeg -i in.wav -c:a libfdk_aac -b:a 192k out.mp4 (common Bluetooth setting)

aptX Settings:
FFmpeg's native aptX encoders and decoders were used. It is NOT aptX LL nor aptX HD.
ffmpeg -i in.wav -c:a aptx -ar 32000 out.aptX
ffmpeg -i in.wav -c:a aptx -ar 44100 out.aptX (common Bluetooth setting at 44.1kHz)

Samples:
15 sound samples from Kamedo2's samples.
12 sound samples from IgorC's samples.
Total 27 sound samples.

Hardwares:
Sony PSP-3000 + RP-HT560.

Results, only the common Bluetooth settings:


All Results:


Tracks list:


Conclusions & Observations:
  • All Bluetooth codecs tested, SBC, AAC, and aptX offer high fidelity on the common settings.
  • AAC offers the fidelity comparable to SBC and aptX, on 41%-45% reduced bitrate.
  • In this test, Fraunhofer FDK AAC Codec Library for Android was used to evaluate the fidelity of the AAC codec. Android users use this for the Bluetooth connection. Apple iPhone users may be able to boost the fidelity even more by using Apple's CoreAudio engine, judging from Guruboolez test, but it cannot be confirmed from this test alone.
  • SBC significantly drops its fidelity, when the rate was adjusted to mere 15% less than the 328kbps setting.
  • aptX is not made to dynamically adjust its bitrate. Using 32kHz sampling rate instead of 44.1kHz severely affects fidelity.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 21
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total              146          52.59
Testers (blocks)    20          10.48
Codecs eval'd        6          22.71    3.79   23.42  0.00E+000
Error              120          19.40    0.16
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.246

Means:

fdk192   sbc328   apt352   sbc280   fdk128   apt256   sbc237  
  4.68     4.60     4.47     4.15     4.08     3.82     3.52  

---------------------------- p-value Matrix ---------------------------

         sbc328   apt352   sbc280   fdk128   apt256   sbc237  
fdk192   0.515    0.087    0.000*   0.000*   0.000*   0.000*  
sbc328            0.285    0.000*   0.000*   0.000*   0.000*  
apt352                     0.013*   0.002*   0.000*   0.000*  
sbc280                              0.540    0.008*   0.000*  
fdk128                                       0.040*   0.000*  
apt256                                                0.019*  
-----------------------------------------------------------------------

fdk192 is better than sbc280, fdk128, apt256, sbc237
sbc328 is better than sbc280, fdk128, apt256, sbc237
apt352 is better than sbc280, fdk128, apt256, sbc237
sbc280 is better than apt256, sbc237
fdk128 is better than apt256, sbc237
apt256 is better than sbc237


Raw data:
Code: [Select]
sbc237k	sbc280k	sbc328k	aac128k	aac192k	aptx256k	aptx352k
%feature 7 SBC SBC SBC AAC AAC aptX aptX
%feature 10 ffmpeg N-99419-g4fceb2634e ffmpeg N-99419-g4fceb2634e ffmpeg N-99419-g4fceb2634e FDK-AAC v2.0.1 FDK-AAC v2.0.1 ffmpeg N-99419-g4fceb2634e ffmpeg N-99419-g4fceb2634e
%feature 11 -c:a sbc -c:a sbc -c:a sbc -c:a libfdk_aac -c:a libfdk_aac -c:a aptx -c:a aptx
%feature 12 237kbps 281kbps 331kbps 130kbps 194kbps 256kbps 353kbps
3.800 4.700 5.000 4.300 4.700 3.900 4.600
3.100 3.800 4.600 4.600 5.000 3.600 4.400
3.900 4.700 5.000 4.100 5.000 4.400 5.000
3.600 5.000 5.000 4.300 5.000 4.500 5.000
4.300 4.400 5.000 3.600 4.500 3.400 3.800
3.200 3.900 4.500 4.100 4.600 4.200 4.400
3.900 4.500 4.400 5.000 5.000 4.100 4.700
3.200 4.100 4.600 4.400 5.000 5.000 5.000
3.000 3.100 4.800 3.300 3.800 4.300 4.600
2.900 3.300 4.300 4.100 4.700 3.500 4.600
4.200 4.600 4.700 3.600 4.300 3.700 4.800
2.900 3.900 4.500 3.500 4.400 3.700 4.200
3.700 4.500 4.700 4.600 5.000 3.400 4.200
3.100 3.800 4.300 3.900 5.000 3.400 4.200
2.900 3.500 3.800 4.100 5.000 3.300 4.300
4.100 4.400 5.000 3.700 4.500 3.900 4.800
4.700 5.000 5.000 3.700 4.400 3.800 5.000
4.300 4.800 4.700 3.800 4.200 3.200 4.500
3.200 3.800 4.400 4.100 4.700 4.200 4.500
3.100 4.200 4.400 4.500 5.000 3.300 3.500
2.900 3.200 3.900 4.300 4.500 3.400 3.700
3.700 4.400 4.300 3.200 4.100 3.800 4.200
3.200 3.900 4.200 3.800 4.400 3.600 5.000
3.800 5.000 4.700 3.400 3.900 4.300 4.500
3.600 4.600 4.700 4.200 5.000 3.800 4.500
3.800 4.400 5.000 5.000 5.000 5.000 5.000
3.800 4.400 4.400 4.500 5.000 4.200 4.600
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella
%samples 01 castanets inst.
%samples 02 fatboy_30sec Techno
%samples 03 eig Techno
%samples 04 Bachpsichord inst.
%samples 05 Enola Techno
%samples 06 trumpet inst.
%samples 07 applaud Live
%samples 08 velvet perc.
%samples 09 Linchpin Rock
%samples 10 spill_the_blood guitar
%samples 11 female_speech Speech
%samples 12 French_Ad Speech


Bitrates:
Code: [Select]
%bitrate
237036 281135 330747 130016 194031 256000 352800
237006 281100 330706 130850 195057 256000 352800
237030 281128 330739 131475 195827 256001 352800
236969 281057 330655 131082 195248 256000 352800
237027 281125 330735 130561 194707 256000 352800
237024 281121 330731 129960 194015 256000 352800
236965 281051 330648 131447 195921 255999 352800
237028 281126 330737 130083 194180 256000 352800
237016 281112 330720 130659 194779 256000 352800
237028 281126 330736 129955 194043 256000 352800
237016 281112 330720 129951 194008 256000 352800
237015 281110 330718 130240 194347 256000 352800
237028 281126 330736 129990 194050 256000 352800
236995 281087 330690 130563 194840 256001 352800
237008 281102 330708 130306 194432 256000 352800
237011 281106 330713 130050 194111 256000 352800
237032 281131 330743 130262 194347 256000 352800
237026 281124 330735 130223 194454 256000 352800
237008 281103 330709 130183 194246 256000 352800
237028 281126 330737 129975 194041 256000 352800
236973 281061 330660 131100 195263 256000 352800
237037 281136 330749 130049 194117 256000 352800
237015 281111 330719 129868 193955 256000 352800
237037 281137 330750 130167 194256 256000 352800
237005 281099 330705 130234 194273 256000 352800
237020 281117 330725 130062 194140 256000 352800
237009 281104 330711 130241 194337 256000 352800

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #1
Captivating! I was waiting for such comparison for years :) I didn't know that SBC and aptX were available in FFMPEG. Great test, thank you very much for this huge effort!

I'm surprised by aptX lower efficiency. I always thought this format was introduced to increase quality. I don't know if FFMPEG's implementation is identical to what audio devices are doing but from this test it's clear that this format is more a selling feature than a real progress in audio quality (maybe a better latency?).

AAC is in a different league though. The most important benefit in my opinion is the perceptual quality at 128 kbps which remains high. Funny though: FDK-AAC-128 ends here at 4.06—and in my own test it ends at 4.04  :)

All formats at full-bitrate are excellent though. SBC subband format seems to handle very well preecho (castanets, eig, applauds…) but is less at ease with tonal samples (trumpet, enola).

Well done!

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #2
Damn, this is listening test season :)

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)
While this kind of test is rare and indeed very interesting, we have to be careful not to draw any conclusions on Bluetooth codecs used in practical situations (meaning when used with phones, cars, etc.) because the encoders used are most likely different, and may exhibit different qualities/flaws.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #3
I'm surprised by aptX lower efficiency. I always thought this format was introduced to increase quality. I don't know if FFMPEG's implementation is identical to what audio devices are doing but from this test it's clear that this format is more a selling feature than a real progress in audio quality (maybe a better latency?).
Don't forget that aptX is also an ADPCM format, so its main selling point is probably that it's easier to encode and decode.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #4
Damn, this is listening test season :)

Encoders:
The latest versions as of 2020 October were used.

ffmpeg version N-99419-g4fceb2634e Copyright (c) 2000-2020 the FFmpeg developers built with gcc 9.2.0 (Rev2, Built by MSYS2 project)
FDK-AAC 2.0.1 (-c:a libfdk_aac)
While this kind of test is rare and indeed very interesting, we have to be careful not to draw any conclusions on Bluetooth codecs used in practical situations (meaning when used with phones, cars, etc.) because the encoders used are most likely different, and may exhibit different qualities/flaws.

AFAIK, the development of the SBC codec focuses on speed, not the quality. The quality of the SBC is left unchanged. It's safe to assume that devices are shipped with the same-old encoders.
https://www.openhub.net/p/sbc/commits/summary
http://git.videolan.org/?p=ffmpeg.git&a=search&h=HEAD&st=commit&s=sbc

I suspect that aptX cannot change its quality, as it is two layers of QMF + APCM.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #5
As indicated above, the only way to effectively change the quality of aptX is to change its sample rate. Then you may be pushing the resulting quantization noise closer to being audible to you, depending on how high your hearing goes.

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #6
Is there any way that aptX implementations could reasonably differ audibly, like if an encoder used a less precise algorithm?

(I'm pretty sure the difference wouldn't be anywhere near as big as an early mp3 encoder and something vaguely competent, though)

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #7
Wondering how close is ffmpeg's SBC encoder to the one used in Android (10). Even if the bitrate is the same, encoder is theoretically allowed to make different decisions about distributing bits between bands, and stuff like that.
some ANC'd headphones + AutoEq-based impulse + Meier Crossfeed (30%)

Re: Personal Blind Listening Test of Bluetooth codecs

Reply #8
The bitrate vs score graph.


 
SimplePortal 1.0.0 RC1 © 2008-2020