11
Listening Tests / Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC
Last post by Kamedo2 -Personal blind sound quality comparison of xHE-AAC and Ogg Vorbis at around 134 kbps, and the Fabrice Bellard's new AI-based codec TSAC: Very Low Bitrate Audio Compression at 7.5kbps.
Encoders:
Except exhale v1.1.9, all of them are latest as of 7 May 2024.
xHE-AAC: exhale-V1.1.9-00423757_x64, exhale-v1.2.1_x64
Ogg Vorbis: aoTuV_b6.03_2020.
TSAC: Windows version (experimental): tsac-2024-04-08-win64.zip.
xHE-AAC commandlines:
exhale-V1.1.9-00423757_x64\exhale 5 in.wav out.mp4
exhale-v1.2.1_x64\exhale 5 in.wav out.mp4
ffmpeg109823 -c:a libfdk_aac -i out.mp4 -c:a pcm_f32le out.wav
Ogg Vorbis commandlines:
aoTuV_b6.03_2020\venc64 -q3.7 in.wav out.ogg
oggdecV1.10.1\oggdec -b 5 out.ogg --wavout out.wav
TSAC commandlines:
tsac-2024-04-08-win64\tsac -v -m tsac-2024-04-08-win64\dac_stereo_q8.bin -M tsac-2024-04-08-win64\tsac_stereo_q8.bin -q 12 c in.wav out.tsac
tsac-2024-04-08-win64\tsac -v -m tsac-2024-04-08-win64\dac_stereo_q8.bin -M tsac-2024-04-08-win64\tsac_stereo_q8.bin d out.tsac out.wav
Sample tracks:
15 sound samples from Kamedo2's samples(full download).
Total 15 diverse music sound samples, including highly critical samples.
Hardware:
Sony PSP-3000 + AKG K712.
Results (only traditional codecs, at around 134 kbps):
Results (including AI codec at 7.5kbps, it is not a bitrate-equalized comparison):
Conclusions & Observations:
- MPEG-4 xHE-AAC (eXtended High-Efficiency AAC), encoded by exhale (Ecodis eXtended High-efficiency And
Low-complexity Encoder), had very high fidelity at around 134kbps, with average score over 4.5. - Ogg Vorbis, encoded by aoyumi's aoTuV beta6.03(latest version as of 2024 May), also had very high fidelity at around 134kbps, with average score more than 4.4.
- It's not clear which encoder, xHE-AAC or Ogg Vorbis, was better, from this test alone. The difference was small.
- Both xHE-AAC and Ogg Vorbis at around 134kbps were better than the TSAC: Very Low Bitrate Audio Compression at 7.5kbps, at its maximum bitrate setting as of 2024-04-08 version. TSAC used 94.4% less disk space, and this test not meant to be a filesize-wise fair comparison.
Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis
Number of listeners: 15
Critical significance: 0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings
Source of Degrees Sum of Mean
variation of Freedom squares Square F p
Total 59 86.42
Testers (blocks) 14 1.00
Codecs eval'd 3 83.62 27.87 651.96 0.00E+000
Error 42 1.80 0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA: 0.152
Means:
exh119 exh121 ogg128 tsac12
4.56 4.55 4.45 1.79
---------------------------- p-value Matrix ---------------------------
exh121 ogg128 tsac12
exh119 0.861 0.141 0.000*
exh121 0.193 0.000*
ogg128 0.000*
-----------------------------------------------------------------------
exh119 is better than tsac12
exh121 is better than tsac12
ogg128 is better than tsac12
Raw data:
Code: [Select]
exhale v1.1.9 exhale v1.2.1 aoTuV beta6.03 tsac-2024-04-08-win64
%feature 3 xHE-AAC xHE-AAC Ogg Vorbis TSAC: Very Low Bitrate Audio Compression
%feature 10 test tracks avg/albums avg test tracks avg/albums avg test tracks avg/albums avg test tracks avg/albums avg
%feature 11 138.7kbps/130.4kbps 139.0kbps/129.6kbps 143.2kbps/126.1kbps 7.4kbps/7.5kbps
%feature 12 exhale 5 in.wav out.mp4 exhale 5 in.wav out.mp4 venc64 -q3.7 in.wav out.ogg tsac -v -q 12 c in.wav out.tsac
5.000 4.500 4.400 2.300
4.400 4.500 5.000 1.700
5.000 5.000 4.300 1.900
4.800 4.600 4.400 1.900
4.400 4.600 4.800 1.800
4.500 4.600 4.700 1.500
4.600 4.700 4.300 2.100
4.700 4.500 4.600 1.700
4.500 4.400 4.700 1.700
4.100 4.300 4.400 1.600
4.400 4.500 4.200 1.500
4.700 4.500 4.300 1.800
4.500 4.700 4.300 1.700
4.500 4.400 4.200 1.900
4.300 4.400 4.100 1.800
%samples 41_30sec Perc.
%samples finalfantasy Strings
%samples ATrain Jazz
%samples BigYellow Pops
%samples FloorEssence Techno
%samples macabre Classic
%samples mybloodrusts Guitar
%samples Quizas Latin
%samples VelvetRealm Techno
%samples Amefuribana Pops
%samples Trust Gospel
%samples Waiting Rock
%samples Experiencia Latin
%samples Heart to Heart Pops
%samples Tom's Diner Acappella
Other tests:
- Personal blind comparison of the Bluetooth codec, AAC and SBC and LC3.
- Personal blind comparison of the Bluetooth codecs, AAC vs LC3, re-encoding
- Personal blind sound quality comparison of the Bluetooth codecs (AAC vs LC3)
- Personal blind sound quality comparison of Opus hard-CBR with framesize options
- Personal Blind Listening Test of the latest codecs at 40kbps, 48kHz
- Personal Blind Listening Test of Bluetooth codecs
- Personal Blind Listening Test of Opus and the exhale xHE-AAC encoder
- Personal Listening Test of AAC-LC and xHE-AAC at 96kbps and 128kbps
- Personal Listening Test of AAC and LAME encoders (old test, translated)
- Personal Listening Test of AAC, WMA, and MP3 encoders (old test, translated)
- Personal Listening Test of MP3 and Opus
- Personal Listening Test of AAC encoders available from FFmpeg
- Personal Listening Test of Experimental Modified Opus Encoders at 36, 48 kbps
- Personal Listening Test of 2 Opus encoders
- Personal Listening Test of LAME, iTunes and Helix MP3 encoders
- Personal Listening Test of MP3/Opus/AAC at 96kbps