Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC (Read 490 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Abstract:
Personal blind sound quality comparison of xHE-AAC and Ogg Vorbis at around 134 kbps, and the Fabrice Bellard's new AI-based codec TSAC: Very Low Bitrate Audio Compression at 7.5kbps.

Encoders:
Except exhale v1.1.9, all of them are latest as of 7 May 2024.
xHE-AAC: exhale-V1.1.9-00423757_x64, exhale-v1.2.1_x64
Ogg Vorbis: aoTuV_b6.03_2020.
TSAC: Windows version (experimental): tsac-2024-04-08-win64.zip.

xHE-AAC commandlines:
exhale-V1.1.9-00423757_x64\exhale 5 in.wav out.mp4
exhale-v1.2.1_x64\exhale 5 in.wav out.mp4
ffmpeg109823 -c:a libfdk_aac -i out.mp4 -c:a pcm_f32le out.wav

Ogg Vorbis commandlines:
aoTuV_b6.03_2020\venc64 -q3.7 in.wav out.ogg
oggdecV1.10.1\oggdec -b 5 out.ogg --wavout out.wav

TSAC commandlines:
tsac-2024-04-08-win64\tsac -v -m tsac-2024-04-08-win64\dac_stereo_q8.bin -M tsac-2024-04-08-win64\tsac_stereo_q8.bin -q 12 c in.wav out.tsac
tsac-2024-04-08-win64\tsac -v -m tsac-2024-04-08-win64\dac_stereo_q8.bin -M tsac-2024-04-08-win64\tsac_stereo_q8.bin d out.tsac out.wav

Sample tracks:
15 sound samples from Kamedo2's samples(full download).

Total 15 diverse music sound samples, including highly critical samples.

Hardware:
Sony PSP-3000 + AKG K712.

Results (only traditional codecs, at around 134 kbps):


Results (including AI codec at 7.5kbps, it is not a bitrate-equalized comparison):



Conclusions & Observations:
  • MPEG-4 xHE-AAC (eXtended High-Efficiency AAC), encoded by exhale (Ecodis eXtended High-efficiency And
    Low-complexity Encoder), had very high fidelity at around 134kbps, with average score over 4.5.
  • Ogg Vorbis, encoded by aoyumi's aoTuV beta6.03(latest version as of 2024 May), also had very high fidelity at around 134kbps, with average score more than 4.4.
  • It's not clear which encoder, xHE-AAC or Ogg Vorbis, was better, from this test alone. The difference was small.
  • Both xHE-AAC and Ogg Vorbis at around 134kbps were better than the TSAC: Very Low Bitrate Audio Compression at 7.5kbps, at its maximum bitrate setting as of 2024-04-08 version. TSAC used 94.4% less disk space, and this test not meant to be a filesize-wise fair comparison.

Anova analysis:
Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 15
Critical significance:  0.05
Significance of data: 0.00E+000 (highly significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               59          86.42
Testers (blocks)    14           1.00
Codecs eval'd        3          83.62   27.87   651.96  0.00E+000
Error               42           1.80    0.04
---------------------------------------------------------------
Fisher's protected LSD for ANOVA:   0.152

Means:

exh119   exh121   ogg128   tsac12   
  4.56     4.55     4.45     1.79   

---------------------------- p-value Matrix ---------------------------

         exh121   ogg128   tsac12   
exh119   0.861    0.141    0.000*   
exh121            0.193    0.000*   
ogg128                     0.000*   
-----------------------------------------------------------------------

exh119 is better than tsac12
exh121 is better than tsac12
ogg128 is better than tsac12

Raw data:
Code: [Select]
exhale v1.1.9	exhale v1.2.1	aoTuV beta6.03	tsac-2024-04-08-win64
%feature	3	xHE-AAC	xHE-AAC	Ogg Vorbis	TSAC: Very Low Bitrate Audio Compression
%feature	10	test tracks avg/albums avg	test tracks avg/albums avg 	test tracks avg/albums avg	test tracks avg/albums avg 
%feature	11	138.7kbps/130.4kbps	139.0kbps/129.6kbps	143.2kbps/126.1kbps	7.4kbps/7.5kbps
%feature	12	exhale 5 in.wav out.mp4	exhale 5 in.wav out.mp4	venc64 -q3.7 in.wav out.ogg	tsac -v -q 12 c in.wav out.tsac
5.000	4.500	4.400	2.300	
4.400	4.500	5.000	1.700	
5.000	5.000	4.300	1.900	
4.800	4.600	4.400	1.900	
4.400	4.600	4.800	1.800	
4.500	4.600	4.700	1.500	
4.600	4.700	4.300	2.100	
4.700	4.500	4.600	1.700	
4.500	4.400	4.700	1.700	
4.100	4.300	4.400	1.600	
4.400	4.500	4.200	1.500	
4.700	4.500	4.300	1.800	
4.500	4.700	4.300	1.700	
4.500	4.400	4.200	1.900	
4.300	4.400	4.100	1.800	
%samples	41_30sec	Perc.
%samples	finalfantasy	Strings
%samples	ATrain	Jazz
%samples	BigYellow	Pops
%samples	FloorEssence	Techno
%samples	macabre	Classic
%samples	mybloodrusts	Guitar
%samples	Quizas	Latin
%samples	VelvetRealm	Techno
%samples	Amefuribana	Pops
%samples	Trust	Gospel
%samples	Waiting	Rock
%samples	Experiencia	Latin
%samples	Heart to Heart	Pops
%samples	Tom's Diner	Acappella

Other tests:

Re: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Reply #1 –
ahaha, wow, TSAC-encoded files look painful to listen to!

Re: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Reply #2 –
Quote from: Lampion on
ahaha, wow, TSAC-encoded files look painful to listen to!

Please note that TSAC used only 1/18th of the disk space of these competitors.

Re: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Reply #3 –
Thanks a lot, Kamedo2, for your meticulous listening! I know how much work blind listening tests at these bitrates are. Some distilled observations and connections with previous discussions on HA:

  • In your earlier (09/2020) tests (https://hydrogenaud.io/index.php/topic,119861.0.html), exhale 1.0.6 reached a mean score of "only" 4.47 on the same 15 samples. It's nice to have some evidence now that exhale's higher-rate audio quality improved some more during the last 3-4 years.
  • Regarding my score estimator (https://hydrogenaud.io/index.php/topic,118888.msg989150.html#msg989150), your average results from yesterday are now above my estimate, similar to IgorC' 2020 results at 192 kbps. Maybe I can finally update my score estimator at the higher rates :)
  • AFAIR, this is the first blind test in which a machine learned end-to-end codec is compared against high-performance classical audio codecs. This is very valuable information since it puts all these "AI codecs outperform MP3" hyped claims into perspective - machine learned codecs deliver decent audio quality, but they don't scale to transparency... at least not in 2024.

Chris
If I don't reply to your reply, it means I agree with you.

Re: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Reply #4 –
Quote from: C.R.Helmrich on
but they don't scale to transparency... at least not in 2024.

They don't target transparency, which is why I think it would be more useful if:

* Samples used in tests are not handpicked for being challenging, but picked for being representative of average everyday use (talk show, pop music, ..etc).

* Classical codecs are tested at a much lower bitrate than in this test. Maybe a something in the range of 32-48kbps.

Re: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Reply #5 –
Thanks a lot Kamedo for yet another very interesting listening test.

Nice to see Vorbis being tested and yielding quite impressive results in the ~128kps range. Is there any particular reason why you chose Vorbis in this test?
 

Re: Personal blind sound quality comparison of xHE-AAC, Ogg Vorbis, and TSAC

Reply #6 –
Quote from: Julien on
Nice to see Vorbis being tested and yielding quite impressive results in the ~128kps range. Is there any particular reason why you chose Vorbis in this test?

Because it hasn't been tested for some years.