HydrogenAudio

TSAC: Very Low Bitrate Audio Compression available at : https://bellard.org/tsac/ . It achieves about 5.5 kb/s for mono or 7.5 kb/s for stereo at 44.1 kHz with a good perceptual quality. It is based on a modified version of the Descript Audio Codec and a Transformer model to further increase the compression ratio.

TSAC (2024-04-08) displays help when running without flags on my end and crashes when encoding.

Command: tsac.exe -v c tolle.wav tolle.tsac
Source: PCM 44.1 kHz, 16-bit, 2 channels
Setup: Core2Duo with SSE3, 4 GB RAM, Windows 7 x64

(https://i2.imageban.ru/out/2024/04/08/ba9ec81afdb8d952ede6be73c40bbb38.png)

Sorry it is not in the readme file : a CPU with AVX2 support is required to use the program.

@fab7, is AVX critical for TSAC? Perhaps you could recompile it to work with a wider range of hardware? Or at least could you explain why a very low bitrate audio compression tool, whose output is designed to use as few resources as possible for storage and transmission, requires modern hardware for encoding? E.g. Google Lyra (https://hydrogenaud.io/index.php/topic,120810.msg1040953.html#msg1040953) works on my end.

@fab7, is AVX critical for TSAC? Perhaps you could recompile it to work with a wider range of hardware? Or at least could you explain why a very low bitrate audio compression tool, whose output is designed to use as few resources as possible for storage and transmission, requires modern hardware for encoding? E.g. Google Lyra (https://hydrogenaud.io/index.php/topic,120810.msg1040953.html#msg1040953) works on my end.

TSAC needs a lot of compute power. That's why a GPU is recommended. Using an AVX2 capable CPU is possible but quite slow. I don't think it is worth supporting older CPUs because it would be too slow to be usable.

I am impressed by the sound sample on the web site! Imagine the small size of podcasts and audio books using the ~2kbbps!

What kind of CPU speeds are needed to decode in realtime?

Assuming the samples posted on the website are genuine, and I'm not suggesting that they are not, the quality at ridiculously low bitrates is quite extraordinary. This should be followed with interest. There comes a point with developments of this nature where supporting older hardware becomes irrelevant and is not the target audience. One cannot, and should not, support decades old hardware and OSs for ever.

Using the Linux build on my Intel(R) Core(TM) i3-7100U CPU @ 2.40GHz the encoding speed using CPU mode was about 0.5x. Decoding was almost the same. The audio I encoded ended up being 8.35 kbps and the sound quality was impressive for that bitrate. The sound quality of distorted guitars sound like too much noise reduction has been used and ambience/reverbs are attenuated. When I listen to my low quality speakers in my laptop I can gladly listen to the TSAC encoded files.

The encoder only used on logical core even though I passed -T 2. That logical core used 100% CPU while the three others were idling.

The music in the attached files is Lagwagon - Beer Goggles.

No source code?

I tried, and it worked wonder.

encoding:

./tsac -v -q 12 --separate_channels -m dac_stereo_q8.bin -M tsac_stereo_q8.bin c ../../music300sec.wav ../../music300sec.q12.separate.tsac

decoding:

./tsac -v -m dac_stereo_q8.bin -M tsac_stereo_q8.bin d ../../music300sec.q12.separate.tsac ../../music300sec.decoded.wav

However, typical use cases should be more concerned about fidelity than the bit rate. Can't I increase the quality even more than the -q 12 --separate_channels?
I would like to perform a private double-blind test of TSAC against USAC, exhale-v1.2.1 and exhale-v1.1.9.

However, typical use cases should be more concerned about fidelity than the bit rate. Can't I increase the quality even more than the -q 12 --separate_channels?

I would like to perform a private double-blind test of TSAC against USAC, exhale-v1.2.1 and exhale-v1.1.9.

These codecs target discernibility, not fidelity. And their biggest problem is potential hallucinations. So, what testers should be looking out for most, IMHO, are issues like completely the wrong letter being pronounced, not the general quality of the output.

One can think of these codecs as good lip readers. Fun, cool, and maybe impressive, but not something one would/should use with trust in practice.

As for exhale. Per the project's own README, the encoder shouldn't be used for very low-bitrate use cases, or considered representative of USAC's full potential at such rates.

These codecs target discernibility, not fidelity. And their biggest problem is potential hallucinations. So, what testers should be looking out for most, IMHO, are issues like completely the wrong letter being pronounced, not the general quality of the output.

I haven't heard a single case like that in my several minutes of listening tsac, but I'm going to be careful whether anything like that is present in the decoded sounds.

-q value(number of codebooks, which sets the quality and bit rate) vs Bitrate table, 44.1kHz music albums, for those wondering.

The commandline is according to the -h output. Spoiler (click to show/hide)

It's a pity to realize that TSAC needs a lot of computing power. Why? Because there are still plenty of places on Earth where people either can barely afford access to electricity (e.g. in Madagascar (https://www.downtoearth.org.in/news/africa/africa-s-energy-poverty-push-to-renewables-more-players-for-power-supply-may-help-madagascar-s-electrification-plans-90503), 600 000 households out of 5.8 million) or it becomes expensive and requires to tighten the belts (e.g. in Europe (https://www.imf.org/en/Publications/fandd/issues/2022/12/beating-the-european-energy-crisis-Zettelmeyer), prices of electricity and gas have surged as much as 15-fold since 2021). As you know, power-hungry operations drain the battery of portable devices much faster, requiring frequent recharging, thereby shortening the lifespan and forcing to break the piggy bank again. And what for? For an audio-book to take up a little less space than, say, 22 kHz 32 kbit/s or 24 kHz 40 kbits/s fast and battery-friendly MP3 (encoded not by LAME, but by the resurrected Helix (https://hydrogenaud.io/index.php/topic,123331.msg1042331.html#msg1042331))? A dubious trade-off. Thus, TSAC, at least until its intended scope is officially announced, appears to be an academic study on sound compression using a GPU, which at first may stun with the grandeur, but in practical terms there is a sore temptation to take it as another exhibition of exotic animals (https://hydrogenaud.io/index.php/topic,125248.msg1042285.html#msg1042285).

The need to hear is so dire on our planet that the compression that makes it difficult to access the myriads of stories conveyed via sound vibrations is doomed either to niche status or to oblivion. A few decades ago in the US, people of color were forced to ride in a different part of the bus (https://en.wikipedia.org/wiki/Montgomery_bus_boycott), and today it is not uncommon to meet a programmer (not you, dear @fab7) who is one step away from forcing users to be content with the sound of rain and wind if they are not in a hurry to buy the latest Silicon Valley stuff. What is it if not the resurgence of segregation, albeit in the digital domain? Think of a Cuban barista, a Palestinian builder, and a Tibetan teacher, not to mention the frugal residents of former Western colonies and millions of families who lost their savings due to tragedies such as earthquakes and floods, bank failures and wars — they are lucky to get second-hand vintage hardware, rent it or receive as a gift. Can we treat them as brothers and consider their cases when developing apps solutions? Or will our vision remain clouded by an intellectual feast divorced from empathy, with arguments about the elusive difference between 44.1 and 192 kHz (https://hydrogenaud.io/index.php/topic,125718.0.html)?

One word - amazing.
How on Earth did you achieve this? Amazing quality at such low bitrate.
It took around ~12-13 seconds with RTX 4070 to encode 4 minute song.
I also included 30 second sample if somebody is interested.

My GPU is too old and weak (GTX950 2GB) so looks like it is unsupported, but in terms of CPU speed (i3-12100) it is not really slow when compared to something like flac -8pe (of course, flac decoding is much faster), especially when considering tsac is based on bleeding edge tech.

I also suspected that it may spew out wrong lyrics and such, so I encoded a song with a less popular language but I found no issue.

One issue I found is that the decoded version is somewhat louder than the input and even with some clipped samples, can it be fixed?

[edit]Also attached the encoded tsac file so that @fooball can have some idea about file size.

I also included 30 second sample if somebody is interested.

Please confirm whether I understand this correctly: the "[FLAC]" file is a FLAC encode of the sound clip, and the "[TSAC]" file is a FLAC encode of the same sound clip after it's been TSAC encoded and decoded?

In which case, how big was the TSAC file?

It would also be interesting to hear a (eg) AAC version of the same clip encoded into the same size file.

On Windows 10 Professional, when --cuda is enabled, error message

Could not load: cublasLt64_12.dll (error=126)

was fixed by installing CUDA Toolkit 12.0, Windows, x86_64, Version 10, exe(local), install it, and copy dll from NVIDIA GPU Computing Toolkit\CUDA\v12.0\bin\cublasLt64_12.dll to the same folder as tsac.exe. The dll is 465,925,120 bytes so be careful.
https://developer.nvidia.com/cuda-12-0-0-download-archive?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exe_local

-q value(number of codebooks, which sets the quality and bit rate) vs Bitrate vs Encoding speed table, 44.1kHz music albums.
TSAC is capable of encoding music at about 17 times faster than the realtime in GPU GeForce RTX 3060.

@Kamedo2

How is the decoding speed when CUDA is utilized. Software decoding is actually slower than encoding on my workstation.

----

I gave it a quick test yesterday (speech tests only). The quality is very impressive. No hallucinations noticed either. There was one case where not the clearest N sound (like the one here (https://translate.googleapis.com/translate_tts?ie=UTF-8&client=gtx&tl=ar&q=%D9%86%D9%8E%D8%B2%D9%92%D9%84%D9%8E%D8%A9%D9%8B%20%D8%A3%D9%8F%D8%AE%D9%92%D8%B1%D9%8E%D9%89)) sounded closer to an L. But that can be attributed to bit starvation, and the subtle issue goes away at higher qualities.

Comparing lower bitrate encodes against LPCNet, and higher bitrate ones against Opus with new ML decoding, and the difference is night and day. But LPCNet and Opus are bounded by practicality and format compatibility, so the comparison is not really fair!

Hmmm, what about Turing support? OpenCL?

It's a pity to realize that TSAC needs a lot of computing power. Why?

I'll take the exceptionally obvious bait, because why not:

Sometimes its nice to program something just to see if something is possible, without any regard to how its supposedly meant to be done by others. Sometimes coding can be a creative exercise to solve a particular problem with completely random out of the box solutions. Such as requiring AVX2 or even GPGPU methods. Or sometimes programming something without actual needed care for resources because it works for yourself fine is enough of a reason. Considering who wrote TSAC, no doubt it was one such exercise, like thier other mini projects.

Case in point: one of my personal things which requires GL4.6/GLES2, Windows 10/modern Linux, C++20, SSE4/NEON etc. I do it because its intended for me only and my stuff runs it just fine. Even a old Linux box and RPI4.

[edit]Also attached the encoded tsac file so that @fooball can have some idea about file size.

Thanks... but I had no idea my question had been answered (until I happened to chance on it now).  Updating something up-thread doesn't get flagged up.

100x compression is incredible!

TSAC looks kinda cool and all but it seems can only accept mono and stereo files.
here's hoping multi-channel coding is a possibility with this one!

TSAC @ 8 kbps is comparable to Opus @ 24 kbps, maybe a bit more too. Great, I'll give you that. But that's it. TSAC has the same limitation as all other AI-based audiocodecs. It doesn't scale up with higher bitrate.  Nothing remotely close even to the modest HE-AAC@48 kbps, leave alone any higher bitrates/transparency levels.

TSAC @ 8 kbps is comparable to Opus @ 24 kbps, maybe a bit more too. Great, I'll give you that. But that's it. TSAC has the same limitation as all other AI-based audiocodecs. It doesn't scale up with higher bitrate.  Nothing remotely close even to the modest HE-AAC@48 kbps, leave alone any higher bitrates/transparency levels.

Comparable how? I'd say TSAC @<any bitrate> is not comparable to <non-AI codec>@<any bitrate>. They are just too different for one to make such comparisons.

TSAC, especially at high bitrates, is very impressive actually. Better than any other codec I came across. Too bad it's not usable in practice, or even open-source (allowing experimentation, alternative models, ..etc).

Comparable how? I'd say TSAC @<any bitrate> is not comparable to <non-AI codec>@<any bitrate>. They are just too different for one to make such comparisons.

Maybe by some double-blind listening tests? I am conducting the listening test of the TSAC, and comparison is possible.
TSAC at -q 12 sounds like a traditional codec at low bitrates, especially like Ogg Vorbis, with its sandpaper-like noise.

@Kamedo2

How is the decoding speed when CUDA is utilized. Software decoding is actually slower than encoding on my workstation.

OK, I will test the decoding speed on RTX 3060.
Yes, CPU decoding is slower than the CPU encoding in my AMD Ryzen 7 5700X, too.

Comparable how? I'd say TSAC @<any bitrate> is not comparable to <non-AI codec>@<any bitrate>. They are just too different for one to make such comparisons.

AI (LACE, NoLACE)  vs non-AI (Opus)
https://opus-codec.org/demo/opus-1.5/nolace_results.png
https://opus-codec.org/demo/opus-1.5/

AI (descript-audio-codec: TSAC based on it) vs non-AI (Opus)
https://github.com/descriptinc/descript-audio-codec/blob/main/assets/objective_comparisons.png

It's possible to compare any audio codec to another one, regardless of technology.

Interesting, but there isn't any "heavy" music (which is the most difficult to shrink) encode sample to evaluate...

Interesting, but there isn't any "heavy" music (which is the most difficult to shrink) encode sample to evaluate...

Well, download it and try for yourself. I've tried it with "Gimme hope Joanna" because reasons :)

AI (descript-audio-codec: TSAC based on it) vs non-AI (Opus)
https://github.com/descriptinc/descript-audio-codec/blob/main/assets/objective_comparisons.png

It's possible to compare any audio codec to another one, regardless of technology.

In your first comment, by comparable, did you mean objectively or subjectively?

@2012
I've indicated both examples of  subjective and objective studies from developers.
You just cropped and quoted my last message to only objective study for some particular reason.

Also It's trivial to provide ABC/HR or ABX logs for TSAC vs "any other format".  And Kamedo2 is already preparing the test  :)

@btc
I was actually asking, not arguing.

Well, download it and try for yourself. I've tried it with "Gimme hope Joanna" because reasons :)

Sorry, I don't have any discrete GPU in my system.

Maybe I'll try when an OpenVINO (https://github.com/openvinotoolkit/openvino)-optimized version will be released.

EDIT
For those who are interested in ML-based audio codecs, I've collected some open source resources/codes in AUDIO category (https://github.com/forart/HyMPS#-) \ AI-based page (https://github.com/forart/HyMPS/blob/main/Audio/AI-based.md#--) \ Codecs subsection (https://github.com/forart/HyMPS/blob/main/Audio/AI-based.md#codecs-). Enjoy.

Sorry, I don't have any discrete GPU in my system.

Neither do I. So I waited a few minutes to encode and decode.

Also, a question: Maybe "TSAC" is named fully as "Transformer Audio Codec" because it uses a transformer model to shrink even more while keeping the quality high?

I compressed a 16-second clip of a song: "Vluarr - They Don't Know" to TSAC @ 3 kbps (setting: -q 5, file size: 6.1 KB) using a Intel i5-4460 CPU (was a pain on compression speed because I don't have a Nvidia GPU, but it worked OK).

Full command line:

E:\tsac-2024-04-08-win64>tsac -m E:\tsac-2024-04-08-win64\dac_stereo_q8.bin -M E:\tsac-2024-04-08-win64\tsac_stereo_q8.bin -q 5 c "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\y2mate.com - Vluarr  They Dont Know_44k_s_flac_16sec.wav" "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\output_q5.tsac"
E:\tsac-2024-04-08-win64>tsac -m E:\tsac-2024-04-08-win64\dac_stereo_q8.bin -M E:\tsac-2024-04-08-win64\tsac_stereo_q8.bin d "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\output_q5.tsac" "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\output_q5.wav"

Please check input and output (compressed as FLAC, because it fits within the 30 MB of files per post limit for the forum). ABX them, and listen them carefully. Post the results when done!

Also attached: TSAC version of the output.

I am conducting the listening test of the TSAC, and comparison is possible.

Cool.

It's probably tool late, but I'd be interested in how the attached sample would fare compared to others.
I deleted my TSAC encodes, but I remember the whispers being especially affected/overshadowed.
And a part of the instrumental play got too messy at lower qualities.

Please check input and output (compressed as FLAC, because it fits within the 30 MB of files per post limit for the forum). ABX them, and listen them carefully. Post the results when done!

foo_abx 2.2.1 report
foobar2000 v2.2 preview 2024-05-01
2024-05-06 17:12:22

File A: input.flac
SHA1: bd824f975ffb8eaa36eedef18a8fba79dc40eb39
File B: output_q5.flac
SHA1: 5b197642f52a03d74215a21597c93cc826a93b37

Output:
XAudio2 : Default Sound Device
Crossfading: NO

17:12:22 : Test started.
17:15:03 : Test restarted.
17:15:03 : 01/01
17:15:08 : Test restarted.
17:15:08 : 02/02
17:15:15 : Test restarted.
17:15:15 : 03/03
17:15:22 : Test restarted.
17:15:22 : 04/04
17:15:28 : Test restarted.
17:15:28 : 05/05
17:15:33 : Test restarted.
17:15:33 : 06/06
17:15:38 : Test restarted.
17:15:38 : 07/07
17:15:43 : Test restarted.
17:15:43 : 08/08
17:15:43 : Test finished.

 ---------- 
Total: 8/8
p-value: 0.0039 (0.39%)

 -- signature -- 
22abf4db7b1e2113f428dde825de5fe41b6c6325

During the first second the input sounds more metallic. The codec simplified the output. But I don't think this kind of thing is even meant to be transparent. Just a tech demo to squish things slowly into tiny space, modern day VQF.

Also I attached a speech sample compressed as TSAC @ 3 kbps (setting: -q 4, file size: 1.3 KB) using a Intel i5-4460 CPU (was a pain on compression speed because I don't have a Nvidia GPU, but it worked OK).

Full command line:

E:\tsac-2024-04-08-win64>tsac -m E:\tsac-2024-04-08-win64\dac_mono_q8.bin -M E:\tsac-2024-04-08-win64\tsac_mono_q8.bin -q 4 c "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\New folder (3)\New folder (2)\speech_female_input.wav" "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\New folder (3)\New folder (2)\speech_female_input.tsac"
E:\tsac-2024-04-08-win64>tsac -m E:\tsac-2024-04-08-win64\dac_mono_q8.bin -M E:\tsac-2024-04-08-win64\tsac_mono_q8.bin d "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\New folder (3)\New folder (2)\speech_female_input.tsac" "C:\Users\Public\New folder (14)\New folder (2)\New folder (2)\New folder (3)\New folder (2)\speech_female_output.wav"

Made with Elevenlabs (Glinda voice).

What I feed in TTS: "Is anyone compressing music and speech in such tiny space?"

Please check input and output (compressed as FLAC, because it fits within the 30 MB of files per post limit for the forum). ABX them, and listen them carefully. Post the results when done!

Also attached: TSAC version of the output.

Sorry, I don't have any discrete GPU in my system.

And me too, I don't have a Nvidia GPU! So I waited minutes and minutes to encode and decode audio.

@fab7 ...what about a Gcolab-notebook to perform compression tests ?

EDIT:
What about a NN-based audio codecs blind comparison ?
I've collected some here (https://github.com/forart/HyMPS/blob/main/Audio/AI-based.md#codecs-).

Just seen the thread over Mastodon (https://meow.social/@mimir/112238998609778334) about an interesting sound this codec made if the input file to decode is corrupted (but manages to decode it anyway)

@fab7 ...what about a Gcolab-notebook to perform compression tests ?

EDIT:
What about a NN-based audio codecs blind comparison ?
I've collected some here (https://github.com/forart/HyMPS/blob/main/Audio/AI-based.md#codecs-).

Just wanted to say thank you for compiling this.

I'm really excited about the potential of NN based audio encoders. I think concerns about power consumption will be less of a problem as upcoming CPUs and SoCs will have neural acceleration built in (if they haven't already).

My guess is that over time there will be a number of standard weights (hopefully static and reproducible) that are bundled with various encoders, perhaps as part of the operating system itself or in the formats specification.

I had an idea, but I'm unsure if it's a viable solution to improve things further. What if we use a lossy codec with a proven track record to compress audio files, then compare the result with the original and extract the differences? We could then encode these differences using a codec like this to achieve a sort of lossless quality while reducing the required resource requirements and compression time associated with computationally heavy codecs like this one. I'm curious to know if this hybrid approach could lead to a versatile solution that offers the best of both worlds.

I had an idea, but I'm unsure if it's a viable solution to improve things further. What if we use a lossy codec with a proven track record to compress audio files, then compare the result with the original and extract the differences? We could then encode these differences using a codec like this to achieve a sort of lossless quality while reducing the required resource requirements and compression time associated with computationally heavy codecs like this one. I'm curious to know if this hybrid approach could lead to a versatile solution that offers the best of both worlds.

I don't think this would really benefit as ultimately you are creating a lossy file so the differencing / difference encoding pass doesn't make sense. I suppose stacking different technologies is a curious one, reminds me of SBR or switching between CELT/SILK in Opus depending on content.

Perhaps blending codecs would be an interesting idea - you could have 0-16khz covered by Opus or similar and anything above that covered by a trained NN model for less sensitive frequency ranges, much like how TSAC works. I'm not a domain expert on that though.

TSAC: Very Low Bitrate Audio Compression was updated from tsac-2024-04-08-win64.zip (https://bellard.org/tsac/tsac-2024-04-08-win64.zip) to tsac-2024-05-08-win64.zip (https://bellard.org/tsac/tsac-2024-05-08-win64.zip).
I thought it could be a typo, but the hashes didn't match in libnc.dll, libnc_cuda.dll, readme.txt (just an addition of a sentence "The CPU
+must support the AVX2 instruction set in order to run the program.") and tsac.exe, so it is a new version.

How is the decoding speed when CUDA is utilized. Software decoding is actually slower than encoding on my workstation.

The decoding speed on a default setting, 281,155bytes 300second stereo music, was 13.97x realtime, 21.5 second on a NVIDIA GeForce RTX 3060, 0.661x realtime, 453 second on a CPU AMD Ryzen 7 5700X. So the GPU was 21 times faster on decoding TSAC.

Measure-Command { tsac-2024-05-08-win64\tsac.exe --cuda -m tsac-2024-05-08-win64\dac_stereo_q8.bin -M tsac-2024-05-08-win64\tsac_stereo_q8.bin d snippet_7sec40.cuda.tsac snippet_7sec40.cuda.tsac.cuda.wav }