Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models (Read 1517 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

This codec apparently appeared in late March. As most ML audio codecs it targets primarily extremely low bitrates (all the examples are < 3kbps). Most people involved are Googlers and they compare it with Soundstream (to which it is superior), so maybe they are part of the same team and it will eventually be included in Lyra? No source code release yet though.

X

X

Quote
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates.

Project page with speech examples

Paper @ Arxiv

Re: LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

Reply #1 –
So LMCodec in its 3/4 setting (0.85 kbps) provides roughly the same audio quality as Opus at 14 times higher bitrate (12 kbps)? I'll have to see and hear that with my own eyes and ears to believe it... on my own set of speech samples.

Chris
If I don't reply to your reply, it means I agree with you.

Re: LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

Reply #2 –
Quote from: C.R.Helmrich on
So LMCodec in its 3/4 setting (0.85 kbps) provides roughly the same audio quality as Opus at 14 times higher bitrate (12 kbps)? I'll have to see and hear that with my own eyes and ears to believe it... on my own set of speech samples.

Chris

There are Audio Examples at the bottom of the page which you could check  ;)

Re: LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

Reply #3 –
Yes, very impressive indeed. But I don't think it is too good to be true.

After all, Opus' SILK still focusses on waveforms, the LPC basis is already ~60 years old. Taking the waveform, extracting a predictor, quantizing the residual etc.

It seems to me like the approach for these machine learning codecs is more like programming a voice synthesizer. In it simplest form you would need to encode a message, timbre, phrasing and attack, more or less. 500 bit/s = 62.5 byte/s. The actual message that is being transferred is closer 10 byte per second. At least, I cannot think of a word of more than 10 characters that I can utter in that time, and even then there is a lot of redundancy in those 10 bytes. Having another 50 byte to encode vocal characteristics does not seem impossible.

The machine learning part is in finding out what parameters the voice synthesis needs, and how to extract them, I think.
Music: sounds arranged such that they construct feelings.
 

Re: LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

Reply #5 –
Ah, yes, didn't think about the latency. According to the paper algorithmic delay is 20ms, so that pretty much rules out using voice recognition indeed.

Still, the paper says
Quote
Transmission of continuous speech features over low-bandwidth channels is achieved via vector quantizers (VQs) [10], where the features are turned into discrete representations while introducing minimal distortion.
So not really turning the speech into words and synthesizing it back, but a more intermediate level, recognizing speech 'features'. I guess that means vowels and consonants, pretty much? The machine tries to encode phonetics instead of samples?
Music: sounds arranged such that they construct feelings.