HydrogenAudio

Introduction

I made several listening tests in my life and at various bitrates (from 48 kbps lately to 190 kbps many years ago). But I never thought I would make one at 12 kbps.

Just to put this little value into perspective: at 12 kbps one hour of music only takes 5.40 Mb—which is usually the size of a single track at 128 kbps. Hotel California full album: 3.97 Mb. Back in Black: 3.82 Mb. Thriller: 3.86 Mb. One full album is half the space needed by a JPG shot done with my cheap smartphone! For older people who knew CD-R burning: 130 hours of music could be stored on a 700 Mb CD-R… Every file is insanely small! And for streaming purpose, EDGE poor network is still enough to stream music without issue (in a tunnel, on mountain…).


But why doing this test?
First, to get some comparison data at unexplored regions and fill some gaps. I am really curious to check how modern audio codecs are dealing with such huge bitrate starvation. When a new video or pictures format appears it’s immediately tested at ultra low bitrate—but audio isn’t. If you take a look on the next graph, you won’t see any popular format below 40 kbps:

(credit: opus-codec.org)

Now that a modern opponent to OPUS is easily available (Exhale first, now Fraunhofer's encoder with EZ Audio Converter) an interesting duel is at least possible.


Second reason: For the sake of curiosity I recently forced myself to listen to ~35 kbps xHE-AAC on my phone, streamed over bluetooth in my car or earbuds and the result was far from disaster. In fact, it was a bit better than acceptable. I even listened to these ultra-low bitrate encoded albums many times during two weeks. This was a real big surprise. Because I’m not fond of low bitrates: 20 years ago I discovered --r3mix, then became a LAME --alt-presets pilgrim and finally turned into a MPC priest. I spent afternoons and weeks on ABX softwares just to get significant scores on castanets samples… It would be an understatement to say that I was totally neurotic about sound quality. Fortunately I’m clearly more tolerant today. So these low bitrate performances are something I wanted to investigate, for practical usage. If I can put on my phone a lot of stuff I rarely listen to without ruining the storage capacity it’s definitely something I should try (it’s less expensive than streaming subscription and even much more reliable—I often get off cellular networks while wandering in the forest or in nature).
Low bitrate encodings is also a very reasonable choice for audiobooks and even movies ripping (Netflix made recently the switch to xHE-AAC to improve quality and efficiency). That’s why 20 samples of voice are included here.



Challengers: The choice of formats/encoder is limited to the most recent ones: OPUS (2012) and USAC/xHE-AAC (2012?). Both are unified formats for voice and music coding. As Exhale open-source xHE-AAC doesn’t handle such low bitrate, only Fraunhofer’s xHE-AAC (included in EZ-CDA Converter) encoder is able to compete here.
Opus : last version on official website (https://opus-codec.org/downloads/) (1.31, executable). VBR is used (auto detection between voice and music).
xHE-AAC : version by Fraunhofer included in EZ CD Audio Converter 9.3.1 (https://www.poikosoft.com). The Fraunhofer (FhG) encoder was updated with 9.3.2—but I missed the update so it’s not tested here. I used VBR preset 0 (24 kbps) and CBR for 12 and 32 kbps.

BITRATE: Average bitrate for a large CD collection

The full bitrate table is available here (https://docs.google.com/spreadsheets/d/18lGNoBB0ZB2A4SGAw3_z-5L7gzl54MG3D_V7tf2jII8/edit?usp=sharing).

SAMPLES: 60 samples in 6 groups:

• music: billboard 2010-2020 greatest hits. Standard range for each: 01.00.000 to 01.20.000. 10 samples, best-sellings music. Difficulty: average.
• music: classical: 10 musical samples from a larger collection. Classical music. Criteria selection: favorite moments. Difficulty: below average.
• music: HA.io: 10 samples submitted by Hydrogenaudio’s members. Many of them were chosen because they revealed some issues with encoders of the past (usually LAME MP3). Difficulty: above average.
• music: problem: 10 samples shared on HA.io that revealed strong encoding issues with encoders of the past. Mostly issues like pre-echo or additional noise on transients. Difficulty: hard to very hard
• voice: audiobooks: 12 samples from audiobooks. 6 male, 6 female. 8 different languages. Criteria selection: random or favorite part. Difficulty: easy.
• voice-movies: stereo samples of famous soundtrack (dialog, with or without music and/or ambient noise). 48 KHz, 16 and 24 bit. Criteria selection: cult dialog, favorite moment, or random. Difficulty: easy.

All samples are available here:
https://www.dropbox.com/sh/zjephy3g54j4gur/AACjGhM9tabl26n7s4ihYl2Ra?dl=0

SOFTWARE & HARDWARE:  I used Java ABC/HR. All files were normalised. Two first seconds of each encoding were discarded from evaluation. An AKG Q701 was plugged on the headphone jack of my computer (I was unable to run my DAC on ABC/HR).



RESULTS:


On the graph I put the overall score for all 60 samples on the middle of each column, the result for the 40 musical samples on the left and the result of the 20 samples of voice on the right.

• At 12 kbps Opus is totally lost on music; it performs significantly better on voice but quality is not truly convincing. The overall result is very poor: sound is muffled, mono-ish, and there is often a bad amount of noise/grain. xHE-AAC sounds differently but much better: lowpass is not as strong, stereo is here but often weird (like out of phase). There’s a hollow and synthetic sound image on many samples. Music is not good to my taste but it’s not catastrophic either. Voice samples (especially movies) are almost acceptable.
  ► 12 kbps has nothing enjoyable but it sounds much better with xHE-AAC than with Opus.
• At 24 kbps Opus is making a big step forward. Sound is not muffled anymore and the stereo image is quite excellent. The problem is a layer of  strong and harsh sibilant artifacts that didn’t exist at 12 kbps. In many cases there’s still an aggressive amount of grain/noise. Music at 24 kbps is still poor (score < 2.0); audiobooks and movies are now acceptable though irritating (score < 2.50). xHE-AAC has much less issue with stereo here and sound is much less synthetic. Music is not stellar but is quite acceptable now (score : 2.77). And voice has minor issues and is good (3.43 on average, and even 3.70 for movies! Hello Netflix!).
  ► 24 kbps is now usable with xHE-AAC (average score: 2.99) and is quite enjoyable with voice recording. OPUS is not really usable on music and its score is still lower than USAC CBR 12 kbps! On voice OPUS 24 kbps VBR reaches the same score than USAC CBR 12. In other words OPUS is far less efficient than xHE-AAC at 12 and 24 kbps.
• At 32 kbps Opus is still not very pleasing with music (score: 2.38) but on voice it’s now more than simply usable (3.19). xHE-AAC makes quite a good job on music (3.21) and is almost stellar on voice (especially movies: score = 4.0, hello again Netflix).
  ► xHE-AAC outperforms OPUS at 32 kbps (in fact 34 kbps) and really plays in a different league.
• 96 kbps USAC (here with Exhale) is excellent in comparison and in absolute terms (probably a bit overrated here as it's included as high anchor)

To finish I just want to remind that this evaluation is personal. OPUS and xHE-AAC have very different kind of distortions and judging one less annoying to a different one if very subjective :)

As always thanks to kamedo2 for its useful tool (https://listening-test.coresv.net/graphmaker6.htm)!

Great test guruboolez!
That Fraunhofer encoder is very impressive.

More proof that mp3 is a dead format.

Thanks,
guruboolez

More proof that mp3 is a dead format.

Not necessarily dead...
Most audio podcasts are still distributed as mp3s and, say, what the TWiT network uses, I believe mono 64 kbps mp3 is perfectly acceptable to my ears for voice.

Sure, there probably are better options for quality in the range below ~128 kbps or so but it's certainly not "dead" as there are still many mp3s being made and they still can be played and probably will be able to for the forseeable future.

More proof that mp3 is a dead format.

Thanks,
guruboolez

Speak for yourself please. There are literally millions of people still using it daily.

More proof that mp3 is a dead format.

lol. not.

For example, the website Bandcamp offers mp3s or FLAC's. And there are others....like Presto for classical music....and then there's this small retailer known as Amazon.

@guruboolez
Excellent work!
This is very interesting test to see the ultimate performance in this extremely low bitrate range. Fraunhofer xHE AAC is definitely very impressive.
From storage affordability point, I personally can't see the reason to go this low in 2021. for local music listening, even for Netflix or other streaming video service it cannot bring too much size reduction because video bitrate is still high.
However, performance and results are still very interesting to see on such low bitrate.

FhG xHE-AAC did 3.41 on 32kbps? That's amazing!

FhG xHE-AAC did 3.41 on 32kbps? That's amazing!

3.41 but on merged results. For music samples score is "only" 3.21 (voice is easier to handle).
As xHE-AAC 32 kbps was the very best encoder of a group of six I would say that the score is *maybe* slightly overestimated. In different testing condition (eg a listening test with 32 - 64 - 96 kbps competitors) I expect a lower score. Same if I put xHE-AAC 32 kbps as low anchor in a 128 kbps test.

But if I spend one hour at home at home on listening music at 32 kbps xHE-AAC I can give a rough estimation of the score at ~3.0 (which literally means "slightly annoying" on the ABC/HR scale). It's listenable with audible but limited annoyance.
Again, also, it's a matter of taste and tolerance.

Did you use the Graphical User Interface of the EZ CD Audio Converter 9.3.1?
If not, could you please share the command line you tested on?

Yes I used EZ CDA Converter through the GUI (does it work on command line too?)

From storage affordability point, I personally can't see the reason to go this low in 2021. for local music listening, even for Netflix or other streaming video service it cannot bring too much size reduction because video bitrate is still high.

Well, it makes sense for Netflix and movies when there is 5.1 & 7.1 surround sound options ;)

-

@guruboolez - as others have said, thank you for this. And again, it really is incredible to see that modern audio can achieve such decent scores from such low bitrates. I have noticed this myself when listening to Mixcloud for example, that streams at 64kpbs with AAC-LC (I'm not 100% sure of the encoder although MediaInfo & foobar report the "writing application" / tool as Lavf57.83.100).

(does it work on command line too?)

No, it doesn't.

Extremely interesting test, thanks a bunch guru!

For example, the website Bandcamp offers mp3s or FLAC's.

Bandcamp offers AAC, Ogg Vorbis and ALAC too.

And I've experienced the case where BandCamp had much larger FLAC files than Apple Lossless, because the FLACs were 48000/24, while the ALAC was 48000/16.

Quote from: synclagz on 2021-05-17 13:04:35

From storage affordability point, I personally can't see the reason to go this low in 2021. for local music listening, even for Netflix or other streaming video service it cannot bring too much size reduction because video bitrate is still high.

Well, it makes sense for Netflix and movies when there is 5.1 & 7.1 surround sound options ;)

You've got a point. I forgot about multichannel sound. :D

@guruboolez
Considering that 32k gives around 3.2-3,4 score, do you think that FhG xHE at 48k could go to 3,5 or maybe 3,8-4.0 quality score?
I think that at 48k (FhG xHE) should be descent for casual listening (especially for untrained ears) :)

Considering that 32k gives around 3.2-3,4 score, do you think that FhG xHE at 48k could go to 3,5 or maybe 3,8-4.0 quality score?

Score must be higher, yes—but how exactly? I can't tell.

I see three limitations here:

► SBR is a great tool at very low bitrate but introduce several flaws. The benefit is quickly fading when bitrate increase.

► Beyond 32 kbps FhG doesn't use Parametric Stereo anymore. For that reason ≥40 Kbps should have a more stereo bitrate-expensive strategy. Stereo image must be better but maybe, I don't know, to the expense of other issues.

► the ABC/HR scale is not very wide. If I had tested 48 Kbps in the same group along with 32 kbps (and 24, 12 and 96 kbps anchor), I maybe had to revamp the scores and compress a little 24 and 32 kbps sliders to the down. Not by much though.


3.4…3.5 for music samples is likely with the tested samples (which I remind are not all difficult ones).

Interesting read. I presume that your "Difficulty" ratings are the obvious thing that killer samples are difficult to encode right? Or is it even so that killer samples are difficult for you to rate because all do them obviously wrong?

(probably a bit overrated here as it's included as high anchor)

When these listening tests came about back in the late neolithic, you wouldn't have expected a sub-100 to be the high anchor for stereo music outside April 1st.
I am impressed.

To illustrate OPUS vs FhG duel at 12 kbps and have a better look on their respective flaws (muffled and narrow sound + agressive layer of noise on some parts for Opus, weird stereo & hollow sound for FhG USAC), here is a small selection:
https://www118.zippyshare.com/v/0KJq3kVP/file.html

and for people who don't have any xHE-AAC player/decoder, the same selection but decoded to FLAC:
https://www29.zippyshare.com/v/FWpYRX5q/file.html


It's a biased selection: all samples were badly ranked for OPUS 12 kbps. But it may illustrate the gap between the two formats at ultra-low bitrate on music material.

Interesting read. I presume that your "Difficulty" ratings are the obvious thing that killer samples are difficult to encode right? Or is it even so that killer samples are difficult for you to rate because all do them obviously wrong?

Yes, those files in group #4 are known difficult samples for at least old lossy encoders (some samples are or were real stars here 20 years ago: castanets.wav, fatboy.wav, enola.wav — happy anniversary Hydrogenaudio!…)

► Beyond 32 kbps FhG doesn't use Parametric Stereo anymore. For that reason ≥40 Kbps should have a more stereo bitrate-expensive strategy. Stereo image must be better but maybe, I don't know, to the expense of other issues.

I didn't know that. Thanks for explanation. FhG xHE will be developed further, I suppose, so quality should increase in the next version or in the near future.

Here is a pack of all 60 samples encoded in both OPUS 12 kbps VBR and Fraunhofer's xHE-AAC CBR 12 kbps. So everyone could take a look on how modern encoders sound at this very small bitrate — compression ratio is 1:117 for RedBook and 1:125 for 48KHz voice samples!).

Thank you guruboolez.

Nice test, Guru.

It would be even greater if HE-AACv2 encoder was there (and here is  a very high quality FhG Winamp HEv2 VBR encoder).

Speaking of Netflix and low bitrates, an average speed of streaming right now is 3300-3400 kbps. https://ispspeedindex.netflix.net/global.  So xHE-AAC is useful for mobile connections  not so much for an average home wifi connection where Netflix still uses E-AC3 and AAC 128-768 kbps.
Also it's seems like Netflix concentrates more on DRC feature of xHE-AAC https://netflixtechblog.com/optimizing-the-aural-experience-on-android-devices-with-xhe-aac-c27714292a33

Why does it need to be the Winamp encoder? Isn't that kind of old by now, compared to the HEv2 encoder in EZ CD Converter?

Because HE-AACv2 is old.
Well, at this point xHE-AAC and Opus are old too (9 years).

Nero has reached a limits what can offer HEv2 more than 10 years ago.
At this point trying something new is like saying "Why try LAME? Try some new MP3 encoder" huh.

All encoders HEv2, v1, LC-AAC, MP3, Vorbis (except Opus and USAC) have reached max quality before 2010-2011.  Since then only some fancy noises about new FDK AAC and some other AAC encoders claiming superiority and comparable quality (which wasn't close to reality by any means!) over Apple AAC (this last one  hasn't seen any meaningful update already for 16 years  (LC) and 12 years for (HE)).

► Beyond 32 kbps FhG doesn't use Parametric Stereo anymore. For that reason ≥40 Kbps should have a more stereo bitrate-expensive strategy. Stereo image must be better but maybe, I don't know, to the expense of other issues.

From what I can recall, the transition point from Parametric Stereo to Stereo is 48 kbps for the Fraunhofer xHE-AAC encoder. My xHE-AAC Internet Radio streams are 40 kbps and I'm pretty sure they are parametric stereo. They do have perceptually excellent stereo imaging that to my ears is better than HE-AACv2 has ever had so it's possible it isn't parametric stereo.
I'm not surprised that xHE-AAC smoked Opus at low bit-rates. When I tested Opus vs. xHE-AAC my ears found Opus was unlistenable for music below 48 kbps and even at 48 was barely tolerable. I find xHE-AAC to be very listenable at 40 kbps and acceptable at 32 kbps.
I tried EZ CD Audio Converter in Windows 10 arm64 running in Parallels 16.5.0 and it works great! Bought the bundle including the tag editor. Now I can encode xHE-AAC outside of my live radio streams! The 64 bit version works best in my Windows 10 arm64 VM. It's possible Apple will add xHE-AAC encoding with macOS 12 but no guarantee that's happening. For now macOS only has decoding.
Thanks to some very smart friends I now have working web players for xHE-AAC with album art and ID tags! The players work great in Safari in macOS 11.x, iPadOS 14.x and should work in some Android browsers in Android 9 or newer. The players require a browser with MSE enabled which for now rules out iOS (maybe iOS 15 will finally enable it?).
Anyone with a qualifying system outside the US is welcome to try the new players. Sorry but to appease Sound Exchange the players are geo-blocked to USA IP numbers. Here are links to the stations:

CKLG (https://www.cklg.ca/), LG73 (https://www.lg73.ca), Max Radio (https://www.maxradio.ca), New West Rock (https://www.newwestrock.ca) and Uptown Radio (https://www.uptownradio.ca).

Look for the "beta xHE-AAC player" links on each site to launch the new web players. I'm not aware of any other existing fMP4/HLS web players that can do album art and ID tags. These new players are a work in progress but I think they're far enough along in development to make them available here. Any feedback from the golden ears here appreciated!

Cheers,
Phil

@imacguru:

According to the last fb2k's decoder made by Kode54, Parametric Stereo is not used beyond 32 kbps CBR:
(https://zupimages.net/up/21/20/ookk.png) (https://zupimages.net/viewer.php?id=21/20/ookk.png)
(tested with 3.5.3 version, EZ CDA Converter 9.3.1)

I'll take a look on those radios (I already tested them last year —it was the only access I had to low bitrate USAC at this time— and I think it was on your recommandations). Thanks for the info!

Nice test, Guru.

It would be even greater if HE-AACv2 encoder was there (and here is  a very high quality FhG Winamp HEv2 VBR encoder).

Yes, indeed. But I already had 7 pairs in this test. It makes 14 files to listen for each sample and 7 files to mark and to put in order. Adding HE-AACv2 would make 10 pairs. I think it's too much. The risk of ranking errors is also greater: less accurate results. Also add fatigue to the equation.

I see xHE-AAC as the successor to the AAC family (LC-AAC, HE-AAC, HE-AACv2). A bit like OPUS is the successor of VORBIS. The successors are supposed to perform better. I made a duel between those successors and discard the formats they replace.

I'm aware that HE-AACv2 is still highly interesting because it's widely supported (DAP, phones, TV, DVD/BR players, cars…). But as I said, it's too much work. A listening test between xHE-AAC, OPUS and HE-AACv2 at a single bitrate is of course more affordable. I would even discard OPUS from such opposition and let HE-AACv2 vs xHE-AAC fight together in a single duel.

Speaking of Netflix and low bitrates, an average speed of streaming right now is 3300-3400 kbps. https://ispspeedindex.netflix.net/global.  So xHE-AAC is useful for mobile connections  not so much for an average home wifi connection where Netflix still uses E-AC3 and AAC 128-768 kbps.
Also it's seems like Netflix concentrates more on DRC feature of xHE-AAC https://netflixtechblog.com/optimizing-the-aural-experience-on-android-devices-with-xhe-aac-c27714292a33

Yes it makes sense. My main usage of Netflix is on a tablet and on downloaded files (because streaming uses a lot of the available ressources in my house, and there are other users who needs bandwidth). I believe it's VP9 and image quality is great (outside some artifacts on dark scenes).
If I take as example a recent Netflix grain-free movie (The Devil all the Time—excellent book IMO), the downloaded file is 828 Mb for 138 minutes including two soundtracks. The average bitrate for 1080p is ~800 kbps (video+audio×2+subtitles). In this example, ~30kbps spared on one audio stream makes near 10% size reduction on this HD movie. Not fully negligible!

Hmm made some comparisons myself and on minimum bitrates xHE-AAC sounds much better indeed, but at 24kbps VBR I begin to prefer Opus...

Opus@24kbps VBR:
+ less pre echo
+ more high freqs
- narrow stereo image

xHE-AAC@24kbps VBR:
+ wide stereo image
- more pre echo
- less high freqs
- annoying short-delay-like artifacts

Except the encoder in EZ CD Converter is just a newer version of the same encoder in Winamp. But I guess it's pointless to see if anything has improved since then.

kode54,
Do You mean  EZ AAC encoder or USAC one?
Because Winamp encoder was AAC encoder.

Hmm made some comparisons myself and on minimum bitrates xHE-AAC sounds much better indeed, but at 24kbps VBR I begin to prefer Opus...

Opus@24kbps VBR:
+ less pre echo
+ more high freqs
- narrow stereo image

xHE-AAC@24kbps VBR:
+ wide stereo image
- more pre echo
- less high freqs
- annoying short-delay-like artifacts

Tastes and colours… :) Thanks for trying on your side.

But I'm curious: what do you mean by “annoying short-delay-like artifacts”? Could you give me a good sample that illustrate it?

Second question: when you say “less high frequencies” do you hear it? I checked on a graphical tool and xHE-AAC has a constant lowpass at ~17.5…17.8 KHz which seems pretty high and should be transparent for most users. Opus has a 20 KHz lowpass (but also 12 KHz on low energy part like on my sample named “Classical A.10. VA Placido Domingo [37.56.000 +30sec].wav”). I couldn't distinguish both encodings on lowpass (but Opus seems to have some additional noise or a kind of energy boost in trebles: is this kind of brightness you perceive as higher lowpass value?)

On Opus I have the same impression as you, there is additional noise at very high frequencies, but the differences are due to SBR which has an unfavorable and different ratio from 2:1 in VBR. Try to encode at 48kHz in CBR at 24kbps and you will hear much better because SBR will only be active above 12kHz.

kode54,
Do You mean  EZ AAC encoder or USAC one?
Because Winamp encoder was AAC encoder.

Huh, I did notice that there are different AAC components in EZ CD Converter, but all of the AAC encoders appear to be different versions of the same ~2.5MB library. I guess that's because the LC/HE/HEv2 portion hasn't changed since last December.

Tastes and colours… :) Thanks for trying on your side.

Yeah at these bitrates it comes down to what kind of artifacts are less annoying : ) At very low bitrates Opus has dropouts which to me are quite disqualifying. But they mostly disappear near 20kbps and then it is mostly noisy and monoish which I prefer over pre echo and ringing.

But I'm curious: what do you mean by “annoying short-delay-like artifacts”? Could you give me a good sample that illustrate it?

I attach a sample.
To me it sounds like there's intermixed another copy of some higher frequencies shifted by tens of milliseconds. It probably is a kind of ringing and/or pre echo artifact which in most codecs smears the transients, but here somehow also adds a shifted shadow?

Second question: when you say “less high frequencies” do you hear it? I checked on a graphical tool and xHE-AAC has a constant lowpass at ~17.5…17.8 KHz which seems pretty high and should be transparent for most users. Opus has a 20 KHz lowpass (but also 12 KHz on low energy part like on my sample named “Classical A.10. VA Placido Domingo [37.56.000 +30sec].wav”). I couldn't distinguish both encodings on lowpass (but Opus seems to have some additional noise or a kind of energy boost in trebles: is this kind of brightness you perceive as higher lowpass value?)

Yes, by high frequencies I don't mean barely perceivable high frequencies but rather those in trebles range - thousands of Hz but rather below 10k - they do seem more bright in Opus and I like it over more muffled xHE-AAC.
I haven't checked the graphics yet, just purely what I hear so far...

BTW kode54 I sometimes get console errors on xHE-AAC seeking in fb2k

Thanks for the sample. I agree that USAC doesn't sound great to me either. I have mixed feelings when I compare it to OPUS. The latter sounds muffled (despite it's high lowpass): I guess it comes from the reduced stereo. It also has audible grain/hiss on transients but in this case it doesn't sound really wrong.

On the other side xhE-AAC provides fresh air with larger stereo image. But on this kind of sample smearing is a real issue. In fact, it's a total mess here. This mess is not fully caused by smearing alone but also by a strange effect I couldn't describe nor explain. Your "shift" explanation makes sense to me and put words on what I hear. There also a metallic sound I dislike.

In this case OPUS sounds less wounded and I could be tempted to give to it a better score.
I checked my test's results: with eig sample (also electronic music) I also gave my preference to OPUS over USAC at 24 kbps.

There also a metallic sound I dislike.

That's exactly what I mean by delay-like artifact. That's how you do cheap metallic/robot voice - apply very short delay effect to it so it sounds like talking in a pot (and that's probably how it was done in early analog era).
Still very impressive performance of xHE-AAC, nice to have a new contender at such low bitrates.

Here is a pack of all 60 samples encoded in both OPUS 12 kbps VBR and Fraunhofer's xHE-AAC CBR 12 kbps. So everyone could take a look on how modern encoders sound at this very small bitrate — compression ratio is 1:117 for RedBook and 1:125 for 48KHz voice samples!).

Thank you very much for posting this. USAC is blowing my mind now - I can't believe what can be done at 12kbps. Genuinely surprised by some of these...

Given the bitrate, I would have said that Opus sounded ok, but my goodness USAC blows it out of the water.

Probably a typo.
It says "CVBR preset 2 98.2 kbps".
Should be preset 3 (?)

Yes exactly. I can't edit the first post. It's preset 3. Thanks for reporting it.

Yes it makes sense. My main usage of Netflix is on a tablet and on downloaded files (because streaming uses a lot of the available ressources in my house, and there are other users who needs bandwidth). I believe it's VP9 and image quality is great (outside some artifacts on dark scenes).
If I take as example a recent Netflix grain-free movie (The Devil all the Time—excellent book IMO), the downloaded file is 828 Mb for 138 minutes including two soundtracks. The average bitrate for 1080p is ~800 kbps (video+audio×2+subtitles). In this example, ~30kbps spared on one audio stream makes near 10% size reduction on this HD movie. Not fully negligible!

I see.
Anyway I think Netflix, Amazon Prime and others overcompress for mobile streaming.

I've just checked that movie and it streams HEVC 5800 kbps/ LC-AAC, stereo 128kbps on my laptop 1080p/HDR and it goes full 4K at higher rates on a Smart TV.
Yeah, if You are already in the bed and don't want to wake up anybody then mobile ~800 kbps with audio-30kbps is the last resort. But quality of audio/video streams isn't optimal.  :(

At 12 kbps Opus is totally lost on music; it performs significantly better on voice but quality is not truly convincing. The overall result is very poor: sound is muffled, mono-ish, and there is often a bad amount of noise/grain.

That's why I tend to use a bare minimum of 13kbps for Opus (v1.3) when it comes to speech since, at least off the top of my head from my previous playing around with Opus on speech (on my Klipsch Pro-Media speakers, which are above average PC speakers), that any lower than 13kbps, the sound quality starts to decline much more rapidly (i.e. the minimal storage space gains are no longer worth any further decline in sound quality). it seems to clean up the muffled aspect a bit going from 12kbps to 13kbps as I think that's ultimately why I settled on 13kbps as a bare minimum on speech. NOTE: although if one wants to play it a bit safer they could try 16kbps or 24kbps or 32kbps as, off the top of my head, I can't see using any more than 32kbps with Opus for speech as I am of the mindset that I don't need near perfection on speech and would rather go for storage space efficiency.

so, at least in my opinion, don't use Opus any lower than 13kbps for speech as a general rule as I would say it's a good starting point for someone trying to find a really low bit rate to use on their speech encoding. I realize that it's possible, depending on the type of voice, that 13kbps might vary a bit. but still, it's probably not going to be bad enough for me to avoid if I want to store plenty of speech (especially if one is not too fussy on speech quality) and keep the file size minimal.

one last thing... at least based on 13kbps, I can't see Opus for speech only having a slightly higher score than music (i.e. call it roughly 1.4 vs 1.8 which is in the 'annoying' and 'very annoying' section of the 1-5 rating system used around here (although I get the OP used 12kbps and not 13kbps like I am generally referring to)) because of this obvious thing for me... speech is usable, unlike music @ 13kbps (NOTE: although considering the VERY low bit rate, the music could be worse ;) ) ; NOTE: I am not dissing the OP (as I thank the OP for the effort), just making a point is all. and....

...I would say Opus @ 13kbps is no worse than the 'slightly annoying' range on the 1-5 scale (i.e. (5)"Imperceptible", "Perceptible, but not annoying", "Slightly annoying", "Annoying", (1)"Very annoying"). because while you can notice the quality decline vs a higher quality source file, it's not a significant enough of a decline for me to flat out not use it and it's great for keeping passable sound quality speech at a very low file size, especially if you got a lot of speech (say many hours) to store as this greatly increases storage space efficiency.

or another way to put it... I would consider 13kbps to be more Thumbs Up than Thumbs Down for whatever that's worth, although 12kbps and lower I start to shift more to Thumbs Down than Thumbs Up for speech as a general guideline for those seeking to use the lowest possible speech bitrate.

p.s. so I think music, unlike speech, most people would tend to be more picky on obtaining a higher quality for that where as speech as long as it's clear enough and easy enough to understand what's being said, then it's 'good enough'. like I can easily tolerate speech quality that's noticeably lower than 'transparent' than I could with music where a little decline here and there can be okay but if it becomes more obvious I just tend to play it safer with a higher bitrate. for the record... I can easily use Opus @ 64kbps for music as while I can go lower to say 48kbps etc, it's no longer worth the minimal storage space savings for the sound quality hit (I think 64kbps Opus is a safe minimum for music for the common person in my estimations). so I am more of the mindset to use Opus @ 13kbps for speech (minimum) and 64kbps for music (minimum). like I have mentioned before with Opus on music, I think if someone is happy with MP3 @ V5 (130kbps), it's plausible Opus @ 64kbps will be good enough to (even though I was told a while ago that Opus @ 80kbps is more of a equivalent to MP3 @ V5, but I think the average person probably won't notice it even though those around here might etc).

one last thing... at least based on 13kbps, I can't see Opus for speech only having a slightly higher score than music (i.e. call it roughly 1.4 vs 1.8 which is in the 'annoying' and 'very annoying' section of the 1-5 rating system used around here (although I get the OP used 12kbps and not 13kbps like I am generally referring to)) because of this obvious thing for me... speech is usable, unlike music @ 13kbps (NOTE: although considering the VERY low bit rate, the music could be worse ;) ) ; NOTE: I am not dissing the OP (as I thank the OP for the effort), just making a point is all. and....

No problem for me. Thanks for sharing your experience.
Let me correct I slight mistake: music was scored at 1.18 and voice/speech at 1.83. The gap is a bit greater than 1.4 vs 1.8.
With some samples I was also tempted to go below 0 for Opus 12 because sound was destroyed. Comments were rare on my logs for this test but here is one:
Example:

4L File: D:\ABXbitrate\very low bitrate opus vs usac\TEST\Classical D.01. Bach [0.00.000 +30sec].opus12.wav
4L Rating: 1.0
4L Comment: 0, or maybe -1!

...I would say Opus @ 13kbps is no worse than the 'slightly annoying' range on the 1-5 scale (i.e. (5)"Imperceptible", "Perceptible, but not annoying", "Slightly annoying", "Annoying", (1)"Very annoying"). because while you can notice the quality decline vs a higher quality source file, it's not a significant enough of a decline for me to flat out not use it and it's great for keeping passable sound quality speech at a very low file size, especially if you got a lot of speech (say many hours) to store as this greatly increases storage space efficiency.

I don't have any opinion for opus 13 kbps (and you're learning me something by saying that 13 kbps is really different than 12 kbps), but at 24 kbps the voice samples get a much more acceptable score (2,42). It corresponds to a bit more than slightly annoying quality. Opus 24 kbps has sometimes annoying issues (on sibiliant for example). But It was only ranked three time below 2.0: on the korean audiobooks (the only audiobook sample with some music in the background), and on two movies (orchestral music in the background).

p.s.  like I have mentioned before with Opus on music, I think if someone is happy with MP3 @ V5 (130kbps), it's plausible Opus @ 64kbps will be good enough to (even though I was told a while ago that Opus @ 80kbps is more of a equivalent to MP3 @ V5, but I think the average person probably won't notice it even though those around here might etc).

I made a similar conclusion last year while comparing OPUS/EXHALE at 64 kbps with LAME -V5 as high anchor:
https://hydrogenaud.io/index.php?topic=119333.0
(https://zupimages.net/up/20/23/dlxe.png)
Opus at 64 kbps got an higher score than LAME -V5 with 25 samples from bliiboard charts. But with classical music only Opus 64 kbps results were also far less convincing.

Let me correct I slight mistake: music was scored at 1.18 and voice/speech at 1.83. The gap is a bit greater than 1.4 vs 1.8.

Yeah, my mistake ( so about 1.2 vs 1.8 ). I thought the center line was it when it was the one on the left.

I don't have any opinion for opus 13 kbps (and you're learning me something by saying that 13 kbps is really different than 12 kbps), but at 24 kbps the voice samples get a much more acceptable score (2,42). It corresponds to a bit more than slightly annoying quality. Opus 24 kbps has sometimes annoying issues (on sibiliant for example). But It was only ranked three time below 2.0: on the korean audiobooks (the only audiobook sample with some music in the background), and on two movies (orchestral music in the background).

I am sure my testing is not as thorough as yours since I was doing limited testing on some male voice (English) speech. so it's possible you might not get too much of a improvement had you switched to 13kbps instead of 12kbps as I think the main benefit is the overall speech clarity is improved (i.e. not as muffled). but if other aspects of the weaker sound bother you, then it might still score fairly weak for you etc.

because at least on what speech sound files I was playing around with (male voice (English)), 13kbps is what I consider a bare minimum usable bitrate (although it's possible sometimes I may opt for going to a higher bit rate), even though I obviously agree with you that bumping up bit rates higher is surely a safer bet across a wider range of voice samples since we are not running the bit rates on the edge at that point like we are in the 12-13kbps (and the like) range. but again, it probably boils down to the kind of voice one encodes and opinions vary on what sounds 'good enough' etc.

sure, I have went down to 12kbps before and I can still hear what's being said etc, so it's not like things completely drop of a cliff from 13kbps down to 12kbps on the speech I briefly tested, but there seems to be a bit more clarity in the overall speech sound at 13kbps which is ultimately why I drew the line at 13kbps as a bare minimum for me. NOTE: this is based on my Klipsch Pro-Media PC speakers.

so for you to be generally in the 1-2/5 range for your 12kbps speech testing... I feel, at least on the speech I was hearing at 13kbps, that your 1-2 scores might be a little low (but how we do those rating scales I am sure are going to vary a bit from person to person and it appears your using a wider range of samples to which will probably further lower the overall results vs what I did (but this is not a dis on you as I suspect this is probably expected since I was in a narrower/limited range of testing unlike you)) as if I had to pick a 1 through 5 with 5 being perfect and 1 being horrible, with no in-between scoring, I feel about a 3 sounds right if we want to keep things really simple on that 1 through 5 rating scale without details.

but since you tested 12kbps and to give a brief comment here from memory... on my 12kbps brief test a while ago, while I can't say I would go to a 1/5 score (since things are pretty much totally shot at this point and I am sure I could simply lower bit rate a fair chunk lower from 12kbps to truly get into the 1/5 score range), I might be closer to the 2/5 range at that point of my opinion of 12kbps on speech for Opus.

so we probably ain't far apart from my best guesstimate and, like I was saying, your testing a wider range of sound samples which will probably further lower your overall opinion of 12kbps vs what brief testing I did at 12kbps with male voice (English). so all-in-all, we probably ain't far apart (maybe some, but nothing major).

I might even say this... at least on the speech files I got, while if I went from say 13kbps to 24kbps I would probably get some level of sound quality increase, I am more of the mindset it's not enough of a difference and I would rather nearly split the file size in half by using 13kbps. but again, some might disagree with me and say it's worth the difference for about twice the file size. but like I mentioned before I try to minimize the file size as this way if I store many hours of speech audio, it's barely going to use any storage space. for example... one 2hr50min speech file of Opus @ 13kbps is only 16.6MB.

I made a similar conclusion last year while comparing OPUS/EXHALE at 64 kbps with LAME -V5 as high anchor:
https://hydrogenaud.io/index.php?topic=119333.0
(https://zupimages.net/up/20/23/dlxe.png)
Opus at 64 kbps got an higher score than LAME -V5 with 25 samples from bliiboard charts. But with classical music only Opus 64 kbps results were also far less convincing.

Given your test there it's almost like Opus is better in some ways (but you said Classical might slip up a little though) even though I realize that statistically they are tied. but just looking at the graph, Opus has about 2 samples that are below a certain point (call it about 3.5) where as MP3 has 10 samples. still, I imagine in general day-to-day real world use, both are probably good enough for many people.

also, if I am reading that chart correctly... it seems like Apple AAC @ 64kbps and MP3 @ V5 (130kbps) are statistically tied (but just barely). but personally I would not hesitate in choosing MP3 @ V5(130kbps) over AAC-LC(Apple) @ 64kbps since MP3 seems like the safer all-around choice. or another thing... I suspect ill be able to more easily ABX AAC-LC @ 64kbps than I could MP3 @ V5 (130kbps). not that this is saying much for AAC-LC since 64kbps is really starting to push it (as in straining that format) where as MP3 is still considered good @ V5 for the average person. but I guess that's where all of these a bit more modern encoders come into play in that anything from about 96/128kbps+ it don't matter too much on what you use for the common person. but if you stay around 64kbps and less, then AAC-LC/MP3 starts to fail compared to Opus (and the like).

thanks for the info.

NOTE: this is based on my Klipsch Pro-Media PC speakers.

I did my evaluation on headphones (AKG q701 “Quincy Jones”) (http://archimago.blogspot.com/2021/05/measurements-akg-q701-quincy-jones.html) and laptop jack output. It may explain why I'm more annoyed by some issue.
I couldn't test on speakers: I removed them to please my wife  :-X

Thanks for doing this test! It got me curious about xHE-AAC and indeed, it works really well in many cases.

However, I can't make it perform better than Opus for mono voice/audiobooks at ~20-30kbps.
To me, Opus at 24k sounds better than xHE-AAC at ~28k, at least with the two clips I tried. The xHE artifacts are quite obvious and unpleasant.

I used exhale 1.1.5 from rarewares and settings 0 and b in foobar (I had to resample to 32kHz to make 0 work).
I'm not too familiar with exhale settings, so maybe I messed something up... anyway, I'm attaching all the samples, source flacs included.

Thanks for doing this test! It got me curious about xHE-AAC and indeed, it works really well in many cases.

However, I can't make it perform better than Opus for mono voice/audiobooks at ~20-30kbps.
To me, Opus at 24k sounds better than xHE-AAC at ~28k, at least with the two clips I tried. The xHE artifacts are quite obvious and unpleasant.

I used exhale 1.1.5 from rarewares and settings 0 and b in foobar (I had to resample to 32kHz to make 0 work).
I'm not too familiar with exhale settings, so maybe I messed something up... anyway, I'm attaching all the samples, source flacs included.

Thanks for your interest :)
I admit that your xHE-AAC encodings don't sound great. Opus is indeed clearer and less distorted.
I joined xHE-AAC encoded with Fraunhofer's encoder. I must be completely tired because I can't get matching loudness  :o
Tell me if it sound better to your ear

Take a TV commercial that I leave you and try, if you do not feel the distortion and the bands removed, the encoder is not the only problem. However, I'm not happy with the result either, especially for the speech after the tenth and twentieth second.

Wave (http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-48k.wav) - xHE-AAC Fh (http://celona.altervista.org/pelizzoni/Promo_Sky_In_Treatment-m.m4a) - xHE-AAC Exhale (http://celona.altervista.org/pelizzoni/Promo_Sky_In_Treatment-m.m4b) - Opus (http://celona.altervista.org/pelizzoni/Promo_Sky_in_treatment-m.opus)

In my opinion the minimum bitrate is 36kbps, between 40 and 48kbps for Exhale. Since for xHE-AAC I can control Exhale better from the command line, deciding not to use SBR for example, it is my favorite when I check the quality. Opus is at its worst in the last few seconds.

Tell me if it sound better to your ear

I have listened to your files but in my opinion for the human voice alone xHE-AAC must go down to 32kHz sampling. For Opus it does not matter because nominally it remains at 48kHz, actually removing the content of the last band removes any signal above 15.625Hz and removing the previous one removes everything that exceeds 12kHz leaving you with excellent telephone quality.

Take a TV commercial that I leave you and try, if you do not feel the distortion and the bands removed, the encoder is not the only problem. However, I'm not happy with the result either, especially for the speech after the tenth and twentieth second.

Is the USAC file made with Exhale? (I wonder because it's SBR without Parametric Stereo).
I listened to both encodings: OPUS has very agressive sibilants which makes the file hard to listen from beginning to the end (on headphones at least; I tried on my laptop sluggish and pathetic speakers it just sound fine :)) ). Voice is also metallic from time to time with this OPUS 24 kbps encoding.

USAC sounds here much less agressive to my ears but sibilants are not fantastic. From second 10 there's also some additional noise on voice that degrades the clarity. It's not enjoying but I'd say it sounds acceptable (much more than OPUS).

Yes, exhale 1.1.5 (x64), the test file is mono.

Quote from: Brand on 2021-05-29 14:27:52

Thanks for doing this test! It got me curious about xHE-AAC and indeed, it works really well in many cases.

However, I can't make it perform better than Opus for mono voice/audiobooks at ~20-30kbps.
To me, Opus at 24k sounds better than xHE-AAC at ~28k, at least with the two clips I tried. The xHE artifacts are quite obvious and unpleasant.

I used exhale 1.1.5 from rarewares and settings 0 and b in foobar (I had to resample to 32kHz to make 0 work).
I'm not too familiar with exhale settings, so maybe I messed something up... anyway, I'm attaching all the samples, source flacs included.

Thanks for your interest :)
I admit that your xHE-AAC encodings don't sound great. Opus is indeed clearer and less distorted.
I joined xHE-AAC encoded with Fraunhofer's encoder. I must be completely tired because I can't get matching loudness  :o
Tell me if it sound better to your ear

Ah, yes, now these sound much better than the exhale encodes. After a quick examination, I'd say about as good as Opus or even a bit better, at least for clip1.

The loudness matching is probably a challenge because my samples are mono and yours are stereo. (The sources are mono.)
Switching to mono playback (with foobar's DSP) made them equally loud for me.

I can't make it perform better than Opus for mono voice/audiobooks at ~20-30kbps.
To me, Opus at 24k sounds better than xHE-AAC at ~28k, at least with the two clips I tried. The xHE artifacts are quite obvious and unpleasant.

I used exhale 1.1.5 ...

Please keep in mind that xHE-AAC is a coding standard, and exhale is a particular encoder for that standard. Also, exhale does not implement the speech coding functionality which the xHE-AAC standard provides, so it's very likely that, at low bit-rates, the Fraunhofer xHE-AAC encoder (which supports the speech coding part) sounds quite a bit better than exhale on voice recordings.

Chris

It's mistake of method. No encoder at the lowest bitrates encodes rightly all bands, and if you only try the human voice, especially male voices, between 12kHz and 15kHz you will stop hearing any difference, simply because they contain only noise.

Opus simply eliminates all information between 12kHz and 24kHz to reserve the bits where they are most useful, in the lower bands (and at lower bitrates it cuts even more). This makes it look best when the content is voice-only, which is the prevailing audio content, but it's not the only one. It makes no sense to compare Exhale with Opus in this case, because Exhale only partially implements xHE-AAC, all patented parts are missing and they are for voice at low bitrates. It makes more sense to compare with the Fraunhofer IIS encoder, but the EZ CD implementation got worse to my credit, I advised against using some sample rates and in less than 24 hours the developer removed all except two of them.

If you encoding monophonic speech content and you step from 32kHz to 48kHz sampling the bitrate has to increase by 50%, with Opus it is not obvious because it deceives the user by removing the bands. For this reason I have indicated 36kbps as necessary (24kbps + 50%).

https://hydrogenaud.io/index.php?topic=120997.msg998434#msg998434

Again for the same reason I have provided a Wave (lossless from the start) file to use as a test. It also contains noise, it is an Italian TV commercial and contains 2 words capable of highlighting the limits of the encoding and a fade-out that show the limit of Opus. The ACELP part, like Opus, requires the human voice to be encoded up to 16kHz, using sampling frequency at 44.1kHz or even worse at 48kHz for voices (except Opus cheating) only serves to wreak havoc on the efficiency of the encoder, in especially with Exhale. In my opinion it is more interesting to observe how they behave just above 32kbps, where the quality changes completely for everyone, for example at 36kbps (use CBR because otherwise you will get much higher average bitrates).

Obviously, when the content is musical, higher sampling rates are required and for a monophonic source, without SBR, given to Exhale 12kbps more than the others, in summary it takes 48kbps to be sure of obtaining a high quality and it was created for this, it takes double the space that today it costs nothing as bandwidth and as space, otherwise you will have to settle for what's left. I can't hear the music encoded with these encoders at this bitrate, but neither can speech, for work I keep hours of political debates every day and I have not yet decided to use any of these new encoders. Guruboolez has perfectly understood that prolonged listening is difficult, I also add that with intramaurals it is also dangerous if at high volume and with Opus too.

Our brains don't find hissing annoying for no good reason. If you need stereo track, you can find it here:

http://celona.altervista.org/pelizzoni/Sky_In_treatment-s-48k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-s-44k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-s-32k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-s-24k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-s-16k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-s-12k.wav

Or mono here:
http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-48k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-44k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-32k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-24k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-16k.wav
http://celona.altervista.org/pelizzoni/Sky_In_treatment-m-12k.wav

You will be able to verify by hand that with xHE-AAC as the bitrate decreases you have to reduce the high frequencies, as Opus does without your knowledge. Otherwise you will hear much worse.

More proof that mp3 is a dead format.

Thanks,
guruboolez

MP3 still very popular format & is transparent for most at 192Kbps VBR. Universal support is a hard thing to shake off with 256GB memory cards many will just encode at V1 ~ 320kbps without much care about the newer codecs.

Look, one thing is to move from 24kbps to 48kbps and another is to reach 192kbps. What you call MP3 is MPEG-1 Layer 3 and you can still use it only because all MPEG-2 NBC decoders (known as AAC) support the older standard.

The advantage of using standard formats is this and should be recognized for containers too.

I think it is useless to deny that MPEG-2 has brought significant improvements allowing to contain with AAC-LC at 64kbps for voice (3 times less than 192kbps) what today we could consider to use between 36 and 48kbps (from 4 to 5.3 times less of 192kbps).

USAC will do the same, with 1/4 of the bitrate and without the old tools of HE-AAC v1 and v2 (SBR first) which now only serve to make unacceptable compromises with quality (like MP3 Pro). My interest is not aimed at reducing the bitrate, but at improving the quality at the same bitrate.

In fact, these bitrates are not far from what can already be achieved with HE-AAC, a format of 18 years ago (2003). The descent of the bitrate can reach the synthesized voices, at which point I have the eyes to read.

and you can still use it only because all MPEG-2 NBC decoders (known as AAC) support the older standard.

That statement has an easy-to-test consequence: "all MP3 players from the last couple of decades, play AAC".
True or false?

You would have had some kind of point about MP2; you had several players that would not accept the ".mp2" suffix - but, happily eat MP2 files when renamed to .mp3.  But people want to be able to play back the files they already have, so not supporting MP3 would remove quite a bit of your market potential. Well nowadays cell phones have relegated dedicated portable players to niche product, so the question is kinda moot anyway.

I played around with the Fraunhofer's xHE-AAC encoder (EZ CD Converter), just with the two audiobook clips I used before.

To me, FhG does worse than 24k Opus at 12k and also at 16k. At 16k it maintains a good timbre, but has some ringing/resonant artifacts, very apparent for example in the second half of clip1.
At 20k, however, I'd give a slight edge to FhG, but it's also a matter of taste... FhG sounds a bit "sharper".
Opus at 12k is a mess, but we already know this. At 16k it's not as bad, though. Maybe you would even prefer it to FhG at 16k, if you find the ringing artifacts annoying.

Anyway, I don't want to make any big conclusions from this. It's just two samples at just a few bitrates. The more interesting fight for audiobooks is probably going to be at around 30-40k, where we're already reaching transparency, or at least no obvious artifacts.

Quote from: soundping on 2021-05-16 22:19:01

More proof that mp3 is a dead format.

Thanks,
guruboolez

MP3 still very popular format & is transparent for most at 192Kbps VBR. Universal support is a hard thing to shake off with 256GB memory cards many will just encode at V1 ~ 320kbps without much care about the newer codecs.

Quote from: soundping on 2021-05-16 22:19:01

More proof that mp3 is a dead format.

Thanks,
guruboolez

MP3 still very popular format & is transparent for most at 192Kbps VBR. Universal support is a hard thing to shake off with 256GB memory cards many will just encode at V1 ~ 320kbps without much care about the newer codecs.

I agree with your thoughts and settings. I'd use something around V1 .
In a general sense, anything from V4 or more.

At this bitrate its obvious that size over quality is favoured .
mp3 can be used similarly say around 96 k for stereo or less for mono.
Try -V5 --lowpass 12.5 -b96 , -b 96 --lowpass 12.5 , -b 48 -mm --lowpass 12.5 ,
-b32 -mm --lowpass 7

I played around with the Fraunhofer's xHE-AAC encoder (EZ CD Converter), just with the two audiobook clips I used before ...

Anyway, I don't want to make any big conclusions from this. It's just two samples at just a few bitrates. The more interesting fight for audiobooks is probably going to be at around 30-40k, where we're already reaching transparency, or at least no obvious artifacts.

With the second clip the previous version of Exhale shows more difficulties. We have many variables in the field, different versions of the encoders, different contents, different bitrates, it is not easy to reach conclusions. If from the example I have proposed we can deduce the difficulty of Opus with the fade in and the fade out of single notes, with only spoken contents the judgment is completely overturned in favor of Opus. I think we need to start distinguishing the new encoders by their ability to behave like telephone encoders for low bitrate voice. We write Opus but we will get similar results with Silk and therefore I also tried the AMR-WB predecessor and it seems to behave better.

We can also add the two EBU tests proposed by Fraunhofer and the substance would not change much even with Opus, up to 32kbps only the encoding of Silk is used, which however is used for sounds up to 8kHz exclusively and up to 16kHz cooperatively even at higher bitrates.

https://www2.iis.fraunhofer.de/AAC/xhe-aac-compare-tab.html (https://www2.iis.fraunhofer.de/AAC/xhe-aac-compare-tab.html) for checking EBU test, tracks 49 and 50 compressed by Fraunhofer xHE-AAC encoder.

We are comparing telephone and MPEG-1 encoders, it just can't stand up to any comparison due to its simplicity and lightness of MP3.

Uncompressed test file:
Clip 1 from Brand (http://celona.altervista.org/EBU/1.wav);
Clip 2 from Brand (http://celona.altervista.org/EBU/2.wav);
Female voice, Track 49 from EBU test (2008) (http://celona.altervista.org/EBU/49.wav);
Male voice, Track 50 from EBU test (2008) (http://celona.altervista.org/EBU/50.wav).

Compressed files at 24kbps:
AMR-WB - Clip 1 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/1.amr) - Clip 2 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/2.amr) - Track 49 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/49.amr) - Track 50 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/50.amr);
MPEG-1 (MP3) - Clip 1 (http://celona.altervista.org/EBU/24kbps/lame_mp3/1.mp3) - Clip 2 (http://celona.altervista.org/EBU/24kbps/lame_mp3/2.mp3) - Track 49 (http://celona.altervista.org/EBU/24kbps/lame_mp3/49.mp3) - Track 50 (http://celona.altervista.org/EBU/24kbps/lame_mp3/50.mp3);
Opus - Clip 1 (http://celona.altervista.org/EBU/24kbps/opus/1.opus) - Clip 2 (http://celona.altervista.org/EBU/24kbps/opus/2.opus) - Track 49 (http://celona.altervista.org/EBU/24kbps/opus/49.opus) - Track 50 (http://celona.altervista.org/EBU/24kbps/opus/50.opus);
HE-AAC - Clip 1 (http://celona.altervista.org/EBU/24kbps/fh-iis_he-aac/1.m4b) - Clip 2 (http://celona.altervista.org/EBU/24kbps/fh-iis_he-aac/2.m4b) - Track 49 (http://celona.altervista.org/EBU/24kbps/fh-iis_he-aac/49.m4b) - Track 50 (http://celona.altervista.org/EBU/24kbps/fh-iis_he-aac/50.m4b);
xHE-AAC - Clip 1 (http://celona.altervista.org/EBU/24kbps/fh-iis_xhe-aac/1.m4a) - Clip 2 (http://celona.altervista.org/EBU/24kbps/fh-iis_xhe-aac/2.m4a) - Track 49 (http://celona.altervista.org/EBU/24kbps/fh-iis_xhe-aac/49.m4a) - Track 50 (http://celona.altervista.org/EBU/24kbps/fh-iis_xhe-aac/50.m4a);
Exhale with SBR - Clip 1 (http://celona.altervista.org/EBU/24kbps/exhale/1.m4b) - Clip 2 (http://celona.altervista.org/EBU/24kbps/exhale/2.m4b) - Track 49 (http://celona.altervista.org/EBU/24kbps/exhale/49.m4b) - Track 50 (http://celona.altervista.org/EBU/24kbps/exhale/50.m4b);
Exhale without SBR - Clip 1 (http://celona.altervista.org/EBU/24kbps/exhale/1.m4a) - Clip 2 (http://celona.altervista.org/EBU/24kbps/exhale/2.m4a) - Track 49 (http://celona.altervista.org/EBU/24kbps/exhale/49.m4a) - Track 50 (http://celona.altervista.org/EBU/24kbps/exhale/50.m4a).

I tried to create files in Opus without Silk encoding, to get more information about it, but I'm not sure if this is actually the case, because the encoder used has evolved over the years.

Opus without Silk in CAF container - Clip 1 (http://celona.altervista.org/EBU/24kbps/opus/1.caf) - Clip 2 (http://celona.altervista.org/EBU/24kbps/opus/2.caf) - Track 49 (http://celona.altervista.org/EBU/24kbps/opus/49.caf) - Track 50 (http://celona.altervista.org/EBU/24kbps/opus/50.caf);
Opus without Silk decoded in Flac - Clip 1 (http://celona.altervista.org/EBU/24kbps/opus/1.flac) - Clip 2 (http://celona.altervista.org/EBU/24kbps/opus/2.flac) - Track 49 (http://celona.altervista.org/EBU/24kbps/opus/49.flac) - Track 50 (http://celona.altervista.org/EBU/24kbps/opus/50.flac);
Opus in ISO/IEC base media file format container - Clip 1 (http://celona.altervista.org/EBU/24kbps/opus/1.mp4) - Clip 2 (http://celona.altervista.org/EBU/24kbps/opus/2.mp4) - Track 49 (http://celona.altervista.org/EBU/24kbps/opus/49.mp4) - Track 50 (http://celona.altervista.org/EBU/24kbps/opus/50.mp4).

And finally I tried to slip AMR-WB with which the new standard maintains compatibility in the ISO container. To play it on the iPhone is a bit complicated, it must be sent via Airdrop from another Apple product and opened with the Voice Notes app. It works.

AMR-WB in ISO/IEC base media file format container - Clip 1 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/1.m4a) - Clip 2 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/1.m4a) - Track 49 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/49.m4a) - Track 50 (http://celona.altervista.org/EBU/24kbps/libvo_amrwb/50.m4a).

Obviously ACELP is only effective with the human voice, so it is a full-fledged telephone encoder, while Opus is a generalist encoder that uses two encoders to get better results than the same when not cooperating. Fraunhofer xHE-AAC does the same while all other encoders belong to the previous generation or implement only part of the standard, such as Exhale.

However, when the content is musical or hybrid, the judgment is reversed against Opus and even Exhale, but only at higher bitrates, it manages to obtain better results (see https://hydrogenaud.io/index.php?topic=120997.msg998446#msg998446 for previous test file). Opus cannot be transparent below 32kbps because it does not use high quality encoding under these conditions. But not even the other encoders achieve sufficient results at such low bit rates.

I do not face the compatibility issue because it is complicated but in summary AAC has no equal among the most recent.

At this bitrate its obvious that size over quality is favoured .

Yes, it's obvious :) 12, 24 and 32 kbps are extreme bitrates, especially for music encoding.

mp3 can be used similarly say around 96 k for stereo or less for mono.
Try -V5 --lowpass 12.5 -b96 , -b 96 --lowpass 12.5 , -b 48 -mm --lowpass 12.5 ,
-b32 -mm --lowpass 7

Yes, quality could be similar but efficiency is way below. The point of this test was to see how efficient new formats are at bitrate that were formerly known as unusable for music encoding :)

Opus at 12k is a mess, but we already know this. At 16k it's not as bad, though.

Yeah. that's why I tend to use 13kbps as a absolute bare minimum with Opus v1.3(or higher) for speech as I feel at 13kbps the sound quality is 'just' good enough as it's a solid option for those who want to keep file size as low as possible while maintaining a sound quality that's not TOO low. but if someone wants to play it a bit safer on speech sound quality then they will obviously have to increase bit rates a fair amount.

but at the same time... I don't know about everyone else but with speech I am not nearly as concerned with keeping it near transparent like I would be with music. hence, I can say it's 'good enough' at a lower point than I would with music as with music I tend to go a little higher instead of trying to run it on THE edge like I did with speech at 13kbps. but speaking of this, I might say the following with Opus in regards to speech and music...

-Opus (speech) = 13kbps
-Opus (music) = 40kbps or 48kbps

those are in regards to what I would probably consider a bare minimum with each (although I am playing it just a touch safer (i.e. higher kbps) with music though). I know opinions will vary on this stuff, but I am trying to roughly take a bare minimum approach to use with both but I played it a bit safer on the music side of things, since like I was saying, I tend to be a bit pickier on the sound quality of music than I would be on speech. so I don't quite think my 40kbps/48kbps for music is on THE edge like I was with the 13kbps speech suggestion, but it's probably close enough given I don't want to run sound quality where things start to become more obviously worse.

p.s. but like my signature currently shows, with Opus I tend to avoid any less than 64kbps for music as I feel 64kbps is a pretty strong balance of those trying to keep file size at near a bare minimum while still maintaining a sound quality in the ball park of MP3 @ V5 (130kbps average) which does well enough in a public listening test.

In a general sense, anything from V4 or more.

I tend to see V4 (LAME MP3) to be more of a odd-ball/kind-of-useless setting because I think one can see MP3 in general more along the lines of use 'V5' or 'V3 or better'. because V4 is only 10kbps lower than V3 and according to the hydrogenaudio wiki page, V3 is the start point of the highest quality settings. so I figure if someone is going to use a lower bit rate than V3 on MP3, they are probably best off sticking with V5 (130kbps average) and forget about it as V5 does pretty well in public listening test and is efficient with bitrate. so at least with V5, besides doing well in public listening test, it also has a decent decrease in bit rates to unlike V4 which is only 10kbps shy of the higher quality settings. so V4 seems like a pointless setting to use if you ask me given bit rate to quality given how quality and bitrate scales from 130kbps(V5) to 165kbps(V4) to 175kbps(V3) to 190kbps(V2) to V1(225kbps) to V (245kbps).

so I think, at least in my mind, when it comes to MP3 it's pretty much V5 (130kbps) or V2 (190kbps) and forget about it as these two options basically should cover a high percentage of people in my opinion. like those who prefer more efficient bit rate (i.e. V5) and those who want higher quality sound but want some level of efficiency (i.e. V2) as beyond V2 efficiency pretty much goes out the window and sound quality gains got to be minimal/negligible in real world use.

but I can't directly fault you for saying "V4 or more" since basically anything from "V5 or more" is good enough in a very basic sense (hell, some might be able to get away with settings lower than V5). although I would probably say for those who prefer the higher bit rates or so, given only the 10kbps difference between V4 to V3 and given what the Hydrogenaudio wiki page says, seems like those types would think more along the lines of "V3 or more". hell, I suppose one could argue that since the difference between V3 to V2 is only 15kbps more and could give a bit of a safety buffer one could use that etc.

but with all of that said... I know storage space is cheap and all nowadays, so what I said above probably ain't going to matter to most people anymore since one could argue it won't really matter much whether someone uses V5 or all the way to the MAX of 320kbps CBR. but it's more of the thought of it for efficiency sake (for us OCD types around here) ;) ; one last little thing... I guess even with the storage space to burn argument taken into account, a more efficient file size would still be a bit wiser like in a situation one were to upload/backup a bunch of their stuff online to where storage space to burn would be less likely, or someone were to store a good amount of music on their smart phone since many are probably in the 8GB or 16GB of internal memory range etc.

Excellent thread.  To me, a classic until Opus gets better in the 24 kps and under, which is the only bitrate I will accept for Internet audio streams that use my phone battery and limited gb plan.

I also listen to too many hours annually of audio books to accept files greater than 24 kps bitrate.  (It amazes me that people don't grasp the potential of this bitrate.)

I am wondering if Opus plans to improve its sub 24 kps performance. And, if Exhale will fix its voice for low bitrate for xHE-AAC encoding.

I am currently using ab-cable audio driver and ffmpeg.  I edit the artist, album, record duration variables. Click the bat file to record voice to listen to while working.  I am recording voice opus files, trying to target 21 kps. I might start recording to 64 kps opus then use ez cd audio converter to convert the 20 hour files to 18 kps xHE-AAC. 

I have noticed a grain noise on peaks with Opus at 21 kps, but wasn't sure if I was hallucinating or had a bad source, or was a speaker issue.  I normally use a 20 hour mono phone bluetooth to listen to the voice files, and probably wouldn't be able to hear the grain.  But, if I record something really good, I will want to share it with my brother who is in the broadcasting industry and is easily turned off by such things on his thousand dollar speakers.

As excited as I am by the huge potential of xHE-AAC and Opus at sub 24 kps for streams and voice, I will probably stick to 192 kps lame mp3 for long term music that I will be listening to years from now. Because, there are cars that support mp3 , but not a whole lot more.  I have never not enjoyed a good song on a good 192 kps vbr warm lame mp3 file, although I don't find it transparent in the least.

Even at these low bitrates, we are still talking several hundred of megs per audio book.

A note for the Original Post, I totally agree that 13 kps or 14 kps should have been tested. 

I just never was able to get any book listenable at 12 kps using lame 3.97 and some extraordinary switches and tricks.   However, I was able to get very listenable vbrs at a range of 14 kps to up to 23 kps, letting lame choose, depending on the speed of the speaker and voice tone. ...  Thus, I cannot imagine any codec going below 14 kps, because the extra 2 kps made all the difference in the world on the quality over a 12 kps file audio book.

It is because of my lame audio book days that I would be skidish going below 17 kps, which was the bitrate if you averaged all the books together that I have encoded and listened to since 2009.  

I have just switched to opus, but it sounds like xHE-AAC is a wiser choice for sound quality in the sub 24 kps range, future compatibility, and battery, since while opus uses less cpu, the phone will use less battery transcoding the opus to xHE-AAC to send to the bluetooth.

Opus is just less of a pain to directly record to than FhG xHE-AAC.  Hypothetically, I could buy the ffmpeg plugin from MainConcept.com for like $79. But as a hobby, $29 is probably my spending limit on such a luxury.

The other thing to mention is that ffmpeg allows a lowpass switch for Opus, and I really could never hear, in the human voice, much to write home about above something like 10.3khz, if I recall. I also had lots of spectral tools back then, since I did lp restoration semi professionally back then.  Opus wisely lowpasses at 12k for voice, auto detection (which would work for non acted out stuff or mixed content) at the low bitrates.  But perhaps the opus 12k lowpass isn't aggressive enough, and we should try to add the ffmpeg opus switch at a 10.3 khz lowpass, in order to make room to eliminate the grain, etc. ?  (With lame, I chose 7.8khz low pass, which worked fine for a single earbud and an audio book. 10.3khz low pass is what I used for a 44 kps targeted mono mp3, as I recall when programing my scripts.)