Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music) (Read 27267 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Introduction

I made several listening tests in my life and at various bitrates (from 48 kbps lately to 190 kbps many years ago). But I never thought I would make one at 12 kbps.

Just to put this little value into perspective: at 12 kbps one hour of music only takes 5.40 Mb—which is usually the size of a single track at 128 kbps. Hotel California full album: 3.97 Mb. Back in Black: 3.82 Mb. Thriller: 3.86 Mb. One full album is half the space needed by a JPG shot done with my cheap smartphone! For older people who knew CD-R burning: 130 hours of music could be stored on a 700 Mb CD-R… Every file is insanely small! And for streaming purpose, EDGE poor network is still enough to stream music without issue (in a tunnel, on mountain…).


But why doing this test?
First, to get some comparison data at unexplored regions and fill some gaps. I am really curious to check how modern audio codecs are dealing with such huge bitrate starvation. When a new video or pictures format appears it’s immediately tested at ultra low bitrate—but audio isn’t. If you take a look on the next graph, you won’t see any popular format below 40 kbps:

(credit: opus-codec.org)

Now that a modern opponent to OPUS is easily available (Exhale first, now Fraunhofer's encoder with EZ Audio Converter) an interesting duel is at least possible.


Second reason: For the sake of curiosity I recently forced myself to listen to ~35 kbps xHE-AAC on my phone, streamed over bluetooth in my car or earbuds and the result was far from disaster. In fact, it was a bit better than acceptable. I even listened to these ultra-low bitrate encoded albums many times during two weeks. This was a real big surprise. Because I’m not fond of low bitrates: 20 years ago I discovered --r3mix, then became a LAME --alt-presets pilgrim and finally turned into a MPC priest. I spent afternoons and weeks on ABX softwares just to get significant scores on castanets samples… It would be an understatement to say that I was totally neurotic about sound quality. Fortunately I’m clearly more tolerant today. So these low bitrate performances are something I wanted to investigate, for practical usage. If I can put on my phone a lot of stuff I rarely listen to without ruining the storage capacity it’s definitely something I should try (it’s less expensive than streaming subscription and even much more reliable—I often get off cellular networks while wandering in the forest or in nature).
Low bitrate encodings is also a very reasonable choice for audiobooks and even movies ripping (Netflix made recently the switch to xHE-AAC to improve quality and efficiency). That’s why 20 samples of voice are included here.



Challengers: The choice of formats/encoder is limited to the most recent ones: OPUS (2012) and USAC/xHE-AAC (2012?). Both are unified formats for voice and music coding. As Exhale open-source xHE-AAC doesn’t handle such low bitrate, only Fraunhofer’s xHE-AAC (included in EZ-CDA Converter) encoder is able to compete here.
Opus : last version on official website (1.31, executable). VBR is used (auto detection between voice and music).
xHE-AAC : version by Fraunhofer included in EZ CD Audio Converter 9.3.1. The Fraunhofer (FhG) encoder was updated with 9.3.2—but I missed the update so it’s not tested here. I used VBR preset 0 (24 kbps) and CBR for 12 and 32 kbps.

BITRATE: Average bitrate for a large CD collection

Opus 12Opus 24Opus 32FhG 12FhG VBR 0FhG 32
12,0 kbps25,7 kbps34,0 kbps12,0 kbps24,0 kbps32,0 kbps
(+0 %)(+7.8 %)(+6.25 %)(+0 %)(+0 %)(+0 %)
VBRVBRVBRCBRVBRCBR
The full bitrate table is available here.

SAMPLES: 60 samples in 6 groups:

music: billboard 2010-2020 greatest hits. Standard range for each: 01.00.000 to 01.20.000. 10 samples, best-sellings music. Difficulty: average.
music: classical: 10 musical samples from a larger collection. Classical music. Criteria selection: favorite moments. Difficulty: below average.
music: HA.io: 10 samples submitted by Hydrogenaudio’s members. Many of them were chosen because they revealed some issues with encoders of the past (usually LAME MP3). Difficulty: above average.
music: problem: 10 samples shared on HA.io that revealed strong encoding issues with encoders of the past. Mostly issues like pre-echo or additional noise on transients. Difficulty: hard to very hard
voice: audiobooks: 12 samples from audiobooks. 6 male, 6 female. 8 different languages. Criteria selection: random or favorite part. Difficulty: easy.
voice-movies: stereo samples of famous soundtrack (dialog, with or without music and/or ambient noise). 48 KHz, 16 and 24 bit. Criteria selection: cult dialog, favorite moment, or random. Difficulty: easy.

All samples are available here:
https://www.dropbox.com/sh/zjephy3g54j4gur/AACjGhM9tabl26n7s4ihYl2Ra?dl=0

SOFTWARE & HARDWARE:  I used Java ABC/HR. All files were normalised. Two first seconds of each encoding were discarded from evaluation. An AKG Q701 was plugged on the headphone jack of my computer (I was unable to run my DAC on ABC/HR).



RESULTS:


On the graph I put the overall score for all 60 samples on the middle of each column, the result for the 40 musical samples on the left and the result of the 20 samples of voice on the right.


  • At 12 kbps Opus is totally lost on music; it performs significantly better on voice but quality is not truly convincing. The overall result is very poor: sound is muffled, mono-ish, and there is often a bad amount of noise/grain. xHE-AAC sounds differently but much better: lowpass is not as strong, stereo is here but often weird (like out of phase). There’s a hollow and synthetic sound image on many samples. Music is not good to my taste but it’s not catastrophic either. Voice samples (especially movies) are almost acceptable.
    ► 12 kbps has nothing enjoyable but it sounds much better with xHE-AAC than with Opus.
  • At 24 kbps Opus is making a big step forward. Sound is not muffled anymore and the stereo image is quite excellent. The problem is a layer of  strong and harsh sibilant artifacts that didn’t exist at 12 kbps. In many cases there’s still an aggressive amount of grain/noise. Music at 24 kbps is still poor (score < 2.0); audiobooks and movies are now acceptable though irritating (score < 2.50). xHE-AAC has much less issue with stereo here and sound is much less synthetic. Music is not stellar but is quite acceptable now (score : 2.77). And voice has minor issues and is good (3.43 on average, and even 3.70 for movies! Hello Netflix!).
    ► 24 kbps is now usable with xHE-AAC (average score: 2.99) and is quite enjoyable with voice recording. OPUS is not really usable on music and its score is still lower than USAC CBR 12 kbps! On voice OPUS 24 kbps VBR reaches the same score than USAC CBR 12. In other words OPUS is far less efficient than xHE-AAC at 12 and 24 kbps.
  • At 32 kbps Opus is still not very pleasing with music (score: 2.38) but on voice it’s now more than simply usable (3.19). xHE-AAC makes quite a good job on music (3.21) and is almost stellar on voice (especially movies: score = 4.0, hello again Netflix).
    ► xHE-AAC outperforms OPUS at 32 kbps (in fact 34 kbps) and really plays in a different league.
  • 96 kbps USAC (here with Exhale) is excellent in comparison and in absolute terms (probably a bit overrated here as it's included as high anchor)

To finish I just want to remind that this evaluation is personal. OPUS and xHE-AAC have very different kind of distortions and judging one less annoying to a different one if very subjective :)

As always thanks to kamedo2 for its useful tool!

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #1
Great test guruboolez!
That Fraunhofer encoder is very impressive.
gold plated toslink fan

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #2
More proof that mp3 is a dead format.

Thanks,
guruboolez

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #3
More proof that mp3 is a dead format.

Not necessarily dead...
Most audio podcasts are still distributed as mp3s and, say, what the TWiT network uses, I believe mono 64 kbps mp3 is perfectly acceptable to my ears for voice.

Sure, there probably are better options for quality in the range below ~128 kbps or so but it's certainly not "dead" as there are still many mp3s being made and they still can be played and probably will be able to for the forseeable future.


Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #5
More proof that mp3 is a dead format.


lol. not.

For example, the website Bandcamp offers mp3s or FLAC's. And there are others....like Presto for classical music....and then there's this small retailer known as Amazon.

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #6
@guruboolez
Excellent work!
This is very interesting test to see the ultimate performance in this extremely low bitrate range. Fraunhofer xHE AAC is definitely very impressive.
From storage affordability point, I personally can't see the reason to go this low in 2021. for local music listening, even for Netflix or other streaming video service it cannot bring too much size reduction because video bitrate is still high.
However, performance and results are still very interesting to see on such low bitrate.
lame --abr 288 -f --lowpass 17 (+ mp3gain@92 dB)

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #7
FhG xHE-AAC did 3.41 on 32kbps? That's amazing!

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #8
FhG xHE-AAC did 3.41 on 32kbps? That's amazing!
3.41 but on merged results. For music samples score is "only" 3.21 (voice is easier to handle).
As xHE-AAC 32 kbps was the very best encoder of a group of six I would say that the score is *maybe* slightly overestimated. In different testing condition (eg a listening test with 32 - 64 - 96 kbps competitors) I expect a lower score. Same if I put xHE-AAC 32 kbps as low anchor in a 128 kbps test.

But if I spend one hour at home at home on listening music at 32 kbps xHE-AAC I can give a rough estimation of the score at ~3.0 (which literally means "slightly annoying" on the ABC/HR scale). It's listenable with audible but limited annoyance.
Again, also, it's a matter of taste and tolerance.


Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #9
Did you use the Graphical User Interface of the EZ CD Audio Converter 9.3.1?
If not, could you please share the command line you tested on?

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #10
Yes I used EZ CDA Converter through the GUI (does it work on command line too?)

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #11
From storage affordability point, I personally can't see the reason to go this low in 2021. for local music listening, even for Netflix or other streaming video service it cannot bring too much size reduction because video bitrate is still high.
Well, it makes sense for Netflix and movies when there is 5.1 & 7.1 surround sound options ;)

-

@guruboolez - as others have said, thank you for this. And again, it really is incredible to see that modern audio can achieve such decent scores from such low bitrates. I have noticed this myself when listening to Mixcloud for example, that streams at 64kpbs with AAC-LC (I'm not 100% sure of the encoder although MediaInfo & foobar report the "writing application" / tool as Lavf57.83.100).



Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #13
Extremely interesting test, thanks a bunch guru!

For example, the website Bandcamp offers mp3s or FLAC's.
Bandcamp offers AAC, Ogg Vorbis and ALAC too.
WavPack 5.6.0 -b384hx6cmv / qaac64 2.80 -V 100

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #14
And I've experienced the case where BandCamp had much larger FLAC files than Apple Lossless, because the FLACs were 48000/24, while the ALAC was 48000/16.

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #15
From storage affordability point, I personally can't see the reason to go this low in 2021. for local music listening, even for Netflix or other streaming video service it cannot bring too much size reduction because video bitrate is still high.
Well, it makes sense for Netflix and movies when there is 5.1 & 7.1 surround sound options ;)
You've got a point. I forgot about multichannel sound. :D

@guruboolez
Considering that 32k gives around 3.2-3,4 score, do you think that FhG xHE at 48k could go to 3,5 or maybe 3,8-4.0 quality score?
I think that at 48k (FhG xHE) should be descent for casual listening (especially for untrained ears) :)
lame --abr 288 -f --lowpass 17 (+ mp3gain@92 dB)

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #16
Considering that 32k gives around 3.2-3,4 score, do you think that FhG xHE at 48k could go to 3,5 or maybe 3,8-4.0 quality score?

Score must be higher, yes—but how exactly? I can't tell.

I see three limitations here:

► SBR is a great tool at very low bitrate but introduce several flaws. The benefit is quickly fading when bitrate increase.

► Beyond 32 kbps FhG doesn't use Parametric Stereo anymore. For that reason ≥40 Kbps should have a more stereo bitrate-expensive strategy. Stereo image must be better but maybe, I don't know, to the expense of other issues.

► the ABC/HR scale is not very wide. If I had tested 48 Kbps in the same group along with 32 kbps (and 24, 12 and 96 kbps anchor), I maybe had to revamp the scores and compress a little 24 and 32 kbps sliders to the down. Not by much though.


3.4…3.5 for music samples is likely with the tested samples (which I remind are not all difficult ones).

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #17
Interesting read. I presume that your "Difficulty" ratings are the obvious thing that killer samples are difficult to encode right? Or is it even so that killer samples are difficult for you to rate because all do them obviously wrong?

(probably a bit overrated here as it's included as high anchor)
When these listening tests came about back in the late neolithic, you wouldn't have expected a sub-100 to be the high anchor for stereo music outside April 1st.
I am impressed.

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #18
To illustrate OPUS vs FhG duel at 12 kbps and have a better look on their respective flaws (muffled and narrow sound + agressive layer of noise on some parts for Opus, weird stereo & hollow sound for FhG USAC), here is a small selection:
https://www118.zippyshare.com/v/0KJq3kVP/file.html

and for people who don't have any xHE-AAC player/decoder, the same selection but decoded to FLAC:
https://www29.zippyshare.com/v/FWpYRX5q/file.html


It's a biased selection: all samples were badly ranked for OPUS 12 kbps. But it may illustrate the gap between the two formats at ultra-low bitrate on music material.


Interesting read. I presume that your "Difficulty" ratings are the obvious thing that killer samples are difficult to encode right? Or is it even so that killer samples are difficult for you to rate because all do them obviously wrong?
Yes, those files in group #4 are known difficult samples for at least old lossy encoders (some samples are or were real stars here 20 years ago: castanets.wav, fatboy.wav, enola.wav — happy anniversary Hydrogenaudio!…)

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #19

► Beyond 32 kbps FhG doesn't use Parametric Stereo anymore. For that reason ≥40 Kbps should have a more stereo bitrate-expensive strategy. Stereo image must be better but maybe, I don't know, to the expense of other issues.
I didn't know that. Thanks for explanation. FhG xHE will be developed further, I suppose, so quality should increase in the next version or in the near future.
lame --abr 288 -f --lowpass 17 (+ mp3gain@92 dB)

 

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #20
Here is a pack of all 60 samples encoded in both OPUS 12 kbps VBR and Fraunhofer's xHE-AAC CBR 12 kbps. So everyone could take a look on how modern encoders sound at this very small bitrate — compression ratio is 1:117 for RedBook and 1:125 for 48KHz voice samples!).

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #21
Thank you guruboolez.

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #22
Nice test, Guru.

It would be even greater if HE-AACv2 encoder was there (and here is  a very high quality FhG Winamp HEv2 VBR encoder).

Speaking of Netflix and low bitrates, an average speed of streaming right now is 3300-3400 kbps. https://ispspeedindex.netflix.net/global.  So xHE-AAC is useful for mobile connections  not so much for an average home wifi connection where Netflix still uses E-AC3 and AAC 128-768 kbps.
Also it's seems like Netflix concentrates more on DRC feature of xHE-AAC https://netflixtechblog.com/optimizing-the-aural-experience-on-android-devices-with-xhe-aac-c27714292a33


Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #23
Why does it need to be the Winamp encoder? Isn't that kind of old by now, compared to the HEv2 encoder in EZ CD Converter?

Re: A Session In The Abyss: xHE-AAC vs OPUS at 12, 24 and 32 kbps (voice & music)

Reply #24
Because HE-AACv2 is old.
Well, at this point xHE-AAC and Opus are old too (9 years).

Nero has reached a limits what can offer HEv2 more than 10 years ago.
At this point trying something new is like saying "Why try LAME? Try some new MP3 encoder" huh.

All encoders HEv2, v1, LC-AAC, MP3, Vorbis (except Opus and USAC) have reached max quality before 2010-2011.  Since then only some fancy noises about new FDK AAC and some other AAC encoders claiming superiority and comparable quality (which wasn't close to reality by any means!) over Apple AAC (this last one  hasn't seen any meaningful update already for 16 years  (LC) and 12 years for (HE)).