Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3 (Read 28060 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

This is a personal listening test, and my plan was to base it on classical music only. Targeted bitrate is 64 kbps VBR and the test includes the two most modern & efficient formats at this bitrate I know: Opus and MPEG-D USAC, also named xHE-AAC.
This test must answer a few questions of mine:

  • To my ears which format sounds the best at ~64 kbps?
  • Is in 2020 any format at 64 kbps good enough for my own taste and my own music?
  • Does any format compete with MP3 at 128 kbps? Or, in another word, can the new formats be 50% more efficient at the selected bitrate?
  • Is the new implementation of USAC (named exhale) already competitive?

General principles
  • VBR only test: I’ll try to test all format with a VBR implementation.
  • 6 formats: I’ll limit the comparison to 6 formats/settings to limit the effort and the statistical noise.
  • Low and high anchors: my plan is to constraint comments and marks with low and high anchor. For this reason, the numbers of real competitors will shrink to four but I’ll try to make anchors as interesting as possible.
  • Number of samples: my purpose is to test as many samples as I can. Instead of testing 12 samples and ABXing each 8 times, I’d rather test 100 samples blindly on ABC/HR procedure.
  • Difficulty: I’ll try to avoid killer samples and rather use common music samples.

Samples
The samples are selected from two big boxset (55CD+29CD=84CD). They are both vast anthologies of classical music. The average FLAC bitrate for these 84 CD is 580 kbps, which is exactly the same that the average bitrate of my entire classical music library (it’s a big one). I add to this set seven other CD all with extreme FLAC bitrate (<300 kbps and >1000 kbps) which makes a basis of 91 CD.
The samples I extract from these discs are either famous hits of classical music or moments I really like (which is a big plus when I have to listen hundred times each sample). There is no over representation of killer or critical samples here but some samples may be sometimes hard to encode. I expect on the other side some samples to be fully transparent even at 64 kbps.

Bitrate
Average bitrate is not calculated from short samples which is in my opinion a bad idea (especially for VBR encodings). My bitrate table is consequently based on the encodings of the full 84 CD (2 boxset without extreme additional discs) which is representative of my FLAC entire collection. My guess is that this selection should also be very representative for other lossless and lossy encoders.

Competitors
I had to keep four contenders and to choose two anchors:
  • OPUS: Probably the real challenger at this bitrate. It’s free, Open-Source, native support on Android.  Excellent VBR adjustement. I choose to maintain the –bitrate 64 mode even if it exceeds the desired and targeted bitrate. TESTED: opus 1.31 –bitrate 64 . AVERAGE BITRATE: 69,0 kbps
  • USAC / xHE-AAC: One of the last MPEG audio tool and the last one in the AAC family. I asked to C.R.Helmrich the permission to use exhale for this test, because it is really a new tool and obviously can’t claim the same maturity level of all other competitors. It must be noted that exhale doesn’t include at this time SBR which is a really known tool for improving sound quality of both MP3 (MP3Pro) and AAC (HE-AAC) at 64 kbps. So, my guess is that exhale won’t probably shine at this bitrate but it will be interesting to see how it performs against older formats like LC-AAC, HE-AAC and MP3 and their most mature implementations. Exhale is open-source and VBR. TESTED: exhale 1.0.4 “E” -mode 1 with resampling done by SSRC (no 44.1 Hz support at this VBR setting with 1.0.4). AVERAGE BITRATE: 65,3 kbps
  • LC-AAC: It should be interesting to see how AAC without parametric techniques (SBR & Parametric Stereo) performs to USAC deprived of the same toys. For this test I chose iTunes/Apple AAC which has a VBR mode that end near 64 kbps. TESTED: qaac 2.67, CoreAudioToolbox 7.10.9.0, AAC-LC Encoder, TVBR q18, Quality 96 . AVERAGE BITRATE: 62,9 kbps
  • HE-AAC: High Efficiency profile brings SBR technique to AAC, and it makes a real difference at 64 kbps.  I want to measure the benefits of this technique. As my goal is to compare VBR encoders I choose FDK-AAC which allows VBR with HE-profile (iTunes AAC doesn’t). TESTED: fdkaac 1.0.0, libfdk-aac 4.0.0, VBR mode 2 . AVERAGE BITRATE: 69,3 kbps
  • MP3: MP3 at 64 kbps could sound as a joke. Therefore, I used it as low anchor. As all other encoders I choose to keep the VBR mode. Unlike true competitors I finely adjust VBR mode to be the closest possible to 64 kbps. I also decided to use MP3 for high anchor but at 128 kbps. Same principle: it will be VBR and adjusted to be as close as possible to 128 kbps. Low anchor is LAME 3.100 at -V 8,85 (63,7 kbps) and high anchor is LAME 3.100 at -V 4,38 (127,8 kbps)


Tools
Java ABC/HR tool was my tool of choice mainly because it corrects the small volume difference between encoded files. I fed the ABCHR tool with WAV file: I decoded and resampled eveything at 48000 Hz to avoid playback issue (low anchor is 22050 Hz, USAC is 32000 Hz, Opus 48000 Hz and AAC is 44100 Hz). For this step I used foobar2000 and SSRC/dBpoweramp resampling component.
 

RESULTS

(A bit messed, sorry kamedo)

CONCLUSION
  • The winner of this test is a huge surprise. USAC is supposed to sound good in theory but how could a new implementation sound so good? It’s impressive! The encoder doesn’t have the quality issues HE-AAC and OPUS often have. There’s some preecho from time to time, some minor distortions common to many low/mid bitrate encoders. Nothing unusal I would say and on most sample nothing really irritating. Output is clean with no weird effects that often Opus and HE-AAC have. Average notation is close to the “not annoying” step but there are some accidents. Exhale performed also worse on the second boxset which is more baroque or early instruments oriented (commonly harder to encode). Exhale is therefore quite good on average but at 64 kbps it’s still not strong enough to be satisfying for a peace-of-mind daily usage. Anyway, I never expected such high quality at 64 kbps.
  • Opus is not bad but is clearly worse than USAC and doesn’t even beat (from statistical analysis point of view) HE-AAC. On many samples the artifacts generated by OPUS irritates me a lot: fat sound, coarseness, hiss—especially on tonal parts. There are also more basic distortions and some smearing too. Several time I found some dullness which I couldn’t understand (lowpass is way too high to hear it) but at the very end of the test I understand that it may be explained by a kind of stereo narrowing. In other words, Opus has a clear sound signature and I wouldn’t call it a good one. Funny thing: I immediately recognized the Ogg Vorbis sound signature I often heard and described in my test a decade ago. Clearly Opus and Vorbis share the same DNA. Anyway, I expected a much better performance here.
  • LC-AAC performed quite poorly. I’m not really surprised here. But I frankly expected a better performance from iTunes AAC which I always liked, especially on “easy” tracks (like simple piano samples). Perhaps is it a TVBR issue: bitrate drops really low and that may affect sound quality. I haven’t really investigated further because I won’t use LC-AAC at 64 kbps. Lowpass was a biggest issue and the next VBR step would partially solve this without inflating too much the bitrate. If I had to do the test again I would perhaps go for testing LC-AAC at 70 kbps rather than 63 kbps. So, there’s a poor sound quality for the AAC core (without SBR) which makes in comparison Exhale core (without SBR too) really extraordinary.
  • HE-AAC is a real and big improvement over LC-AAC but I must immediately add that SBR is far from a panacea. SBR brings indeed very often the same sound issues (a lot of grain/harsh noise + some metallic color). It’s nothing new: this sound signature was already audible when MP3Pro was released. If twenty years ago this defect is still here I guess nothing will change in the future. So my bet is that SBR won’t bring anything except damage to USAC at 64 kbps which is probably too good for SBR.
  • Low anchor was clearly low quality enough. I must insist that I often had some troubles to rank MP3@64 and AAC@64. While they sound very different, MP3 has often less irritating distortions but is much more muffled. It’s like choosing between a highly macroblocked jpeg file and a 64 colors GIF. Problems are both huge but very different, and rating is sometimes a simple matter of taste. For the anecdote, MP3 at 64 kbps could sometimes be transparent (on low pitched piano recordings) and on one or two samples I ranked it better than Opus and USAC!
  • High Anchor is LAME at VBR 4,38. This setting gives me 128 kbps on classical music but I must warn people that I’m pretty sure bitrate will be higher on other musical genres with this setting. But here, and to my ears, MP3 at 128 kbps VBR outperforms every competitor… at half bitrate. On 75 samples there were only two encoding accidents (an organ and a harpsichord samples). LAME performs excellently here and still shine at least when compared to low bitrate competitors.
  • Last but not least, I must really insist that the sound signature of four encoders (low anchor, LC-AAC, HE-AAC, OPUS and to some extend Exhale) was really obvious and it was really easy to guess them. I often wrote my own guess in the ABCHR comment tool and I was nearly always right. Is it important? Yes maybe, because I knew what I rated. It’s a bit like blindly test apples and… fish

Some reflections…

Now my question is: am I different to other people to rank Opus so low? Or is simply Opus not as efficient on classical music?
Tonal issues on Opus have already been reported (by Jean-Marc Valin itself on this board). And classical music is often tonal. I guess there are also other sound characteristics that makes classical behave differently with lossy encoders. Volume might play a role: classical recordings don’t usually suffer from so-called loudness war. I remember reading somewhere here that some opus encoding tools can be troubled because it can’t guess at what volume playback is (someone could explain this better or correct me). Indeed, the Opus hiss/fat sound is not as irritating at quieter playback volume.
But the alternative to these explanations is a different subjectivity.
APPENDIX TEST: CONTEMPORY MUSIC

To answer this, I decided to not test further classical music (yes, my original plan was to do a second batch of classical samples…) but to test all these codecs again with modern music. It’s an appendix to this test because it’s not as rigorous (I can’t get a large CD collection nor build any solid bitrate table). My musical culture is nearly inexistent, so I visited Billboard.com website, and I took one of their playlist called “decade-end charts: hot 100 songs 2010s”. I managed to get a lossless copy of the 25 first titles of this chart. For each song, I create 20 seconds sample from selecting each time the music from the same range  (00h01m00s to 00h01m20s). It can’t be less biased here. Again, there shouldn’t be any over representation of killer sample but only common music.
https://www.billboard.com/charts/decade-end/hot-100
I tested the same encoders: lame, exhale, opus, FDK-AAC and iTunes AAC. But I lowered the VBR settings for both low and high anchor (V9 which seems to be a bit higher than 70 kbps; V5 which is close to 130 kbps). I also boost iTunes LC-AAC quality by using TVBR27 instead of TVBR18 (bitrate is now much closer to 64 kbps). Other competitors haven’t changed. Methodology is the same.

Results of additional test (25 samples from a popular Billboard list)

Conclusion of this additional test
Results are significantly different with a totally different kind of samples. Opus sounds as the most convincing encoder—and while it’s still statistically tied, its average ranking is clearly superior to MP3 VBR at 130 kbps! Sound quality is really great and there’s no more coarseness or fat signature (or it’s too low to be noticed). I only get one failure on 25 files (but for this one every encoders were also bad). Exhale USAC is a bit less impressive here. It’s tied with both Opus and MP3@130 which is again remarkable for a such new-born encoder. But it has some more accident so I would clearly prefer Opus which seems safer. HE-AAC is a big step lower and thanks to SBR artifacts not very far from LC-AAC. The latter performed better than expected now that lowpass has been raised and the audible distortions were not as bad as with classical music samples. An educated guess would be that LAME at -V5 is the best quality you may get with MP3 format at around 128 kbps. And on these 25 non-discriminative samples our high anchor didn’t shine at all: many failures, several strong distortions.
I also must add few things: on this additional test I was unable to guess the encoders name. HE-AAC had maybe the strongest sound signature but the other don’t. I was truly lost and the few guess I did were usually wrong. Many samples from this billboard list were also very disturbing from a technical point of vue: they indeed sound distorted. I don’t know what they’re doing during the mastering stage. Voices are sometimes curious: is it an Auto-Tune effect? As a consequence I had sometimes a lot of trouble to find the encoded version as the reference already sounds so distorted.


Final words
My original plan wasn’t to oppose classical music to popular modern recordings. But it was instructive. Opus isn’t the killer-tool I expected when I first started this testing procedure but I must admit that it really shines with louder and modern music—and 64 kbps encoding is really satisfying for such music. USAC brings a very interesting alternative to OPUS at low/mid bitrate: sometimes better, sometimes worse (and clearly better with classical) but much cleaner. I don’t know what USAC offers as low encoding tools but compared to SBR and Opus algorithm (is it “band folding”?) sound is way more proper. SBR technique seems to be more damaging: I dislike it and HE-AAC is clearly worse on pop music.

To answer to my first questions:
  • To my ears which format sounds the best at ~64 kbps?
=> it depends. Exhale is better for classical; Opus seems to have my preference for other music.
  • Is in 2020 any format at 64 kbps good enough for my own taste and my own music?
=> again, it depends. Opus is really fine at 64 kbps but absolutely not usable for my classical library. Exhale/USAC is interesting with classical but I would appreciate some improvements.
  • Does any format compete with MP3 at 128 kbps? Or, in another word, can the new formats be 50% more efficient at the selected bitrate?
=> MP3 is stronger on classical music; but seems to be beaten by 64 kbps contenders with pop music
  • Is the new implementation of USAC (named exhale) already competitive?
=> Yes, it performs better than many mature encoders of other formats. Congratulations!


Detailed tables

(clic to enlarge)






PS: If asked I can upload samples later


Thanks
A big thank you to:
  • Kamedo2 for the amazing graph tool: https://listening-test.coresv.net/graphmaker5.htm
  • Christian R. Helmrich for allowing me to test Exhale “E” 1.0.4
  • Peter Pawlowski for foobar2000
  • Schnofler for his java ABC/HR tool
  • Christopher “Kode54” Snowhill for the USAC component
  • Jean-Marc Valin for his work on Opus
  • John33, Netranger and other members providing binaries

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #1
Nice, Guru. Thanks!

I'll try to digest all of this at some point. 
First observation that comes to mind is that both FhG (Winamp 5.666) and Apple HE-AAC encoders  should/could be somewhat superior to FDK HE-AAC encoder (at least in my experience). However FDK is popular choice for many people.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #2
Congratulations, exhale, congratulations, Guru!



Code: [Select]
% Sample data.(ff123's FRIEDMAN version 1.24 compatible)

MP3 LC-AAC HE-AAC OPUS USAC / xHE-AAC (MP3)
%feature 10 LAME 3.100 iTunes,CoreAudioToolbox 7.10.9.0 libfdk-aac 4.0.0 Opus 1.3.1 exhale 1.0.4 LAME 3.100
%feature 11 -V 8,85 TVBR q18 Quality 96 VBR mode 2 –bitrate 64 -mode 1 -V 4,38
%feature 12 63.7kbps 62.9kbps 69.3kbps 69.0kbps 65.3kbps 127.8kbps
1.3 2.0 2.5 4.0 4.5 4.8
1.5 2.0 3.0 3.8 3.7 4.9
3.0 2.0 4.7 4.0 4.0 5.0
1.5 2.0 3.0 4.2 3.2 4.9
2.0 1.5 4.0 4.7 4.0 5.0
1.3 1.5 2.5 2.7 3.8 4.4
1.0 2.0 2.7 3.0 3.7 4.2
3.0 2.5 3.7 4.0 4.0 5.0
1.5 3.0 3.0 4.0 4.0 5.0
1.0 1.6 2.5 3.0 4.5 4.7
2.5 3.5 3.0 5.0 5.0 5.0
1.0 1.5 3.0 4.8 3.8 4.5
1.0 1.6 3.7 2.5 4.0 5.0
2.0 2.5 4.6 4.0 3.5 5.0
1.3 1.7 2.2 2.5 3.2 3.5
1.0 1.5 2.5 2.0 4.0 4.5
1.0 1.5 3.0 1.8 3.7 4.7
5.0 3.2 5.0 3.7 4.0 5.0
1.0 2.5 3.5 2.0 4.8 4.0
4.0 3.0 5.0 5.0 5.0 5.0
1.3 1.0 3.0 2.5 4.7 4.0
1.0 1.5 3.0 2.0 5.0 3.5
1.5 2.0 4.5 4.0 4.0 4.5
1.5 1.8 4.0 3.7 5.0 5.0
1.0 2.0 4.5 5.0 3.7 5.0
1.0 2.0 4.5 3.0 3.7 5.0
2.5 4.0 5.0 5.0 5.0 5.0
5.0 5.0 5.0 5.0 5.0 5.0
1.0 1.3 2.3 2.0 4.6 5.0
1.5 2.0 2.0 2.5 4.5 5.0
2.5 3.5 5.0 5.0 5.0 5.0
1.0 1.5 1.8 4.4 2.5 4.0
1.5 2.3 3.5 4.5 3.5 5.0
1.5 3.0 2.0 5.0 3.8 5.0
1.0 2.2 2.7 3.8 3.4 3.8
2.0 2.5 3.5 5.0 5.0 5.0
1.0 1.8 2.2 2.5 2.8 5.0
2.5 2.5 4.0 3.5 4.0 5.0
1.0 1.4 2.3 1.8 3.5 4.0
3.5 2.0 2.0 4.0 5.0 5.0
1.5 2.0 3.5 2.5 4.5 4.5
1.0 1.5 2.3 2.0 3.8 3.3
2.5 4.0 4.8 3.5 4.7 5.0
2.0 2.2 2.6 3.7 3.0 5.0
2.0 3.4 1.0 2.5 4.5 4.5
1.0 2.5 2.0 1.5 4.0 5.0
2.0 2.5 3.5 5.0 5.0 4.5
1.0 1.5 2.2 2.2 5.0 4.7

2.0 1.7 2.0 2.0 3.0 1.8
1.8 2.1 3.0 4.0 2.5 4.7
1.0 3.0 3.5 3.8 3.0 4.6
1.2 1.7 3.7 3.0 2.5 5.0
1.0 1.7 2.3 1.4 2.6 4.0
1.0 1.3 4.0 2.0 2.5 4.5
1.0 2.0 2.3 2.7 3.5 3.9
1.5 3.0 3.7 4.5 4.0 4.2
1.0 1.4 2.0 2.3 4.5 3.7
1.0 4.0 3.3 3.0 2.7 5.0
1.0 1.8 5.0 2.5 2.7 5.0
1.0 1.0 2.7 3.2 3.7 5.0
1.0 1.5 2.5 2.5 5.0 5.0
1.0 2.0 1.8 2.5 5.0 5.0
1.3 1.5 2.0 2.0 3.8 4.5
1.0 1.8 2.3 2.7 4.4 5.0
3.0 2.5 3.9 4.5 4.2 4.8
1.5 1.0 2.5 2.5 2.8 4.2
1.0 2.9 2.0 2.3 4.0 3.2
5.0 3.5 4.5 5.0 4.0 5.0
1.0 1.7 2.0 2.3 5.0 5.0
2.0 3.0 5.0 4.5 4.0 5.0

5.0 5.0 5.0 5.0 5.0 5.0
3.5 3.0 5.0 4.5 3.8 5.0

1.0 1.0 2.0 1.9 2.5 2.2
1.0 1.0 2.2 2.2 3.5 3.0
1.0 1.5 2.2 2.0 3.0 4.2

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #3
Nice, Guru. Thanks!

I'll try to digest all of this at some point. 
First observation that comes to mind is that both FhG (Winamp 5.666) and Apple HE-AAC encoders  should/could be somewhat superior to FDK HE-AAC encoder (at least in my experience). However FDK is popular choice for many people.

I also hesitate to use FHGAAC instead of FDKAAC. Main issue was the average bitrate. Winamp's AAC encoder only gives the possibility of testing a VBR step which ends at 59 kbps or the next which ends at 90 kbps. I found 59 kbps a bit low, especially if OPUS is used at an average bitrate of 69 kbps. So FDKAAC with its 69 kbps seemed to be less problematic. And as you said, it's also more popular, and is also cross-platform, easier to find and not as outdated as Winamp.
Apple's AAC encoder doesn't allow VBR with HEAAC. So I discarded it.
I haven't include Nero in my bitrate table but I think it do VBR with HE-AAC. But development has ceased for ten years.

 
Congratulations, exhale, congratulations, Guru!

Thank you Kamedo2 :)
It's much better now but I can't edit my first post anymore :/
I also noticed a mistake in the last table: I'll redo it later.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #4
"Apple's AAC encoder doesn't allow VBR with HEAAC."
AAC-HE Encoder, CVBR 64kbps, Quality 96

anyway, a general understanding of the features of the exhale comparison gives. thx

by the way, this is a great reason to compare qaac tvbr lc/cvbr he, fdk lc/he/he2, faac tns/notns/pns0..10.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #5
Was USAC at 22Khz sampling rate? If so, how does Opus 22Khz compare to 22Khz USAC?

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #6
Here is the bitrate table (full CD for classical music only—the full table exceed the limit of 20.000 characters). The screenshot include the bitrate of the Billboard playlist.

Code: [Select]
                                                                                                              FLAC         MP3    LCAAC   HEAAC    OPUS    USAC     MP3
                                                                                   
                  BOX N°1                                               
A.01. Brahms [Claudio Abbado & Wiener Philharmoniker] 21 Ungarische Tanze (DG, C…      ORCHESTRAL             602 kbps      66      63      68      65      68      133
A.02. Beethoven [Amadeus SQ] Quartette Op. 59 No. 1, Op. 131 (DG, CD, 1960)…           CHAMBER                690 kbps      69      65      64      71      76      144
A.03. Chopin [Martha Argerich] 24 Preludes Op. 28 (DG, CD, 1975)…                      PIANO                  470 kbps      60      61      75      70      57      117
A.04. Ravel [Daniel Barenboim & Orchestre de Paris] Boléro, La Valse, etc (DG, C…      ORCHESTRAL             557 kbps      62      62      70      66      66      128
A.05. Debussy [Arturo Benedetti Michelangeli] Préludes, Vol. 1 (DG, CD, 1978)…         PIANO                  499 kbps      57      63      77      69      63      125
A.06. Bernstein [Leonard Bernstein] West Side Story (DG, CD, 1985)…                    LYRICAL                597 kbps      64      65      67      65      67      132
A.07. Mozart [Karl Böhm & Wiener Philharmoniker] Requiem (DG, CD, 1971)…               CHORAL & ORCHESTRAL    619 kbps      71      64      70      63      64      138
A.08. Stravinski [Pierre Boulez & Cleveland Orchestra] Petrouchka, The Rite of S…      ORCHESTRAL             559 kbps      61      65      72      67      64      126
A.09. Vivaldi [Giuliano Carmignola & Venice Baroque Orchestra] Concertos RV 331,…      CONCERTO VIOLIN        650 kbps      70      69      69      74      74      139
A.10. VA [Placido Domingo, Carlo Maria Giulini & LA Philharmonic Orchestra] Oper…      LYRICAL                568 kbps      65      61      64      69      67      125
A.11. Mahler [Gustavo Dudamel & Simon Bolivar Youth Orchestra] Symphony No. 5 (D…      ORCHESTRAL             498 kbps      58      62      69      66      58      115
A.12. Bach [Emerson SQ] Die Kunst der Fuge (DG, CD, 2003)…                             CHAMBER                613 kbps      70      68      70      72      68      135
A.13. Schubert [Dietrich Fischer-Dieskau & Gerald Moore] Winterreise (DG, CD, 19…      VOCAL                  532 kbps      64      69      84      69      69      133
A.14. Bach [Pierre Fournier] Sei Solo. a Violoncello senza Basso accompagnato (A…      CELLO                  670 kbps      65      56      70      70      65      130
A.15. Bach [Pierre Fournier] Sei Solo. a Violoncello senza Basso accompagnato (A…      CELLO                  667 kbps      64      53      69      70      66      127
A.16. Verdi [Ferenc Fricsay & RIAS Symphonie Berlin] Requiem (DG, CD, 1954)…           CHORAL & ORCH… (MONO)  459 kbps      40      39      45      55      49      83
A.17. Schumann [Wilhelm Furtwängler & Berliner Philharmoniker] Symphonie No. 4 (…      ORCHESTRAL (MONO)      479 kbps      40      37      42      58      50      82
A.18. Monteverdi [John Eliot Gardiner & English Baroque Soloists] Vespro della b…      CHORAL & ORCHESTRAL    534 kbps      63      68      72      64      62      126
A.19. Monteverdi [John Eliot Gardiner & English Baroque Soloists] Vespro della b…      CHORAL & ORCHESTRAL    522 kbps      63      66      71      69      65      127
A.20. Beethoven [Emil Gilels] Piano Sonaten - Waldstein, Les Adieux, Appassionat…      PIANO                  426 kbps      58      61      74      70      57      113
A.21. VA [Reinhard Goebel & Musica Antiqua Köln] Pachelbel, Handel, Vivaldi, Bac…      ORCHESTRAL             779 kbps      70      63      68      73      79      146
A.22. VA [Hélène Grimaud, Esa-Pekka Salonen & Swedish Radio Orchestra] Credo (DG…      CONCERTO               515 kbps      56      63      70      70      56      109
A.23. Bach [Hilary Hahn, Jeffrey Kahane & LA Chamber Orchestra] Violin Concertos…      CONCERTO VIOLIN        728 kbps      72      65      68      73      75      144
A.24. VA [Vladimir Horowitz] Horowitz in Moscow (DG, CD, 1986)…                        PIANO                  435 kbps      57      56      76      80      64      126
A.25. Orff [Eugen Jochum & Orchester Opern Berlin] Carmina Burana (DG, CD, 1968)…      CHORAL & ORCHESTRAL    650 kbps      66      64      77      65      67      138
A.26. Beethoven [Herbert von Karajan & Berliner Philharmoniker] Symphonie No. 9 …      ORCHESTRAL             685 kbps      70      64      70      65      66      142
A.27. Beethoven [Wilhelm Kempff, Ferdinand Leitner & Berliner Philharmoniker] Kl…      CONCERTO               590 kbps      71      61      73      69      62      135
A.28. Beethoven [Carlos Kleiber & Wiener Philharmoniker] Symphonien Nr. 5 & Nr. …      ORCHESTRAL             631 kbps      67      64      69      66      66      135
A.29. Haendel [Magdalena Kozena, Andrea Marcon & Venice Baroque Orchestra] Ah! m…      LYRICAL                548 kbps      63      64      73      68      66      125
A.30. Dvorák [Rafael Kubelik & Berliner Philharmoniker] Symphonies No.8 & 9 (DG,…      ORCHESTRAL             622 kbps      65      65      69      66      66      134
A.31. VA [Lang Lang, Daniel Barenboim, Chicago Symphony Orchestra] Tchaïkovsky, …      CONCERTO               502 kbps      63      62      70      69      58      116
A.32. Mendessohn [Lorin Maazel & Berliner Philharmoniker] Italienische, Reformat…      ORCHESTRAL             648 kbps      70      63      68      66      66      139
A.33. VA [Mischa Maisky & Semyon Bychkov] Adagio (DG, CD, 1992)…                       CHAMBER                518 kbps      61      61      70      68      66      121
A.34. Berlioz [Igor Markevitch & Orchestre Lamoureux] Symphonie Fantastique (DG,…      ORCHESTRAL             664 kbps      68      65      71      66      70      142
A.35. Praetorius [Paul McCreesh & Gabrieli Consort & Players] Christmette (Archi…      CHORAL & ORCHESTRAL    548 kbps      63      61      67      65      65      126
A.36. Rameau [Marc Minkowski & Les Musiciens du Louvre] Une Symphonie imaginaire…      ORCHESTRAL             656 kbps      67      63      72      67      67      135
A.37. Brahms [Anne-Sophie Mutter, Herbert von Karajan & Berliner Philharmoniker]…      CONCERTO               607 kbps      66      60      63      71      69      131
A.38. VA [Anna Netrebko, Gianandrea Noseda] Opera Arias (DG, CD, 2003)…                LYRICAL                567 kbps      66      68      75      68      67      130
A.39. Tchaikovsky, Wienawsky [David and Igor Oistrakh, Konwitschny] Violin Conce…      CONCERTO (MONO)        465 kbps      43      38      43      63      52      86
A.40. VA [Anne-Sofie von Otter, Reinhard Goebel & Musica Antiqua Köln] Lamenti (…      LYRICAL                539 kbps      61      66      73      69      67      123
A.41. Vivaldi [Trevor Pinnock & The English Concert] Le Quattro Stagioni (DG, CD…      CHAMBER                687 kbps      69      66      68      73      77      141
A.42. Chopin [Maria João Pirès] Nocturnes (DG, CD, 1996)…                              PIANO                  390 kbps      50      65      67      71      50      95
A.43. Chopin [Maria João Pirès] Nocturnes (DG, CD, 1996)…                              PIANO                  400 kbps      52      63      68      72      47      96
A.44. Scarlatti [Ivo Pogorelich] Sonaten (DG, CD, 1992)…                               PIANO                  378 kbps      56      65      74      74      53      104
A.45. Chopin [Maurizio Pollini] Etudes (DG, CD, 1972)…                                 PIANO                  443 kbps      62      60      70      71      55      115
A.46. VA [Thomas Quasthoff, Christian Thielemann & Orchester Oper Berlin] Die St…      VOCAL                  545 kbps      65      67      71      65      64      125
A.47. Bach [Karl Richter & Münchener Bach-Orchester] h-Moll Messe (Archiv, CD, 1…      CHORAL & ORCHESTRAL    657 kbps      72      66      68      67      66      140
A.48. Bach [Karl Richter & Münchener Bach-Orchester] h-Moll Messe (Archiv, CD, 1…      CHORAL & ORCHESTRAL    658 kbps      70      66      67      65      67      139
A.49. Rachmaninov [Sviatoslav Richter, Stanislaw Wisloki & Warsaw Orchestra] Pia…      CONCERTO ; PIANO       568 kbps      67      62      70      67      63      130
A.50. VA [Mstislav Rostropovitch, Herbert von Karajan & BPO] Dvorák, Tchaïkovski…      CONCERTO               611 kbps      65      63      72      68      70      136
A.51. VA [Bryn Terfel, Malcolm Martineau] The Vagabond (DG, CD, 1995)…                 VOCAL                  398 kbps      49      62      69      66      55      98
A.52. VA [Rolando Villazón, Daniele Callegari & Milano Orchestra] Cielo e Mar (D…      LYRICAL                619 kbps      68      64      66      68      68      128
A.53. Bach [Helmut Walcha] Toccata & Fugue. Organ Works (Archiv, CD, 1959)…            ORGAN                  730 kbps      75      65      65      79      69      151
A.54. VA [Fritz Wunderlich & Hubert Giesen] Schumann, Beethoven, Schubert (DG, C…      VOCAL                  613 kbps      68      64      73      68      67      135
A.55. Liszt [Krystian Zimerman, Seiji Ozawa & Boston Symphony] Piano Concertos (…      CONCERTO               521 kbps      63      62      69      69      62      121

                                                                                       AVERAGE (55CD)         570 kbps      63 kbps 62 kbps 69 kbps 68 kbps 64 kbps 126 kbps
               
  BOX N°2 (29CD)                                               
B.01. Couperin [Michel Chapuis, Francis Chapelet] Les Orgues Historiques (HM, CD…      ORGAN                  706 kbps      73      62      67      77      71      144
B.02. Muffat [René Saorgin] Apparatus Musico-Organisticus (HM, CD, 1972)…              ORGAN                  625 kbps      73      64      70      75      62      142
B.03. Purcell [Alfred Deller & Deller Consort] King Arthur (HM, CD, 1979)…             LYRICAL                698 kbps      70      65      66      67      71      146
B.04. Purcell [Alfred Deller & Deller Consort] King Arthur ; The Folksong Recita…      LYRICAL                653 kbps      67      70      80      68      70      142
B.05. Bingen & Anoymous [Various] Voices from the the Middle Ages (HM, CD, 1977……      CHORAL                 585 kbps      65      71      80      72      71      132
B.06. Dufay [Paul Van Nevel & Huelgas Ensemble] Motets isorythmiques (HM, CD, 20…      CHORAL                 626 kbps      70      68      72      73      72      137
B.07. Bach [Philippe Herreweghe & La Chapelle Royale] Matthäus-Passion (HM, CD, …      LYRICAL                584 kbps      68      69      71      67      65      136
B.08. Bach [Philippe Herreweghe & La Chapelle Royale] Matthäus-Passion (HM, CD, …      LYRICAL                547 kbps      66      70      73      67      65      131
B.09. Bach [Philippe Herreweghe & La Chapelle Royale] Matthäus-Passion ; Musikal…      LYRICAL                676 kbps      66      66      71      73      72      137
B.10. Bach [Fretwork] Die Kunst der Fuge BWV 1080 (HM, CD, 2002)…                      CHAMBER                658 kbps      67      62      68      75      72      138
B.11. Lully [William Christie & Les Arts Florissants] Atys (HM, CD, 1987)…             LYRICAL                616 kbps      64      61      66      66      71      133
B.12. Lully [William Christie & Les Arts Florissants] Atys (HM, CD, 1987)…             LYRICAL                587 kbps      63      61      68      67      70      129
B.13. Lully [William Christie & Les Arts Florissants] Atys (HM, CD, 1987)…             LYRICAL                711 kbps      65      65      68      71      77      140
B.14. Monteverdi, Bieber, Vivaldi [Various] Various Works (HM, CD, 1995…2004)…         CHAMBER                628 kbps      66      67      71      76      71      133
B.15. Telemann, D'Anglebert [Konrad Junghänel, Kenneth Gilbert] Various Works (H…      LYRICAL ; HARPSICHORD  747 kbps      63      59      63      75      71      135
B.16. VA [Paul O'Dette] Lord Herbert of Cherbury's Lute Book (HM, CD, 1992)…           LUTE                   647 kbps      68      61      82      74      72      144
B.17. Banchieri, Telemann, Couperin, Schobert [Various] Various Works (HM, CD, 1…      CHAMBER                725 kbps      69      68      68      71      74      145
B.18. Keiser [René Jacobs & Akademie für Alte Musik] Croesus (HM, CD, 2000)…           LYRICAL                602 kbps      65      61      66      67      70      133
B.19. Keiser [René Jacobs & Akademie für Alte Musik] Croesus (HM, CD, 2000)…           LYRICAL                585 kbps      64      62      66      67      69      131
B.20. Keiser [René Jacobs & Akademie für Alte Musik] Croesus (HM, CD, 2000)…           LYRICAL                582 kbps      64      62      66      66      67      130
B.21. Haydn, Mozart [Various] Various Works (HM, CD, 1994…2007)…                       ORCHESTRAL             638 kbps      68      63      68      71      71      137
B.22. Rossini [Markus Creed & RIAS Kammerchor] Petite Messe solennelle (HM, CD, …      CHORAL & ORCHESTRAL    488 kbps      59      65      70      66      60      118
B.23. Rossini, Schumann, Brahms, Wolf [Various] Various Works (HM, CD, 1969…2005…      VARIOUS                546 kbps      64      64      75      71      66      128
B.24. Rameau, Beethoven [Alexandre Tharaud, Georges Pludermacher] Various Works …      PIANO                  449 kbps      57      63      73      72      56      114
B.25. Berlioz, de Falla [Philippe Herreweghe, Josep Pons] Nuits d'été ; El Amor …      LYRICAL ; ORCHESTRAL   511 kbps      60      67      73      68      64      123
B.26. Schubert [Paul Lewis, Isabelle Faust, Alexander Melnikov] Fantaisie D934 &…      PIANO                  399 kbps      56      62      69      72      53      106
B.27. Chopin, Liszt, Chostakovitch [Cédric Tiberghien, Jerusalem SQ] Various Wor…      CHAMBER                426 kbps      56      62      69      70      56      109
B.28. Chausson, Copland, Janacek [Alain Planès, Trio Wanderer] Various works (HM…      VARIOUS                500 kbps      59      64      68      72      61      116
B.29. Janacek, Bartok, Part [Paul Hillier & Theater of Voices] Berliner Messe (H…      VARIOUS                583 kbps      66      66      73      71      67      130

                                                                                    AVERAGE (29CD)            598 kbps      65 kbps 64 kbps 70 kbps 71 kbps 67 kbps 132 kbps

                                                                                   ———————————————————————————————————————————————————————————————————————————————————————————
                                                                                    AVERAGE (84CD)            579 kbps      64 kbps 63 kbps 69 kbps 69 kbps 65 kbps 128 kbps
                                                                                   ———————————————————————————————————————————————————————————————————————————————————————————  
            EXTRA CD: VERY LOW BITRATE                                      
C.01. Feldman [Schleiermacher] The Late Piano Works, Vol.2 (MDG, CD, 2009)             PIANO                  219 kbps      22      56      59      68      40      51
C.02. Mompou [Perianes] Música Callada (Harmonia Mundi, CD, 2006)                      PIANO                  274 kbps      38      61      64      69      40      84
C.03. Silvestrov [Blumina] Piano Works (Grand Piano)                                   PIANO                  255 kbps      36      57      62      69      39      76
C.04. VA [Haochen Zhang] Schumann, Liszt, Janacek & Brahms. Piano Works (BIS, CD, 2017)PIANO                  308 kbps      46      59      65      69      42      91
                                                                                     AVERAGE (4CD)            264 kbps      36 kbps 58 kbps 63 kbps 69 kbps 40 kbps 76 kbps
                  EXTRA CD: VERY HIGH BITRATE
D.01. Bach [Moroney] Das Wohltemperierte Klavier (Harmonia Mundi, CD4, 1988            HARPSICHORD            1053 kbps     64      66      68      80      86      146
D.02. Byrd [Tilney] Contrapuntal Byrd (Music & Arts, CD, 2016)                         HARPSICHORD            1014 kbps     65      67      71      81      86      148
D.03. Haendel [Lesaulnier] Al piacere del signor (ORF, CD, 2008)                       HARPSICHORD            1019 kbps     67      65      65      76      83      145
                                                                                     AVERAGE (5CD)            1029 kbps     65 kbps 66 kbps 68 kbps 79 kbps 85 kbps 146 kbps


Here is an screenshot (in red, the 10% highest value for each codec; in yellow the 10% lowest value for all codec — the three grey lines are the mono discs).


Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #7
Was USAC at 22Khz sampling rate? If so, how does Opus 22Khz compare to 22Khz USAC?
I don't get it. Isn't Opus always 48kHz?

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #8
 Also it lacks support in vlc and foobar mobile.
I didn't want to say it was a problem, 64 kbps is a bit too low like you said.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #9
To me the problem is the low sample rate. Exhale quality is excellent for example in Bastille Pompeii at 64 kbps. But 32 khz sample rate makes the vocals sounds bad. Also it lacks support in vlc and foobar mobile.
Resampling is maybe an answer to the quality issue and not the real problem.
My guess is that exhale begins to starve at some point which is lower than the starvation point for MP3 (which is ~120 kbps) or AAC. Maybe 64 kbps is too low for exhale. Resampling could therefore increase quality on many samples (and lower it on others).
But I must say that I would be curious to check how exhale would sound without resampling at mode 1.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #10
"Apple's AAC encoder doesn't allow VBR with HEAAC."
AAC-HE Encoder, CVBR 64kbps, Quality 96
Apple's encoder only allows CVBR, and I was rather looking for TVBR.
Here is the comparison between FDKAAC HE VBR and iTunes AAC HE CVBR:



the same, data sorted from min to max:



With iTunes CVBR: on 91 CD, 72 end with 66 kbps. Nine CD are 67 kbps and the four highest bitrate are 68 kbps… The three mono files are 58 kbps. The lowest stereo is 60 kbps. There's almost no variation here.

But all other competitors and anchors are truly VBR. So I don't think it would be a good idea to replace a true VBR encoder with another one which offers on the very best a constraint VBR mode that seems very close to CBR/ABR. At least not in this test and not without some evidence that iTunes HE-AAC would sound better.

Quote
by the way, this is a great reason to compare qaac tvbr lc/cvbr he, fdk lc/he/he2, faac tns/notns/pns0..10.
Yes, comparing various AAC implementation is something I'd like to do for years.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #11
Was USAC at 22Khz sampling rate? If so, how does Opus 22Khz compare to 22Khz USAC?
No, USAC (exhale 1.0.4 mode 1) is 32000 Hz in this test. The low anchor (MP3 at 64 kbps) is 22050 Hz.
As kode54 says, Opus resamples everything to 48000 Hz before encoding, so comparison is either impossible or pointless (maybe both ;) ).

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #12
I wouldn't say impossible/pointless. It isn't really an apples to apples comparison.
Give the Opus encoder less information to work with by resampling the input files to 32000 Hz. Maybe the Opus encoder is wasting bits on frequencies that the USAC encoder doesn't even have to deal with.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #13
LithosZA,

Opus --bitrate 64  spends 0.1-0.2 kbps for frequencies higher than 15.6 kHz.  The decision to keep HF  was taken by developers long time ago and for a really good reason.  32 kHz won't benefit Opus@64k in any way.

Also if someone tests codec  it's best practices just go with what developers suggest ( read default).

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #14
I wouldn't say impossible/pointless. It isn't really an apples to apples comparison.
Give the Opus encoder less information to work with by resampling the input files to 32000 Hz. Maybe the Opus encoder is wasting bits on frequencies that the USAC encoder doesn't even have to deal with.
First, you must take into account that resampling to 32 KHz has a negative consequence on temporal resolution. It can audibly increase smearing and pre-echo.
An alternative would be lowpassing to 16 KHz without resampling to 32KHz.
But Opus is a mature format. Tests have been done long time ago to ensure that it keeping higher frequencies don't audibly affect sound quality. I'm not technician but I guess Opus has some advanced techniques to code efficiently HF contents at low bitrate [Edit: IgorC already answered this more precisely]. I'm pretty confident that sound quality can not enhanced that way and that developers already explored that path long time ago.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #15
First, you must take into account that resampling to 32 KHz has a negative consequence on temporal resolution. It can audibly increase smearing and pre-echo.
That would be true when signals are coded with variable Fs AND fixed-sized MDCT.
In case of opus, 32kHz input is just resampled to 48kHz... so, I wouldn't say temporal resolution degrades by that.

Quote from: guruboolez
But Opus is a mature format. Tests have been done long time ago to ensure that it keeping higher frequencies don't audibly affect sound quality.
I agree.

Re: Personal listening test at 64 kbps: Opus, Exhale, FDK HE-AAC, iTunes LC-AAC +MP3

Reply #16
I've noticed when Opus fails under <128kbps or from a killer sample, It had the same puffy noise. With quiet ambient that just synth tones/sine waves the bit rate can bloat hard 96k = 145, 128k = 175, 160k = 255. xHE AAC with exhale doesn't have those issues.
Got locked out on a password i didn't remember. :/