Will the next test be a 96kbps test, with the same player?
Would be interessting if LC can beat HE in that range or if Lame 128 will be beaten by any codec.

Will the next test be a 96kbps test, with the same player?
Would be interessting if LC can beat HE in that range or if Lame 128 will be beaten by any codec.

he-aac isnt available for 96kbps afaik

I'm not at ease with vorbis at this bitrate during a blind test : it sounds too particular (hiss, desquilibrated tonal range : more treble, poor low-medium, and limited stereo), and it's easy for me to detect the encoder. I'm rating vorbis, and not an unknow encoder. So it isn't blind anymore.

I agree 100%.  I often knew right away which one was vorbis, and I struggled to not let that influence my ratings.  Am I rating it too high because I'm an OSS true-believer?  Am I rating it too low because I'm overcompensating?

I also agree that its performance was spotty.  A few samples had very serious problems (for me, those included Illinois, Polonaise, gone).  I hope 1.0.1 fixes these issues and some of the problems exposed in the 128k test.

he-aac isnt available for 96kbps afaik

bitrate = 95
channels = 2
samplerate = 44100
codec = AAC+SBR
tool = Nero AAC Codec 2.5.5.3


Nero 6 can do it. Whether it's smart I don't know.

I hope 1.0.1 fixes these issues and some of the problems exposed in the 128k test.

The actual quality problems won't be fixed until 1.1 I guess. Most of the major issues I heard have been there since 1.0RC2 or so...

he-aac isnt available for 96kbps afaik

Nero 6 can do it. Whether it's smart I don't know.

which settings did you use to create such a file?

he-aac isnt available for 96kbps afaik

Nero 6 can do it. Whether it's smart I don't know.

which settings did you use to create such a file?

Umm, just select cbr 96kbps and High Efficiency profile..

Umm, just select cbr 96kbps and High Efficiency profile..

hm, i meant the vbr profiles, as i dont think that ahead wanted to make it possible for example to create he-aac files with 448kbps by using the cbr option 

also the presets use lc aac for ~96kbps (vbr and cbr)

Umm, just select cbr 96kbps and High Efficiency profile..

hm, i meant the vbr profiles, as i dont think that ahead wanted to make it possible for example to create he-aac files with 448kbps by using the cbr option 

Well you can't do that, it changes to LC if you try to create a cbr 112kbps HE file. 96kbps cbr is the highest allowed HE profile setting.

Gosh!  I only listened to one sample (New York City)... but I gave Vorbis a 1.3    Even the muffled lo-fi sounds of QT and WMA sounded nicer to me than the lavish sweeps of distortion and generous, unrestrained servings of noise that were dished out by Vorbis.  I really wasn't expecting that at all.  Still, I suppose I shouldn't judge it on one sample alone...

And Lame really is very good, (even if twice the bit-rate)... it's nice to know that.

Anyone else think it interesting that Ogg Vorbis was beaten by QT in the 128kbps test, but crushed QT in this test?

Of course, HE-AAC beat Ogg Vorbis, as did MP3Pro.  Thus, SBR technology beats out Ogg Vorbis.

Anyone else think it interesting that Ogg Vorbis was beaten by QT in the 128kbps test, but crushed QT in this test?

Not really. It was pretty much expected, since QT doesn't have a High Efficiency profile.

Great test! However, I wonder who is doing 64kbps encoding. I believe that most people use at least 128kbps or 96kbps at the very minimum...flash card prices are getting lower and lower...anyway, that is why I'm very glad that a 128kbps test already took place...
I hope that other test will follow...how about a 160kbps test

By the way: What is HE-ACC?

Thanks

By the way: What is HE-ACC?

Thanks

HE AAC is one of the profiles of MPEG-4 AAC. It uses SBR (Spectral Band Replication) in order to achieve a high efficiency at low bitrates. Currently there's only one publicly available HE AAC encoder implementation - Nero AAC/AAC-HE encoder.
Check here for more info about HE AAC.

I hope that other test will follow...how about a 160kbps test

Actually, I'd like to see it go even lower... a 32kbps test (with samples containing some music, but mainly speech) would be a fairly good reflection on low-bitrate streaming -- it's around the rate used by the BBC's RealAudio streams, anyhow. Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.

Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original. I've not yet been able to figure out a decent way to analyse the results, though. Not having a scale opens the possibility of a non-transitive chain: i.e. a set of samples X_1,..., X_k where X_1 is preferred to X_2, X_2 to X_3, ..., *and X_k to X_1*. I'd love to see something like that happen in practise.

This is the table of my results:



According to the averege the best my preferred encoder seems to be HE AAC even if i should say that i’m quite surprised for the result of mp3PRO (or maybe i’ve overestimated HE AAC).

I’ve done a sort of  ranking for the best, real antagonists at 64 Kbps. The first position is colored in green, the second in yellow and the third in red. With this direct comparison Mp3PRO is (according to my preferences) often better than HE AAC.

Mp3Pro has shown a detectable lowpass (16Khz) but this is not the real problem for a 64 Kbps, artifacts are more annoying. There is an interesting thing i’ve perceived: with HE AAC the high frequencies seem unnatural, attenuated, as if it was lowpassed. While doing the blind test i imputed this to the lowpass, but later i’ve discovered that the HE AAC files are lowpassed at about 20 kHz (surely inaudible for me). Does FAAD use dithering when decoding ? If not, i think that the fact could be explained with the SBR “problem” of which guruboolez give us an excellent description.

While the two codecs above scored a very close quality level, i can’t say the same for Vorbis that is often behind the others two. I surely agree with guruboolez: Vorbis in this bitrate range is easy detectable because of noise and exaggerated highs (with sharp attacks the result is quite annoying).
We all are waiting for the 1.1 version that should give better result with this type of artifacts.

At the end, i sincerely want to thank Roberto for his effort organizing this useful test. The number of participants is increased and this is a clear indication of good organization.

hmm, surprisingly wma was very close to the low anchor FhG on some samples. 

wma was probably the standard edition?
lame was probably abr 128?

hmm, surprisingly wma was very close to the low anchor FhG on some samples.  

wma was probably the standard edition?
lame was probably abr 128?

*  Ahead/Nero 6.0.0.15 HE AAC VBR profile Streaming :: Medium, high quality
    * Ogg Vorbis post-1.0 CVS -q 0
    * MP3pro (from Adobe Audition 1.0) VBR quality 40, Current Codec, allow M/S and IS, allow narrowing, no CRC
    * Real Audio Gecko (from Real Producer 9.0.1 64kbps
    * Windows Media Audio v9 VBR quality 50
    * QuickTime 6.3 AAC LC 64kbps, Best Quality
    * Lame MP3 encoder 3.90.3 --alt-preset 128 --scale 1. high anchor
    * FhG MP3 encoder (from Adobe Audition 1.0) 64kbps CBR, Current codec, allow M/S, no I/S, allow narrowing, no CRC. bottom anchor

Will the next test be a 96kbps test, with the same player?

Well, definitely not from me. I'm not interested in testing 96kbps.

Anyway, I replaced all the plots. Now Lame MP3 is Lame 128. This should help avoid confusion.

@bond: Thanks, I used your zoomed in version

@bond: Thanks, I used your zoomed in version

great that it was usefull

At first, let me thank Roberto for his efforts: thank you! It was nice to see that the participation level was high.

For me HE AAC and MP3 Pro came out on top, while Ogg was just mediocre. Forget the rest. The below is only true for my personal ratings and usually the anchors are disregarded, unless mentioned otherwise.

The most surprising result was with sample 06, Illinios. Here HE AAC totally sucked, while Ogg shone (and the others did well too). This is contrary to the average public ranking.

On sample 9, Polonaise, MP3 Pro was worst. MP3 Pro also has the highest standard deviation of all encoders (including anchors).

Sample 07 was also interesting. While all other encoders dipped real low in quality, HE AAC was doing well.

Also worth mentioning is that I didn't rate Lame 128k as best on 6 samples. In 5 of these cases, HE AAC was rated higher than Lame.

WMA std was rated worst 9 out of 12 times. WMA std was also most consistent in quality. It consistently sucked.

On the 4 occasions that FHG was not rated lowest, WMA std was worst.

Great test! However, I wonder who is doing 64kbps encoding.

Me! I use 64-80kbps Vorbis to put albums on an USB RAM stick and carry home to use in my mom's portable while coding.

Time to switch to HE-AAC for that...

Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.

You got that right!

Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original.

Doesn't work. Why? Well, see you own comment above...

Hello.

Menno came with an idea: Replacing "Lame 128" and "FhG MP3" with "High/Low Anchor" on the plots.

As to completely avoid confusion, since several people are thinking Lame won, and it wasn't there to win or lose to start with.

Any comments? Suggestions on alternatives?

Hello.

Menno came with an idea: Replacing "Lame 128" and "FhG MP3" with "High/Low Anchor" on the plots.

As to completely avoid confusion, since several people are thinking Lame won, and it wasn't there to win or lose to start with.

Any comments? Suggestions on alternatives?

Yes, it's a good thing. A different color maybe for each anchor, in order to avoid subconscious confusion.
Is it possible to re-assignate a different place for the different plot ? I mean : from left to right, the winner (HE-AAC) to loser (Real?). This may be useful, to see on which sample winner(s) fail(ed). Random position aren't useful in my opinion.