AAC at 128kbps v2 listening test - FINISHED

Topic: AAC at 128kbps v2 listening test - FINISHED (Read 64037 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

AAC at 128kbps v2 listening test - FINISHED

Reply #25 – 2004-03-01 10:56:21

One more thing - please do not extrapolate results from CBR to VBR (saying: if the codec was used in CBR/VBR mode it would be better/worse, etc..)

MP3 listening test clearly shows that:

http://www.rjamorim.com/test/mp3-128/results.html

One codec vendor's VBR codec was inferior to CBR implementation (slightly tuned) - which means that extrapolations do not work - CBR and VBR bit allocations could be quite different.

AAC at 128kbps v2 listening test - FINISHED

Reply #26 – 2004-03-01 11:05:48

Quote

so the efficiency argument is false.

If efficiency is a relation between quality and size, therefore Nero AAC seems to be less efficient (you said it, hierarchy is arguable).

I wonder: if there's space enough for the next test, why not include both iTunes and Nero AAC? Afterall, AAC is in a unique situation, with 6 or 7 (at least) editor currently working on AAC encoder (should be discuss in the netx pre-test thread, but people should start thinking on the idea).

AAC at 128kbps v2 listening test - FINISHED

Reply #27 – 2004-03-01 11:16:46

Am I reading this graph correctly?

iTunes and Nero are statistically tied because the extended arm of Nero rises higher than the middle point of iTunes, am I right?

If so then you cannot say that Nero is better than Faac by the same criterion, though you can say that iTunes is better that Faac as they only overlap at the very edges (and definately better than the ones it doesn't overlap at all).

I'd say that makes iTunes a winner as it is statistically better than all but one of the other codecs, whereas Nero statistically beats all but two (and 'looks' very close to the rest).

Does that make sense?

edit: answering my own question it seems that any overlap at all means less than 95% confidence so I assume some sophisticated statistical method beyond my ken was used to decide who was tied for which position.

AAC at 128kbps v2 listening test - FINISHED

Reply #28 – 2004-03-01 11:23:49

Quote

iTunes and Nero are statistically tied because the extended arm of Nero rises higher than the middle point of iTunes, am I right?

Not sure - generally there should not be an overlap between the error bars themselves either, but the addition of errors is not linear. It would probably be better to figure this out from the raw data than from the graph, especially as I cannot align robertos text with what the graph shows.

To answer your question, from eyeballing the result it's more likely that Nero is better than Compactaac! and Real than that iTunes is better than Nero, though it's also quite likely iTunes is simply the best encoder, right now. You just can't conclude it with a very high degree of certainty.

AAC at 128kbps v2 listening test - FINISHED

Reply #29 – 2004-03-01 11:26:56

Quote

@Continuum: I'm afraid I messed the anon ordering a little. Sorry :/

No, the one result is actually missing. No problem, it just bugs me that I have to wait for the decryption key now.

AAC at 128kbps v2 listening test - FINISHED

Reply #30 – 2004-03-01 11:29:57

I don't kow if it's possible, but I would be interested but a specific graph with lowscorers only (as ff123 (?) did it for the multiformat test last year). There are a lot of results with few discrimination between encoders, and if there are significant about the transparency of modern AAC encoders, they don't help to see the existing difference among them.

AAC at 128kbps v2 listening test - FINISHED

Reply #31 – 2004-03-01 12:02:38

Quote

Anyway, it's safe to say that all codecs represented here are pretty mature and, no matter what your choice among them is, it's very likely you'll get very good results for your encodings.

Personally, i strongly disagree with this statement
I'd like to see some organized results from listeners who can discriminate codecs better, maybe these results can't give us statistical sureness but could be useful for developers/tuning.

AAC at 128kbps v2 listening test - FINISHED

Reply #32 – 2004-03-01 12:30:37

Quote

BTW, the obvious outcome of this is: iTunes will be featured at the 128kbps multiformat test, that shall start next month.

Hmmm... I already look forward to seeing how QuantumKnot's tuned Ogg Vorbis encoder compares to iTunes at 128kbps.

Of course, it's not certain that this particular Ogg Vorbis encoder will be featured, but it seems a logical choice to me at this moment.

Thank you for the test, Roberto.

AAC at 128kbps v2 listening test - FINISHED

Reply #33 – 2004-03-01 12:40:02

VERY interesting results:

1) apple knows whats good! its still on a position were someone can say its the best
also considering that its still cbr and was now tested against vbr nero

2) real knows why they offer 192kbps aac in their music store (sorry karl )
for a big company wanting to compete against itunes it surely needs to offer a higher bitrate to keep up qualitywise (but well against all the wma stores around 128kbps real would also have been enough i assume )

thinking about the "possible" results for the multiformat test i think someone can say that there is no way around itunes when it comes to buying online music with a great quality/size ratio

3) faac seems to be now on par with the other codecs and in fact also looking at my personal results its great!!!
but
when i did the listening test there was on nearly every sample one codec, which was easy to sort out
i always thought that this was faac (sorry about that ) but now looking at my results its Compaact!!!
i wonder how this codec got such a good ranking here? it was really worse and easy to sort out imho (sorry alex)
didnt someone else had the same feelings/results like me?

and not to forget:
THANKS A LOT ROBERTO for this great test!!!
you know how much value these tests bring to us, the costumers, we cant say thank you for this often enough!

AAC at 128kbps v2 listening test - FINISHED

Reply #34 – 2004-03-01 12:52:31

Quote

when i did the listening test there was on nearly every sample one codec, which was easy to sort out i always thought that this was faac (sorry about that ) but now looking at my results its Compaact!!! i wonder how this codec got such a good ranking here? it was really worse and easy to sort out imho didnt someone else had the same feelings/results like me?

What do you mean? Are you surprised about faac good quality or compaact! good performances?

In my opinion, faac new lowpass value (16 Khz, whereas 15 Khz last year) is more friendly for ABX tests. In this test, I've often ranked Real badly due to its lowpass. I'm not especially annoyed by lowpass on daily listening (portable use, and poor earbud performances), but on direct comparison, lowpass have significant impact on quality: it gives a metallic coloration, sometimes stereo feeling is affected (dull sound), etc... lowpass is sometimes difficult to interpret even if it's easy to detect.

AAC at 128kbps v2 listening test - FINISHED

Reply #35 – 2004-03-01 12:59:33

Quote

What do you mean? Are you surprised about faac good quality or compaact! good performances?

no i am surprised that overall compaact was voted that high cause only from my results it would have got a score of 3.4 (which means its cearly worse than the others for me)

AAC at 128kbps v2 listening test - FINISHED

Reply #36 – 2004-03-01 13:09:13

Quote

Quote
What do you mean? Are you surprised about faac good quality or compaact! good performances?

no i am surprised that overall compaact was voted that high cause only from my results it would have got a score of 3.4 (which means its cearly worse than the others)

OK
I don't have an overall view of my results (score table). I tried to have an approximate opinion by reading some results of mine. Apparently, compaact! is very unconstant. Absolutely fabulous with hongroise.wav (i.e. the piano sample), compaact! is on some samples the worse encoder, suffering from heavy troubles in the background. Faac seem to share the same position - and Nero AAC too, but problem is less pronounced.

I second [proxima] when he disagree about Roberto's conclusion. In my opinion, most AAC encoders are not mature enough. What is maturity for AAC? I guess that an AAC encoder should be named 'mature' when the performance are superior to MP3 on most samples. And I'm not sure that's the case for faac, real AAC, compaact! (and Nero AAC too, but I suppose that this encoder is more mature than the three other challengers).

AAC at 128kbps v2 listening test - FINISHED

Reply #37 – 2004-03-01 13:22:50

The message of this test I like most is probably this: Lack of recent development is not a good indicator for poor quality (Take that, you Vorbis and MPC moaners! )

But seriously now, I'm a bit disappointed of AAC. With some samples (Sample 1, most noticeably) the codecs still all sounded worse to me than Lame at 128kbps. And 128kbps mp3 is not even supposed to be any competition for AAC at that bitrate. AAC is supposed to deliver equivalent quality at a lower bitrate, after all. And that advantage is just not huge at the moment. Maybe it can compete with mp3 at 160kbps, but it's no match for Lame APS. It's not such a big surprise then that modern formats still haven't really managed to replace mp3. A 25% advantage is just not enough to justify a switch in standards. I think the new codecs have to deliver the same quality as mp3 at half the bitrate before they will become a real danger to mp3's popularity.

I also think that Compaact! behaves strangely. In my results it won on the DaFunk sample and on Velvet, but got some terrible ratings on other samples, due to an awfully annoying metallic ringing artifact.

AAC at 128kbps v2 listening test - FINISHED

Reply #38 – 2004-03-01 13:42:31

Going by the overlap rule it can not be said that either Nero or Quicktime is better than FAAC. Hmmmm... I wonder what would happen if someone did some tuning and came up with a preset for FAAC that used a slightly higher lowpass than 16k traded off against a slightly lower q setting than 115 at the same file size.

I wonder what we will see when Roberto reveals the bitrates for the samples and the results are "adjusted" by this fudge factor. The debate goes on.

AAC at 128kbps v2 listening test - FINISHED

Reply #39 – 2004-03-01 13:43:46

It's on slashdot now. Batten down the hatches.

AAC at 128kbps v2 listening test - FINISHED

Reply #40 – 2004-03-01 13:52:20

Quote

I think the new codecs have to deliver the same quality as mp3 at half the bitrate before they will become a real danger to mp3's popularity.

According to Microsoft wma sounds better at a fraction of any bitrate.

Besides: compared to the development that went into mp3 (Lame) AAC is still in it's infancy...

AAC at 128kbps v2 listening test - FINISHED

Reply #41 – 2004-03-01 14:00:12

Would it be possible to get another visualization of the results, too? Instead of the confidence intervals show the variance (or something similar) of each result.

It would be interesting to see, where the opinions differ most.

AAC at 128kbps v2 listening test - FINISHED

Reply #42 – 2004-03-01 14:18:53

just a quick question. Why wasn't QuickTime included in this test? In your last AAC 128kbps test, it was the winner. So it would have been interesting to see, how well they would perform against each other.

Anyway, awesome test! Thank you!

AAC at 128kbps v2 listening test - FINISHED

Reply #43 – 2004-03-01 14:20:50

Quote

just a quick question. Why wasn't QuickTime included in this test? In your last AAC 128kbps test, it was the winner. So it would have been interesting to see, how well they would perform against each other.

Anyway, awesome test! Thank you!

iTunes is more friendly for end-user and daily use. You can't rip a CD with QT. You can't (natively) batch encode with this software.
iTunes allows CD ripping/encoding, and batch encoding from .wav files. iTunes uses the QT encoder in "BETTER" profile (whereas QT allows you a "BEST" profile).
Last but not least, iTunes is free

AAC at 128kbps v2 listening test - FINISHED

Reply #44 – 2004-03-01 14:26:09

Quote

Quote
just a quick question. Why wasn't QuickTime included in this test? In your last AAC 128kbps test, it was the winner. So it would have been interesting to see, how well they would perform against each other.

Anyway, awesome test! Thank you!

iTunes is more friendly for end-user and daily use. You can't rip a CD with QT. You can't (natively) batch encode with this software.
iTunes allows CD ripping/encoding, and batch encoding from .wav files.

Don't these two use the absolute same encoding engine?

AAC at 128kbps v2 listening test - FINISHED

Reply #45 – 2004-03-01 14:27:49

I've just edited my previous post.
Encoder are the same for both programs. Just a difference in profile (but with limited if not inaudible impact on quality for 44.1/16 PCM files).

AAC at 128kbps v2 listening test - FINISHED

Reply #46 – 2004-03-01 14:28:34

Quote

Quote

Quote
just a quick question. Why wasn't QuickTime included in this test? In your last AAC 128kbps test, it was the winner. So it would have been interesting to see, how well they would perform against each other.

Anyway, awesome test! Thank you!

iTunes is more friendly for end-user and daily use. You can't rip a CD with QT. You can't (natively) batch encode with this software.
iTunes allows CD ripping/encoding, and batch encoding from .wav files.

Don't these two use the absolute same encoding engine?

that is what I just thought too - QuickTime is made by Apple...so the one that is used in iTunes should be the newest release

AAC at 128kbps v2 listening test - FINISHED

Reply #47 – 2004-03-01 14:52:03

Since I could not differentiate anything in this test (congrats to those with better hearing acuity than me!) I reached the personal conclusion not to worry about it anymore. I use Nero AAC encoder at the transparent VBR profile, which for my ears should be plenty of "overkill"

AAC at 128kbps v2 listening test - FINISHED

Reply #48 – 2004-03-01 14:53:59

Roberto, I just saw this comment on the Slashdot thread:

Quote

As a physicist, I'd just like to draw everyone's attention to the error bars on these charts. For the majority of the tests, it's possible to draw a horizontal line through the 95% confidence intervals of nearly all the points.

Hence, the conclusions declaring clear winners/losers in these cases are invalid. If 99% confidence intervals were used (which gives a better statistical test), I feel that no clear winners or losers would be drawn.

Be careful with these sort of studies - even though the author has used confidence intervals, he has failed to use them to infer the proper conclusions.

That said, it's awfully nice to see error bars on this sort of website. Simple data points give such a false sense of precision, I find...

Any comments? I wouldn't have a clue

AAC at 128kbps v2 listening test - FINISHED

Reply #49 – 2004-03-01 15:36:21

What is the link to the slashdot thread?

Notice