Skip to main content

Topic: AAC at 128kbps v2 listening test - FINISHED (Read 48827 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #150
Anyway, AAC quality sucks, according to Microsoft, so we should as well stop discussing this shitty format and move on to WMA standard

http://www.macnn.com/news/19380
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • JohnV
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps v2 listening test - FINISHED
Reply #151
Quote
Quote
Not anykind of reliable approximation of cbr quality from vbr. The coding methods are so different. CBR has often different kind of issues than VBR, and CBR can avoid undercoding problems which may happen with VBR especially at mid-low bitrates like near 128kbps average. The tweaking of CBR and VBR profiles must be done totally separately.

We are not trying to approximate CBR from VBR. We are approximating a theoretical VBR mode that stayed at an average bitrate of 128kbps across test samples.

It's pointless, because it's not tweaked and emphasis in real life can be put on psychoacoustic settings which lower bitrate but decrease the quality relatively less than some other setting or this so called linearly approximated theoretical VBR-mode preset.
Juha Laaksonheimo

  • ScorLibran
  • [*][*][*][*][*]
  • Banned
AAC at 128kbps v2 listening test - FINISHED
Reply #152
Quote
Quote
ScorLibran
1. Tell me how it's wrong. That's been addressed to some extent (though I'm not enough of an expert to know what the answer really means)... and ...


Psychoacoustic coders do not scale quality with bit rate in a linear fashion - meaning that you can't just project quality result to some other bit rate by simple proportion.

Understood.  So naturally my next question would be...

Then what does the math look like for at least getting a fairly accurate composite rating?  Or at least closer than a linear calculation?

Quote
Quote

From among the Eternal Questions of Psychoacoustic Audio Encoding™: "My quality target is X. How many files encoded to my quality target with Nero AAC can I put on my 20GB hard drive? OK, how many files encoded to the same quality target with QT AAC can I put in the same space?"


Well - we did these tests before listening test, and Nero ended up in 130.5 (correct me if I am wrong) in average - which indicates how much stuf you would be able  store.

Of course, it depends on signal statistics - so if you want to know exactly how much, you'll either have to encode files,  or to use CBR.  Different music has different masking patterns, and if you use 'real' VBR - it is not easy to predict final bit rate without, at least, psychoacoustic processing.

So to answer the question of the non-expert music encoder asking how to calculate capacity requirements for each encoding format...

"Encode your collection with a setting you know you like, and if you can't fit all the music you want to onto your drive, then try a little lower setting and start over."

-- Not feasible.

"Use CBR."

-- Not good advice for the best sound quality, in many cases.

So, that person is stuck without a good answer, unless some kind of composite rating system is used with test results such as these.

First, I'd need to have an example of math that would be more accurate than my linear calculations.

And regarding variances across samples and types of music, well, that's where the (now mentioned at least 3 times) qualification of composite rating interpretation would come into play.  Everyone knew going in that in Roberto's test there could only be a limited number of samples, and it was my understanding that it would be safe to quote those results with the qualifier of "with the samples tested, here's how the codecs performed..."

Once we have a more accurate calculation method, we should just go with what we have so far, and as more tests are done, use those results to provide new composite ratings over time.

I'm just trying to speak for the hundreds of people that have posed these questions to me over the past year, and never have gotten a good, straightforward answer.  And if my presentation of these questions and ideas is getting a bit too "simplistic", then forgive me, but inquiring minds want to know! 
  • Last Edit: 02 March, 2004, 07:18:24 PM by ScorLibran

  • QuantumKnot
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps v2 listening test - FINISHED
Reply #153
Quote
Anyway, AAC quality sucks, according to Microsoft, so we should as well stop discussing this shitty format and move on to WMA standard

http://www.macnn.com/news/19380

hehehe Interesting article.

Quote
It’s true: Apple’s AAC cuts sound great with the tiny little speakers that come with computers. And they sound pretty good on an original (but AAC upgraded) iPod through the stock headphones. But listen through good headphones and what you’ll hear is dull-sounding bass, slightly sibilant voice quality and a lack of three-dimensionality.


Three-dimensionality?  Perhaps he lacked those special blue and red headphones

  • tcristy
  • [*]
AAC at 128kbps v2 listening test - FINISHED
Reply #154
I encoded 145 rock songs ranging from classic rock through metal with Compaact Q5 VBR and no low-pass filtering (I hate the low pass, which is why I use Compaact and not iTunes, which sounds dead to me) and got the following distribution:

http://home.columbus.rr.com/tcristy/CPDistro.jpg

The average was 126 kpbs.

The highest bit rate song at a bit over 143 kbps average was ZZ Top's Sleeping Bag, which has a synthed cymbal at about 6 beats per second through the whole thing.

Tim
  • Last Edit: 02 March, 2004, 07:38:01 PM by tcristy

  • bidz
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #155
If all encoders was tested in CBR mode, this discussion wouldn't be a issue

That would - atleast - be a 100% fair comparison.
myspace.com/borgei - last.fm/user/borgei

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #156
Last year, Roberto was criticized for choosing CBR for all encoders, especially for Nero AAC, which had a VBR mode. Some people were sure that Nero AAC with VBR would beat QuickTime AAC CBR... That was another discussion, and people didn't considered the forced CBR comparison as something fair. And that's probabably true.

  • kwanbis
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #157
Quote
That would - atleast - be a 100% fair comparison.

it won't, we are trying to compare what is the "standard" among them, and the way it was done is perfect, for me at least.

  • bidz
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #158
Quote
Quote
That would - atleast - be a 100% fair comparison.

it won't, we are trying to compare what is the "standard" among them, and the way it was done is perfect, for me at least.

Well, IMHO, a perfect comparison consists of identical bitrates. If one encoder uses VBR, and gets 140kbps - thats not fair to compare against a 128kbps CBR codec IMHO. (even though if the CBR wins).

Just my opinion.
myspace.com/borgei - last.fm/user/borgei

  • ScorLibran
  • [*][*][*][*][*]
  • Banned
AAC at 128kbps v2 listening test - FINISHED
Reply #159
For my own concern, I'm not so interested in what's "fair".  The bitrate a codec averages on a sample is what it averages...whether it's high or low is incidental to me.

What I'd like to see, though, is some kind of "quality per bit" rating, or a "composite rating".  Like I said, although VBR doesn't really put relevance on bitrate, people do, since bitrate determines filesize, and filesize is used to determine how much music you can fit into a limited space.

Simple concept, but it seems the answer may be quite complex. 

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #160
The bitrate distribution table is up
http://www.rjamorim.com/test/aac128v2/results.html

Just under the individual plots, as usual.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • bidz
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #161
140 average against 128 .. (Nero - iTunes). I'm really curious on how Nero's rating would be if it would use 128kbps CBR... Oh well..
myspace.com/borgei - last.fm/user/borgei

  • Stux
  • [*][*]
  • Developer
AAC at 128kbps v2 listening test - FINISHED
Reply #162
Quote
Quote
So that warrants allowing a codec to use more bits than allowed by the test setup?


Hmm.. we had this discussion before - and if I remember correctly, in the time of 128 extension test  it was decided to use values for near 128 kb/s on average content, not on test items.

The fact that encoder used more bits on this particular sample set just means that it judged them as "hard to encode"

Of course, you could fine-scale encoder for each sample to even give you 128 kb/s for each sample with VBR - but what's the use of that - unless encoder has 2-pass VBR supported and used by most users?

Quote
What are you trying to say there?


That linear scaling of quality to projected bit rate is dead flat wrong.

By telling a CBR encoder to encode at 128kbps you are telling it to sacrifice quality to encode at 128kbps.

Then you place VBR encoders in the same situation and tell them to sacrifice bitrate accuracy to maintain quality... in a quality oriented test this is not fair.

Personally (and we do this in video tests), if a codec operates best in VBR mode then a VBR setting should be used which comes as close as possible to the CBR bitrate ON THAT SAMPLE, otherwise what's the justification for not using the VBR bitrate with the CBR encoder, thus asking it to constrain its quality to the same bitrate as the VBR encoder.

Unless of course the granularity on VBR aac encoders isn't that crash hot.


Finding the VBR settings which matches a CBR bitrate might take multiple encodings, but I think its the only way to have a fair test, the test is not how each encoder does at "ultra streaming" quality, but rather at 128kbps.

  • sven_Bent
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #163
Quote
Anyway, AAC quality sucks, according to Microsoft, so we should as well stop discussing this shitty format and move on to WMA standard

http://www.macnn.com/news/19380

This article is a joke

He compared the downloaded track to HIS original CD.... who say its from the same source ?

anyway her clearly take time to say bad things about AAC but he really lacks to mention any better alternative.

The burned disc did NOT play in any of my CD players. Not in the ones hooked up to my stereo, my portable players, or even in an old laptop without DVD capabilities. Nor did they play on either of my older MP3 players.

(May 15 update: It turned out the blank CD I was using was bad. I used other discs and was able to play them on my CD players.)


So it took him NINE days to figure out the CD was bad..... now thats a technical guy i can put my trust in...DOH
He is clearly missing the science rule number one: "anything has to be done twice"


<i>Any device that can play a DVD can play burned copies of Apple’s AAC-compressed songs</i>
What the HECK. this is just plain wrong.. unless he burns an AudioCD which he DOES NOT mention at all.
  • Last Edit: 03 March, 2004, 01:15:09 AM by sven_Bent
Sven Bent - Denmark

  • Jerethi
  • [*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #164
Quote
Anyway, AAC quality sucks, according to Microsoft, so we should as well stop discussing this shitty format and move on to WMA standard

http://www.macnn.com/news/19380

This article is a pretty pathetic attempt at journalism.  He has obviously done little or no research, and clearly evinces a biased point of view.  This article is completely devoid of merit.

Either he's accusing Dolby of falsely representing the fact that AAC has undergone extensive listening tests, or he's claiming to have the shiniest golden ears this world has ever known.

What the hell are we paying editors for these days?   

EDIT: Well, I guess there's some meaningful content in the comments he makes about iTunes online store, but this hardly what I would call a "review."
  • Last Edit: 03 March, 2004, 01:33:24 AM by Jerethi

  • sony666
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #165
I would have preferred only CBR encodings in this test and the previous mp3 one. but that's just me
hope there will be a poll regarding this question for the next test.

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #166
Quote
I would have preferred only CBR encodings in this test and the previous mp3 one. but that's just me
hope there will be a poll regarding this question for the next test.

Impossible. How do you expect me to come up with CBR Musepack? Or VBR iTunes?

Also, Vorbis zealots would slaughter me if I forced it to use CBR.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • Continuum
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #167
Quote
The bitrate distribution table is up
http://www.rjamorim.com/test/aac128v2/results.html

Great!

Can we get the decryption key now too?

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #168
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • harashin
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #169
Decrypted my results. Somehow I cannot find my result for sample05 at http://www.rjamorim.com/test/aac128v2/comments/results05/.
Other results are available there.

Code: [Select]
ABC/HR for Java Version 0.4b3 SE, 20 2月 2004
Testname: Hongroise listening test

Tester: harashin

1L = Sample05\Hongroise_1.wav
2R = Sample05\Hongroise_5.wav
3L = Sample05\Hongroise_2.wav
4R = Sample05\Hongroise_3.wav
5R = Sample05\Hongroise_4.wav

---------------------------------------
General Comments:
---------------------------------------
2R File: Sample05\Hongroise_5.wav
2R Rating: 4.5
2R Comment:
---------------------------------------

ABX Results:
Original vs Sample05\Hongroise_1.wav
   25 out of 41, pval = 0.105
Original vs Sample05\Hongroise_2.wav
   15 out of 28, pval = 0.425
Original vs Sample05\Hongroise_3.wav
   16 out of 32, pval = 0.569
Original vs Sample05\Hongroise_4.wav
   6 out of 14, pval = 0.788
Original vs Sample05\Hongroise_5.wav
   6 out of 6, pval = 0.015
  • Last Edit: 03 March, 2004, 02:47:09 AM by harashin
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/

  • sony666
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #170
I subconsciously associated multiformat with "mp3 vs. AAC" by wishful thinking, and unjustly pushed aside mpc and vorbis in my (sleepy) mind. my apologies.
  • Last Edit: 03 March, 2004, 02:47:34 AM by sony666

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #171
For some strange reason, I couldn't decrypt some of the Sample05 (and a few other samples) results.

It must be a bug in ABC/HR Java

Here's an example:
http://pessoal.onda.com.br/rjamorim/results09.zip

BTW: I already reported the bug to schnofler.
  • Last Edit: 03 March, 2004, 02:49:15 AM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • harashin
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #172
Quote
For some strange reason, I couldn't decrypt some of the Sample05 (and a few other samples) results.

It must be a bug in ABC/HR Java

Here's an example:
http://pessoal.onda.com.br/rjamorim/results09.zip

BTW: I already reported the bug to schnofler.

I see. The results09.erf can't be decrypted here, either.
Folding@Home Hydrogenaudio.org Team ID# 32639
http://folding.stanford.edu/

  • Continuum
  • [*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #173
Quote
Decrypted my results. Somehow I cannot find my result for sample05 at http://www.rjamorim.com/test/aac128v2/comments/results05/.
Other results are available there.

My result for sample 5 is missing there as well, for some unknown reason. No problem decrypting it locally:

Code: [Select]
ABC/HR for Java Version 0.4b3 SE, 20 Februar 2004
Testname: Hongroise listening test

Tester: Continuum

1L = Sample05\Hongroise_3.wav
2R = Sample05\Hongroise_2.wav
3L = Sample05\Hongroise_5.wav
4R = Sample05\Hongroise_4.wav
5L = Sample05\Hongroise_1.wav

---------------------------------------
General Comments:
---------------------------------------
1L File: Sample05\Hongroise_3.wav
1L Rating: 3.0
1L Comment: pre-echo? smeared at 10.6
---------------------------------------
2R File: Sample05\Hongroise_2.wav
2R Rating: 4.5
2R Comment: ABC-HR click moved (start pos 9.03)
little problem at sec 15
---------------------------------------
3L File: Sample05\Hongroise_5.wav
3L Rating: 2.0
3L Comment: severly smeared at ~10.6
---------------------------------------
4R File: Sample05\Hongroise_4.wav
4R Rating: 4.8
4R Comment: very little pre-echo at sec 22-23
---------------------------------------
5L File: Sample05\Hongroise_1.wav
5L Rating: 2.5
5L Comment: sec 15
---------------------------------------

ABX Results:
Original vs Sample05\Hongroise_2.wav
   1 out of 1, pval = 0.5
Original vs Sample05\Hongroise_1.wav
   5 out of 5, pval = 0.031


I find this sample particulary interesting. All codecs had problems at different sections. And looking at the bitrate table,
Code: [Select]
            Nero  Real  Faac  Comp!  iTunes
Hongroise   148   128   105    123    128

I am quite shocked by Nero's performance. Faac on the other hand got a respectable rating, considering its low bitrate.

  • spoon
  • [*][*][*][*][*]
  • Administrator
AAC at 128kbps v2 listening test - FINISHED
Reply #174
Quote
Nope. Faac and Compaact! are VBR, the settings I used output average 128kbps over a very big amount of tracks, and still they behaved, staying at 128kbps +-4

I need to publish the bitrate deviation table ASAP >_<

The audio tracks I compresses were a mix of Pop, Rock, Rap - popular tracks. I could upload a playlist if required.

If you were to publish the bitrates over 2GB then any squabbling should stop, anyone looking at the results page for the first time could get the wrong impression that Nero was encoded too high.