Skip to main content

Topic: AAC at 128kbps v2 listening test - FINISHED (Read 48871 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #50
In any case, since I can't find the slashdot thread, the response to the criticism of choosing 95% confidence over 99% confidence is that I don't know what he's so uptight about.  We're not talking about introducing a new drug here.  This is a listening test, for God's sake!  95% confidence is a standard accepted threshold, and it was properly set prior to the test.  As an engineer I think this physicist should focus less on theory and more on practicality.

I wish that slashdotters could discern the real weaknesses in the test, which were already mentioned here:

1. not enough samples to be definitive
2. different set of samples could produce different results
3. highly discriminating listeners could spread the results more

ff123

  • seanyseansean
  • [*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #51
Quote
What is the link to the slashdot thread?

Sorry it took so long, the link is here

sean

p.s. I post on there as 'doofusclam'...

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #52
Quote
But seriously now, I'm a bit disappointed of AAC. With some samples (Sample 1, most noticeably) the codecs still all sounded worse to me than Lame at 128kbps. And 128kbps mp3 is not even supposed to be any competition for AAC at that bitrate. AAC is supposed to deliver equivalent quality at a lower bitrate, after all. And that advantage is just not huge at the moment. Maybe it can compete with mp3 at 160kbps, but it's no match for Lame APS. It's not such a big surprise then that modern formats still haven't really managed to replace mp3. A 25% advantage is just not enough to justify a switch in standards. I think the new codecs have to deliver the same quality as mp3 at half the bitrate before they will become a real danger to mp3's popularity.

I'm sorry, but I think the facts don't agree with you.

Evidence 1: Results of previous 128k extension test.

Evidence 2: Results of previous 64k test.

Conclusion: Modern AAC codecs are much superior to MP3 at the same bitrate and _approach_ similar efficiency at half the bitrate.

I think the chance is very good that HE-AAC + Parametric Stereo equals 128k MP3 at 64 kbps.
  • Last Edit: 01 March, 2004, 11:41:36 AM by Garf

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #53
Let me try to reply to everything:

-Including Nero and iTunes in the multiformat test, probably getting rid of the anchor in the process since I won't accept more than 6 codecs:
Don't even start me.

-About Slashdot:
 

-About AAC being mature enough:
I strongly believe all these encoders performed wery well, considering the bitrate they were tested at. All of them are above or close to the ITU transparency cutoff, which surely wasn't the case in the MP3 test. Actually, if we compare graphs (I know that's not a good idea), the MP3 winner gets tied to the AAC loser.

Did I miss anything?

I will post the bitrate table later today.

Regards;

Roberto.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #54
Quote
Actually, if we compare graphs (I know that's not a good idea), the MP3 winner gets tied to the AAC loser.

In my opinion, AAC is now competitive with MP3, and the best AAC encoders are surely superior to LAME at 128 kbps on most case. The current problem don't lie in overall performances, but on occasionnal artifacts. Faac, Real, Compaact have still serious issues. Too many for my taste... I've also realized that when I've tested many AAC encoders and compared them to mp3 lame (OK, it was only one single musical genre, but there's a great diversity of instruments within): mp3 superiority on faac or Nero AAC was not something exceptionnal.

The next multiformat test will surely give a better idea on mp3/aac difference.

I don't say that mp3 is a reference, and that AAC isn't a great progress. But I'm more disappointing when I hear an AAC problems than with a mp3 artifact. Just because AAC seems to have a great potential, and that current implemtation are surely far from it. I suppose that in two or three years, quality will be amazing

  • Dologan
  • [*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #55
Quote
3. highly discriminating listeners could spread the results more

Hmm... how would the results look like if people scoring 5.0 (or more than one 5.0) on a particular sample were not counted? The number of listeners would definitely be not enough to draw statistically valid results, but I would be curious to know if it would look qualitatively different.

  • karl_lillevold
  • [*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #56
Quote
2) real knows why they offer 192kbps aac in their music store (sorry karl  )
for a big company wanting to compete against itunes it surely needs to offer a higher bitrate to keep up qualitywise (but well against all the wma stores around 128kbps real would also have been enough i assume  )

I find it very interesting to view my own scores for "Waiting" (sample 12), considering I do not have very experienced or expert ears, or have listened much to any of these codecs before. I did however borrow a good quality DAC and headphones.

Quote
1 = Nero
2 = Real
3 = Faac
4 = Compaact!
5 = iTunes


1: 3.0 (Nero)
2: 4.0 (Real)
3: 2.5 (Faac)
4: 3.5 (Compaact!)
5: 4.2 (iTunes)

I apologize if I misunderstood, misread, or mixed up something, but based on these numbers, it seems I personally did not like Faac or Nero for "Waiting", while iTunes did the best. I do remember Faac and Nero stood out as clearly the worst for this clip to my ears.

Preferences vary, and if anything, even though 128 kbps for me is close to transparency for most clips, at 192 kbps I know it would sound transparent, and if I had to choose to purchase at 192 kbps or 128 kbps, the overall results tell me that I would be confident in whichever codec is chosen, but rather select based on bitrate.

(I tried other clips that Waiting as well, but with the time I had, I had trouble hearing differences)
Sr. Codec Engineer (video) | RealNetworks Codec Group | helixcommunity.org 
This information is provided "AS IS" with no warranties,  grants no rights, and reflects my personal opinion.

  • schnofler
  • [*][*][*]
  • Developer
AAC at 128kbps v2 listening test - FINISHED
Reply #57
Quote
Quote
But seriously now, I'm a bit disappointed of AAC. With some samples (Sample 1, most noticeably) the codecs still all sounded worse to me than Lame at 128kbps. And 128kbps mp3 is not even supposed to be any competition for AAC at that bitrate. AAC is supposed to deliver equivalent quality at a lower bitrate, after all. And that advantage is just not huge at the moment. Maybe it can compete with mp3 at 160kbps, but it's no match for Lame APS. It's not such a big surprise then that modern formats still haven't really managed to replace mp3. A 25% advantage is just not enough to justify a switch in standards. I think the new codecs have to deliver the same quality as mp3 at half the bitrate before they will become a real danger to mp3's popularity.

I'm sorry, but I think the facts don't agree with you.

Evidence 1: Results of previous 128k extension test.


Yes, these results show that AAC is clearly superior to mp3 at 128kbps. I don't doubt that. My point is, that's not enough. Unfortunately, we don't have any public listening test results comparing AAC at 128kbps to mp3 at a higher bitrate (similar to the 64kbps test). However, I stand by my claim that AAC's advantage in this higher bitrate range is not huge. The codecs were easily discernible on many of the samples and had big problems on some (for me these were the samples 1, 3, 8, 11, as you can see from my results (anon02, for some reason), but I realize that I don't agree with the majority vote there). For me at least, this is not the case with Lame APS, which is usually transparent to my ears and doesn't quite reach twice the bitrate of this test. From this I drew the conclusion that in this bitrate range AAC still has quite a way to go until it can deliver same quality at half the bitrate.

Quote
Evidence 2: Results of previous 64k test.

Conclusion: Modern AAC codecs are much superior to MP3 at the same bitrate and _approach_ similar efficiency at half the bitrate.

I think the chance is very good that HE-AAC + Parametric Stereo equals 128k MP3 at 64 kbps.

There, you have a point. I'm almost sure that for my hearing HE-AAC + Parametric Stereo will indeed surpass 128kbps mp3 (in my personal results from the 64kbps test, even HE-AAC without PS tied to Lame at 128kbps). And I agree with you that HE-AAC's advantage over mp3 is indeed stunning, even more so if you decrease the bitrate even further (I was shocked when I first tried HE-AAC at around 30kbps).
However, practically (as opposed to technically) HE-AAC is a completely different format than LC AAC, which is why I regard discussions about AAC (as if it was only a single format) as flawed.

Thus, my conclusion is: HE-AAC, in its native bitrate range, is much superior to mp3 at the same bitrate and very close to similar efficiency at half the bitrate.
LC AAC is clearly superior to mp3 at the same bitrate but far from approaching mp3's efficiency at half the bitrate.

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #58
Quote
(anon02, for some reason)

In the next test, it'll be "listenerXX". It's still from the days I added all login names by hand to the filenames when the user asked to be associated with his results.

Quote
LC AAC is clearly superior to mp3 at the same bitrate but far from approaching mp3's efficiency at half the bitrate.


Well, according to MPEG's goals, LC AAC is not meant to sound the same as MP3 at half the bitrates, but at 30% smaller bitrates.  And I reckon the best AAC implementations deliver this quality.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #59
Quote
I think the chance is very good that HE-AAC + Parametric Stereo equals 128k MP3 at 64 kbps.

Will Parametric Stereo be useful at 64 kbps?
I sometimes read that the tool is efficient at very low bitrate only (24-48 kbps).


http://www.hydrogenaudio.org/forums/index....ndpost&p=178765
http://www.hydrogenaudio.org/forums/index....ndpost&p=178100
http://www.audiocoding.com/phorum/read.php...4453#reply_4495

AAC at 128kbps v2 listening test - FINISHED
Reply #60
Quote
Well, according to MPEG's goals, LC AAC is not meant to sound the same as MP3 at half the bitrates, but at 30% smaller bitrates.  And I reckon the best AAC implementations deliver this quality.

How did you come to this conclusion? If that was true then best LC-AAC at 128kbps quality should be close to LAME --alt-preset standard -Y quality (~166kbps?). It will be really interesting to see a valid listening test whether that industry hype holds water.
-- Floydian Slip

  • schnofler
  • [*][*][*]
  • Developer
AAC at 128kbps v2 listening test - FINISHED
Reply #61
Quote
In the next test, it'll be "listenerXX". It's still from the days I added all login names by hand to the filenames when the user asked to be associated with his results.

From what you wrote in the readme file, I just got the impression you would name the results files according to what was entered in the listener text field. Anyway, doesn't matter really.

Thanks again for your work. It's interesting and enlightening as usual.

Quote
Well, according to MPEG's goals, LC AAC is not meant to sound the same as MP3 at half the bitrates, but at 30% smaller bitrates. And I reckon the best AAC implementations deliver this quality.

Aah, I see, so the flaw is in the concept, not the execution 

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #62
Quote
Will Parametric Stereo be useful at 64 kbps?
I sometimes read that the tool is efficient at very low bitrate only (24-48 kbps).


http://www.hydrogenaudio.org/forums/index....ndpost&p=178765
http://www.hydrogenaudio.org/forums/index....ndpost&p=178100
http://www.audiocoding.com/phorum/read.php...4453#reply_4495

It's probably at the upper edge of the usefulness.

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #63
Quote
How did you come to this conclusion? If that was true then best LC-AAC at 128kbps quality should be close to LAME --alt-preset standard -Y quality (~166kbps?). It will be really interesting to see a valid listening test whether that industry hype holds water.

OMG! It's the guy that hates my tests because they bring down HA's quality standards.

To start with, I said "I reckon". Don't come being smart assed on me, I didn't conclude anything as you are saying. Second, I evaluated that by looking at the superior score iTunes gets compared to Lame and other MP3 encoders.

If you want a "conclusion" feel free to conduce a test and while at it, raise HA's quality standards that I shamefully brought down.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #64
Quote
From what you wrote in the readme file, I just got the impression you would name the results files according to what was entered in the listener text field. Anyway, doesn't matter really.

It would be doable, but would also be a pain :B

Maybe you can add a routine to the decryption module that searches for listener name and adds it to the decrypted filename?
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #65
Quote
How did you come to this conclusion? If that was true then best LC-AAC at 128kbps quality should be close to LAME --alt-preset standard -Y quality (~166kbps?). It will be really interesting to see a valid listening test whether that industry hype holds water.

128kbps LC-AAC being comparable with 160kbps MP3?

Certainly! I don't think thats 'hype' at all.
  • Last Edit: 01 March, 2004, 01:01:22 PM by Garf

AAC at 128kbps v2 listening test - FINISHED
Reply #66
"However, practically (as opposed to technically) HE-AAC is a completely different format than LC AAC, which is why I regard discussions about AAC (as if it was only a single format) as flawed."  ~schnofler

I personaly disagree, since the MPEG chose SBR as a method to improve quality knowing full well that it would be somewhat backward compatible, i.e. an LC AAC codec can play back an HE-AAC file, albeit without the SBR, making it an approximately interopperable solution. Try it, it's fun; Listening to an HE-AAC track on quicktime and then listening to it with COREAAC.ax, i keep showing it off to my friends (depressing when they can't tell the difference).

I don't know if the same thing goes for PS

AAC at 128kbps v2 listening test - FINISHED
Reply #67
Quote
OMG! It's the guy that hates my tests because they bring down HA's quality standards.

Don't get personal here. I am just wondering whether the industry statement "LC-AAC can achieve similar quality level at 30% less than MP3's bitrate" is a hype.
-- Floydian Slip

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #68
Quote
I don't know if the same thing goes for PS

It'll play back mono. So it's backwards compatible.

But I think the difference between supported or not will be more easily heared than with HE-AAC, of course.
  • Last Edit: 01 March, 2004, 01:13:40 PM by Garf

AAC at 128kbps v2 listening test - FINISHED
Reply #69
Quote
Quote
How did you come to this conclusion? If that was true then best LC-AAC at 128kbps quality should be close to LAME --alt-preset standard -Y quality (~166kbps?). It will be really interesting to see a valid listening test whether that industry hype holds water.

128kbps LC-AAC being comparable with 160kbps MP3?

Certainly! I don't think thats 'hype' at all.

Ummm, I am not that sure. Actually to be fair, to get around 30% less bitrate you have to compare 180kbps LAME with 128kbps LC-AAC. That's why I mentioned LAME --alt-preset standard -Y which will be a valid bitrange to compare.
-- Floydian Slip

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #70
Quote
Ummm, I am not that sure. Actually to be fair, to get around 30% less bitrate you have to compare 180kbps LAME with 128kbps LC-AAC. That's why I mentioned LAME --alt-preset standard -Y which will be a valid bitrange to compare.

--preset standard -Y is one of the highest quality level of mp3 ever reached by mp3 format. For good comparison, wait three years: good AAC encoder, with well-tuned VBR mode, will probably be competitive and surely better (think about smearing) than mp3.

If you want a good comparison, try to compare iTunes 128 and Fastenc at 180 kbps. It wouldn't surprise me to see AAC sounding better.

  • schnofler
  • [*][*][*]
  • Developer
AAC at 128kbps v2 listening test - FINISHED
Reply #71
Quote
Quote

From what you wrote in the readme file, I just got the impression you would name the results files according to what was entered in the listener text field. Anyway, doesn't matter really.


It would be doable, but would also be a pain :B

Maybe you can add a routine to the decryption module that searches for listener name and adds it to the decrypted filename?

Hehe, damn, dug my own hole there, didn't I?
Anyway, I was thinking about changing the results file format altogether, XML possibly. It's a lot easier to parse and process, and for the 64kbps test phong made that nifty script which produced browsable html pages (here), which is much nicer than reading text files anyway, so maybe we can dispense with text files altogether.

Quote
I personaly disagree, since the MPEG chose SBR as a method to improve quality knowing full well that it would be somewhat backward compatible, i.e. an LC AAC codec can play back an HE-AAC file, albeit without the SBR, making it an approximately interopperable solution. Try it, it's fun; Listening to an HE-AAC track on quicktime and then listening to it with COREAAC.ax, i keep showing it off to my friends (depressing when they can't tell the difference).

Well, if your friends' ears are that terrible, they probably won't hear a difference between mp3 and HE-AAC anyway. I mean, get serious: HE-AAC played back by an LC-AAC decoder is lowpassed at 9kHz (!). Even if the remaining bandwidth is reproduced perfectly, this still sounds just plain terrible. "Approximately interoperable" is extremely euphemistic in these circumstances.
  • Last Edit: 01 March, 2004, 01:45:25 PM by schnofler

AAC at 128kbps v2 listening test - FINISHED
Reply #72
Quote
If you want a good comparison, try to compare iTunes 128 and Fastenc at 180 kbps. It wouldn't surprise me to see AAC sounding better.

To get a meaningful conclusion you have to compare best encoders of the both formats. That's why I am saying to compare LAME --alt-preset standard -Y  with whatever best LC-AAC encoder at 128kbps (currentlly looks like iTune). IMHO, comparing iTune with Fastenc won't be a good choice. But even with vbr mode, it would be interesting to see whether current best LC-AAC at 128kbps vbr can stand up against LAME --alt-preset standard -Y.
-- Floydian Slip

  • music_man_mpc
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps v2 listening test - FINISHED
Reply #73
Quote
1) iTunes result is nice, but I think our progress is even nicer! In the previous test iTunes clearly beat Nero. Now it's getting quite arguable whether it's really better  Next major Nero release will be very intresting.

Wasn't CBR used for Nero in the last test?  I'm not trying to say that Nero hasn't  made any progress, I am sure that it has, but wouldn't the CBR - VBR issue be part of Nero's better score in this test?
  • Last Edit: 01 March, 2004, 01:50:55 PM by music_man_mpc
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame

  • [proxima]
  • [*][*][*]
AAC at 128kbps v2 listening test - FINISHED
Reply #74
Quote
The current problem don't lie in overall performances, but on occasionnal artifacts. Faac, Real, Compaact have still serious issues.

I agree with you.
Quote
The next multiformat test will surely give a better idea on mp3/aac difference.

As you already said, there is no doubt that @128 kbps the best AAC implementation is superior to the best MP3 implementation. The next multiformat test will confirm what we already know. So, i understand your desire to see Ahead AAC codec but i'm afraid this could not happen because lack of space. Maybe we can exclude iTunes: i see no point to compare again two codecs (LAME and iTunes) that, quality speaking, remained almost the same after the last multiformat test.

I think we are speaking of AAC vs MP3 in a wrong manner here. There are mature and bad MP3 encoders and there are good and "not yet mature" AAC encoders. The maturity of an AAC encoder should be evaluated taking into account the potential. If all codecs tested were "mature" we could say for sure that Itunes one is the "more mature", so (according to the last multiformat test) we can conclude that a mature AAC encoder is equal to Musepack performance @128 kbps....and i've read many time from Frank Klemm itself that musepack PNS implementation is in "alpha stage" !!!. Can we conclude this ?? I simply think we have reached an absurd

From what i've read, AAC is considered the state of art and i simply think that it has the potential to be much better than today.
WavPack 4.3 -mfx5
LAME 3.97 -V5 --vbr-new --athaa-sensitivity 1