Skip to main content

Topic: AAC at 128kbps test - FINISHED (Read 52367 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
I'd like to announce that my first listening test successfuly ended.

The results page is available here:
http://rarewares.hydrogenaudio.org/test/aa...st/results.html
(veteran forum members will notice that page resembles to someone else's)

So far, you only get results for each sample, and overall results.

Tomorrow, I'll add fancy things like user comments, bitrates, etc.

If someone notices flaws or typos, please report in this thread or through PM. Thanks a lot.
(Also, info on the 41_30sec sample is welcome)

For those really hurried, here is the overall plot:


Kind regards;

Roberto Amorim.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • sven_Bent
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #1
wow
QT really shines wonder how it compares to psytell -streaming.
Sven Bent - Denmark

AAC at 128kbps test - FINISHED
Reply #2
Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Interesting test. I guess we shouldn't be too surprised to see how well the QuickTime's AAC did. I was a little surprised to see FAAC getting beaten so badly though. Would you say it's still a better alternative to 128kbps MP3? (I assume we're using this bitrate for the purpose of portables use.)

Quote
If you don't mind about illegality, I suggest using AACenc, since it has good quality and is free.


This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?)  Or is it just a licensing issue?
Folding@Home HydrogenAudio.org Team ID# 32639
http://folding.stanford.edu/

AAC at 128kbps test - FINISHED
Reply #3
Quote
(Also, info on the 41_30sec sample is welcome)

the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf

regards; ilikedirt

  • dev0
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps test - FINISHED
Reply #4
Very interesting, especially considering how bad Dolby did in the 64kbps test.
Overall I was quite suprised by the high quality of all codecs except for FAAC, but there definetly is still room for massive improvments. None of the codecs was transparent to a degree where I'd consider using it for archival purposes, but for portable/casual use, which is the main field of usage at 128kbps anyway, AAC looks very interesting.

Thanks to Roberto and everyone else, who helped and participated, for investing time and work on this test. I can't wait for the next test to see how AAC compares to Vorbis, Musepack and MP3...

dev0
  • Last Edit: 16 June, 2003, 03:56:18 AM by dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

  • den
  • [*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #5
I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.

As for FAAC vs mp3 128 kbit, I haven't done a direct back to back comparison on identical samples, but some of the FAAC samples in this test were really bad, as can be seen from the charts. I'm relatively new to codec listening tests, but in some FAAC samples, the voices/instruments actually sounded different in their tonality from the others, let alone pre-echo or other more typical codec introduced artifacts. 

Den.

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps test - FINISHED
Reply #6
Quote
I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.


There is a ongoing improvement to Ahead Nero  "streaming" preset as well,  some changes already were made (still not out) as long as some grouping improvements and bug fixes. This will probably be available in the next web version of the Nero.

  • Dalkus
  • [*]
AAC at 128kbps test - FINISHED
Reply #7
Interesting results. Does this test say anything about how the decoders are doing with VBR files in that bit rate area? (i.e. the encoders that are able to produce VBR files). I'm thinking if 128 CBR isn't  good enough for portable use, maybe ~128 VBR is?

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #8
Quote
I am also interested to see how aacenc -streaming compares to QT.

I did some preliminary tests, and I compared blindly QT 6.3 to PsyTEL & Nero -streaming. Tested on 6 samples of this test.
Difference is impressive. Most audible one is the higher lowpass, which provide a richer sound than QT (appreciable on Atrain and 41_30), and less artifacts in some case. Now, PsyTEL & Ahead Nero AAC encodings are close to QT (for my taste), and sometime better than QT (41_30 for exemple). Unfortunately, I didn't have too much time last week to test it further. And I broke my headphone. The new one is here now, and I have some time the 2 next days

My results are here :

http://membres.lycos.fr/guruboolez/AUDIO/a...128/tableau.txt

and for VBR :
http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt

  • bawjaws
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #9
minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

-- a happy Mac user B) off to harass Apple for VBR capabities

  • Paspro
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #10
What I find strange is the difference between the Sorenson (FhG Pro codec) and the QT (Dolby codec) results. I have the impression that the Dolby codec is based on the FhG codec with optimisations for speed rather than quality. Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting. Could it be that this setting disables all options which reduce the quality of the encoding in favour of speed?

Another interesting point is the following link from Dolby Labs:

http://www.dolbylabs.com/DolbyAAC/FAQ/

which contains the following note:

Quote
How does Dolby AAC differ from AAC?
Dolby AAC consists of a standard MPEG-AAC core plus proprietary enhancements that enable a wider frequency range and better audio quality at lower bit rates.


So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?
  • Last Edit: 16 June, 2003, 06:33:44 AM by Paspro

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #11
Quote
Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting.

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt

  • Paspro
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #12
Quote
Quote
Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting.

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt

Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #13
Quote
Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #14
Nice test, Roberto.

A few comments:

1.  I think classical and jazz could have been better represented.

2.  It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez.  You might link to his listening results.

3.  The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test.

4.  You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used.  The similar presentation format might make people think that all the analysis was identical.  The difference is mainly one about risk.  The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs.  On the other hand, it's more sensitive than the Tukey HSD.

5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.

6.  In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result.  Also, I'd like to see some mention of the results you threw out for rating the original less than 5.


Some ideas for a future test:

1.  Perhaps another call for samples -- classical and jazz samples -- would be profitable.

2.  You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA.  This can be done with a small filesize penalty using Sox.  That would help to keep the ratings in perspective.

3.  Verifying VBR average bitrates:  I think that this task could be split up among several people, each encoding whole albums with all codecs.

ff123

Edit:  Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.

Edit2:  The next test you'll probably want to be sure to check for level (volume) differences too.
  • Last Edit: 16 June, 2003, 07:14:47 AM by ff123

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #15
ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?
  • Last Edit: 16 June, 2003, 07:14:24 AM by Garf

  • Paspro
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #16
Quote
Quote
Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.

This is true. The QuickTime interface does not allow the batch processing of files and therefore it is not easy to use it for mass encoding of CDs. Also note that on MacOS X the iTunes application does not allow the use of the highest quality setting of the AAC codec and therefore the only option left for someone is to use the Ovolab AAChoo front-end

http://www.ovolab.com/

for batch encoding at any quality setting. This is what I use for my iPod encodings and it works well. Still, I hope that the next version of iTunes will allow the setting of the encoding quality. I also hope that the next QuickTime revision will support VBR AAC encodings...
  • Last Edit: 16 June, 2003, 07:14:26 AM by Paspro

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #17
Quote
ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?

The problem I was having with just using the data directly to get an overall summary is the idea that one sample might have a greater influence on the overall results than another.  For example if one sample had a clear winner with a mean of 4.5, but another sample had a different clear winner with a mean of 2.5, I was thinking that the first sample would be given more weight in the overall results.

However, perhaps that's not really an issue when using a "blocked" analysis, which is supposed to take care of such things.  Different listeners have varying internal quality scales.  So I suppose that's analagous to the different samples having varying difficulty levels.

In that case, I would just try the exact same ANOVA / Fisher LSD on the data directly.

Problem with bootstrap is that it doesn't easily give you the nice 95% confidence intervals for the graphs.  Also, it's similar to the Tukey HSD in being more conservative with risk (and less sensitive).

ff123

Edit:  Roberto, don't forget to include the ATrain and Layla numbers when you do this analysis
  • Last Edit: 16 June, 2003, 07:35:27 AM by ff123

  • Cobra
  • [*][*][*][*]
  • Banned
AAC at 128kbps test - FINISHED
Reply #18
Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?

  • dev0
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps test - FINISHED
Reply #19
Quote
Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?

Learn to read! It was mentioned in the original announcement, that the winner of the AAC test will compete against MP3, Musepack and Vorbis. I'd probably try GT2's 128kbps mode too.

dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #20
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.
  • Last Edit: 16 June, 2003, 08:09:29 AM by Garf

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #21
Quote
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).

  • feces1223
  • [*][*][*]
  • Banned
AAC at 128kbps test - FINISHED
Reply #22
Yeah I'd Never Expect QT to come in as top cuz i thought there audio support was crap. The whole domination over movie trailers is pretty good though (QT trailers dominate WMP or REAL). Do you think Apple Store rips their albums using Quicktime? And which version and up supports AAC ripping (obviously Pro)?

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #23
Quote
Quote
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).

Yes, but doesn't GT2's 128kbps mode give more than 128k on average?

GT2 128kbps is Vorbis RC2 128kbps with lossless channel coupling

I would be surprised if that's still state-of-the-art as far as Vorbis at the most common bitrate goes.
  • Last Edit: 16 June, 2003, 08:36:43 AM by Garf

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #24
Quote
Yes, but doesn't GT2's 128kbps mode give more than 128k on average ?

Don't really know. I haven't played a lot with vorbis, especially at this bitrate.
I encoded the 12 samples with GT2 (bitrate is of course not representative of a general behaviour) :

Code: [Select]
001. 41_30sec :       150 kbps
002. ATrain :         144 kbps
003. BeautySlept :    123 kbps
004. Blackwater :     130 kbps
005. death2 :         106 kbps
006. FloorEssence :   132 kbps
007. Layla :          152 kbps
008. LifeShatters :   138 kbps
009. MidnightVoyage : 148 kbps
010. thear1 :         142 kbps
011. TheSource :      128 kbps
012. Waiting :        148 kbps



==> 136.75 kbps (according to foobar2000)