Skip to main content

Topic: AAC at 128kbps test - FINISHED (Read 52511 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • dev0
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps test - FINISHED
Reply #25
I can only support Guru's comment: GT2's 128kbps mode seems to be the most transparent Vorbis encoder at around 130kbps at least in the few tests I've done. Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0
"To understand me, you'll have to swallow a world." Or maybe your words.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #26
Quote
Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0

I can't imagine a big listening test in july 2003, including an early codec, dated from november 2001. Most people will use a CVS or PostCVS (never heard anything different) version for their ~128 encodings.

Nevertheless, a dedicated listening test between GT2 and 1.0 should be started. If conclusions are in favor of the old codec version, it may convinced Xiph people of some flaws in their 'final' codec. The noise issue is acceptable at low bitrate (50-90 kbps), and I prefered some hiss that metallic distorsion. But at ~120 kbps, I prefer some occasional distorsion than a continuous noise, perceptible on most music. Especially when I tested QT encondings, hiss-free and with very few artifacts.
  • Last Edit: 16 June, 2003, 10:42:37 AM by guruboolez

  • treech
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #27
AACenc = Sorenson ?

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #28
Quote
AACenc = Sorenson ?

PsyTEL AACenc.exe version 2.15

  • treech
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #29
ahh, ok thanks

also, would interresting to se if/how much worse psytel fastenc is compared to the normal psytel...

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #30
Quick and dirty new analysis, in response to complaints about the "eyeball" method of ranking

Substituting all 4.5's for the two samples in which means are not shown, I quickly came up with the following graph for the overall results, using ANOVA / Fisher LSD instead of the "eyeball" ranking method:

http://ff123.net/export/aac.png

The rankings are about the same as before, with QT the clear winner, FAAC the clear loser, and the rest tied. However, this method has the advantage of showing how close the codecs are to the reference score of 5.0

In a future test, I recommend using an anchor (filtered version of the original) to put things in better perspective.

Edit:  the graph label should be corrected to read "Rating" instead of "Ranking"
  • Last Edit: 16 June, 2003, 11:53:15 AM by ff123

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #31
Back to some results.

I'm really surprised to see BeautySlept notation. I tested this sample in two test, this one and a preliminary one.

Code: [Select]
              General    Personal    Previous
Quicktime     4.42        3.7          4.2
Ahead Nero    3.81        1.4          2.1 *
PsyTEL        4.25        2.9          1.8
Sorenson      4.26        2.5          2.5
FAAC          3.92        2.0          --- **

[span style='font-size:8pt;line-height:100%']* Ahead MP4 Codec was an older version
** FAAC was not tested previously
[/span]
I systematically found all encoding, except QuickTime, more than annoying. And I can't bear Ahead codec distorsions ! Even with VBR (-streaming) , Ahead was rated 2.5 and PsyTEL 3.5 (QT = 5.0 on this test !).
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ? Or maybe headphone issues ? In fact, I tried to perform this test with a set of poor earbuds (Sennheiser MX-500), began with BeautySlept sample, and found all codecs, including Ahead & FAAC, to be near transparency ! I gave up on this result... With a good set of headphone, the distorsion was not only easily perceptible, but awfully annoying.

P.S. According to the general conclusion, BeautySlept is one of the two samples where FAAC provide a better sound than Nero encoding (3.81 vs 3.92  -  second one is LifeShatters, 3.88 vs 3.89)
  • Last Edit: 16 June, 2003, 11:43:22 AM by guruboolez

  • ckjnigel
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #32
Thanks to Roberto and know that my admiration continues to grow.  I think this is fascinating.
1) Any speculation as to why ATrain and Layla were so easy for AAC?  Surely they were chosen because the samples were challenging to MP3, Ogg and MPC.  (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?
3) One guesses that Apple is devoting considerable resources to further development.  Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has  and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support  for MPC.  "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public.  "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.
  • Last Edit: 16 June, 2003, 01:18:09 PM by ckjnigel

  • Ivan Dimkovic
  • [*][*][*][*][*]
  • Developer
AAC at 128kbps test - FINISHED
Reply #33
Guruboolez,    ask Menno for the internal NERO plug-in update - I think I found the reason for the BeautySlept problem, and it should be much better now - 

File was ended up heavily undercoded at 112 kbps - clearly a CBR bug (which is being fixed now)

Quote
1) Any speculation as to why ATrain and Layla were so easy for AAC? Surely they were chosen because the samples were challenging to MP3, Ogg and MPC. (I'm guessng the next test will address issues of what samples are handled relatively better or worse by the codecs.)


Atrain has relatively low perceptual entropy, making it easy to encode at 128 kbps for AAC - also, most passages are "noisy" which gives more masking abilities for the psych model.  Same goes for layla (noisy)

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #34
Quote
Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Fixed. Thank-you very much.

Quote
This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?)   Or is it just a licensing issue?


Only licensing issues. It's not warez, or pirate, because the copyright owner (Ivan) allowed it to be distributed.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #35
Quote
the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf

regards; ilikedirt

Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #36
Quote
minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

Fixed (i think)

Thanks a lot.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #37
Quote
So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?

It's a FhG codec improved for speed. And, miraculously, it also improves quality while at it. :B
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #38
Quote from: ff123,Jun 16 2003 - 08:08 AM
1.  I think classical and jazz could have been better represented.

Well, Lizst and Bach were removed because people guessed they would be too easy, and therefore not significative.

Besides, we needed to make space for problem samples, and JohnV wouldn't accept a test with more than 12 samples.

Quote
2.  It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez.  You might link to his listening results.


Addressed. Thanks.

Quote
3.  The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test.


He didn't participate in this one already. :-P

Quote
4.  You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used.  The similar presentation format might make people think that all the analysis was identical.  The difference is mainly one about risk.  The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs.  On the other hand, it's more sensitive than the Tukey HSD.


Addressed

Quote
5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.


Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

Quote
6.  In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result.  Also, I'd like to see some mention of the results you threw out for rating the original less than 5.


OK, I will do that.

Of course, I won't make public the names of the participants that rated the reference. That wouldn't be nice to them, I believe.

In your test, you had people rating the reference too?


Quote
1.  Perhaps another call for samples -- classical and jazz samples -- would be profitable.


Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

Quote
2.  You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA.  This can be done with a small filesize penalty using Sox.  That would help to keep the ratings in perspective.


OK, I think I'll do that. Thanks for the idea.

Quote
3.  Verifying VBR average bitrates:  I think that this task could be split up among several people, each encoding whole albums with all codecs.


Yes, I plan to start a taskforce on this later this month.

Quote
Edit:  Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.


Done

Quote
Edit2:  The next test you'll probably want to be sure to check for level (volume) differences too.


OK, I will.

Regards;

Roberto.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • NickSD
  • [*][*][*]
AAC at 128kbps test - FINISHED
Reply #39
Quote
Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?

I don't know who submitted it, but the style would be something along the lines of jazz-rock.  Dave Matthews Band is very hard to categorize... (Incredible live band, by the way.)

Nick

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #40
Quote
Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

That argument simply doesn't work at all.

If you believe another set of samples would produce a significant performance difference between the AAC codecs compared to what we've seen here, it follows directly from that the next test would be highly unfairly biased towards AAC if you did use the same samples.
  • Last Edit: 16 June, 2003, 02:10:23 PM by Garf

  • niktheblak
  • [*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #41
Quote
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ?

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I have to ask before I lose my mind, did the BeautySlept sample originate from me (in this thread) or is it a funny coincidence

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #42
Quote
1) Any speculation as to why ATrain and Layla were so easy for AAC?  Surely they were chosen because the samples were challenging to MP3, Ogg and MPC.  (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)

No, they were chosen because they were featured in ff123's 64kbps test.

I will probably ditch them for the extension test though (thanks for the clarification, Garf)

Quote
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?


No idea.

Quote
3) One guesses that Apple is devoting considerable resources to further development.  Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?


There's another developer, that actually offered to send me a demo encoder that I can use in some future listening test, but I don't know if I can decline informations now.

What prevents dolby from doing what?

Quote
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has  and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support  for MPC.  "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public.  "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.


There's no standardized DRM system for AAC. And no, none of these encoders offer the option of DRM'ing your AACs, differently from WMA/Windows Media Encoder.

Regards;

Roberto.

@NickSD: Thanks for the info.
  • Last Edit: 16 June, 2003, 02:44:17 PM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
AAC at 128kbps test - FINISHED
Reply #43
Quote
Quote
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ?

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I was usually embarassed by the synthetic consistency of this harpsichord (and by the metronomic performance of the player). Thank you for this confirmation
Here is a real harpsichord sample. Bitrate is higher than with BeautySlept (265 kbps with --standard and 1.15r). It's a real killer too for most codec, if you want to reach 128 kbps and not more. I strongly suggest anyone to test it with WMA9 (PRO), incredibly transparent at 128 kbps compared to all other format/codecs I heard.

http://membres.lycos.fr/guruboolez/AUDIO/s...chpsichord.flac

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #44
Quote
Quote
5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.


Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

I think the best way is to just use the same procedure that you did for each sample.  But you'll have to calculate the mean scores for ATrain and Layla first (easiest to just use Excel for that).

ff123

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
AAC at 128kbps test - FINISHED
Reply #45
Regarding 41_30sec.wav

Quote
2 more pieces of info: who submitted it (if it's known), and what's the style?

Originally submitted by Filburt

style:  rock
  • Last Edit: 16 June, 2003, 03:50:21 PM by ff123

  • ErikS
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #46
Does ANOVA take into consideration the varying scales that people use when they grade the test samples? For example someone who thinks that the goal of the codec is to achive total transparancy may grade everything between 1 and 3 when someone else who doesn't care that much would give grades between 4 and 5 to the same set.

So could someone very briefly explain how ANOVA works (or post some good links)? Which types of data sets is it possible to use ANOVA on?

  • Agent86
  • [*][*]
AAC at 128kbps test - FINISHED
Reply #47
Does anyone know anything about the AAC codec that XM is using to broadcast its satellite radio?

XM AAC Information

It apppears as though it is heavily customized, but I wonder if it originally came from anything used in this listening test.

- Agent 86

  • rjamorim
  • [*][*][*][*][*]
AAC at 128kbps test - FINISHED
Reply #48
XM is probably using gear from Thales
http://www.thales-bm.com/
Probably this baby.
It's the same gear most DRM stations are using.

Thales uses AAC encoder from FhG and SBR encoder from CT. Probably  integerized versions.
  • Last Edit: 16 June, 2003, 05:16:57 PM by rjamorim
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

AAC at 128kbps test - FINISHED
Reply #49
Very interesting.  I'm quite disappointed in Nero, yet another reason I just keep re-using their trial. :roll:

I'm starting to wish my player supported AAC now.