HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: rjamorim on 2003-06-16 08:12:51

Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 08:12:51
I'd like to announce that my first listening test successfuly ended.

The results page is available here:
http://rarewares.hydrogenaudio.org/test/aa...st/results.html (http://rarewares.hydrogenaudio.org/test/aac128test/results.html)
(veteran forum members will notice that page resembles to someone else's)

So far, you only get results for each sample, and overall results.

Tomorrow, I'll add fancy things like user comments, bitrates, etc.

If someone notices flaws or typos, please report in this thread or through PM. Thanks a lot.
(Also, info on the 41_30sec sample is welcome)

For those really hurried, here is the overall plot:


Kind regards;

Roberto Amorim.
Title: AAC at 128kbps test - FINISHED
Post by: sven_Bent on 2003-06-16 08:35:33
wow
QT really shines wonder how it compares to psytell -streaming.
Title: AAC at 128kbps test - FINISHED
Post by: ChrisGranger on 2003-06-16 08:39:14
Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Interesting test. I guess we shouldn't be too surprised to see how well the QuickTime's AAC did. I was a little surprised to see FAAC getting beaten so badly though. Would you say it's still a better alternative to 128kbps MP3? (I assume we're using this bitrate for the purpose of portables use.)

Quote
If you don't mind about illegality, I suggest using AACenc, since it has good quality and is free.


This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?)  Or is it just a licensing issue?
Title: AAC at 128kbps test - FINISHED
Post by: ilikedirtthe2nd on 2003-06-16 08:50:07
Quote
(Also, info on the 41_30sec sample is welcome)

the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf (http://www.allmusic.com/cg/amg.dll?p=amg&uid=MISS70305210557&sql=A4c6tk6dxqkrf)

regards; ilikedirt
Title: AAC at 128kbps test - FINISHED
Post by: dev0 on 2003-06-16 08:54:46
Very interesting, especially considering how bad Dolby did in the 64kbps test.
Overall I was quite suprised by the high quality of all codecs except for FAAC, but there definetly is still room for massive improvments. None of the codecs was transparent to a degree where I'd consider using it for archival purposes, but for portable/casual use, which is the main field of usage at 128kbps anyway, AAC looks very interesting.

Thanks to Roberto and everyone else, who helped and participated, for investing time and work on this test. I can't wait for the next test to see how AAC compares to Vorbis, Musepack and MP3...

dev0
Title: AAC at 128kbps test - FINISHED
Post by: den on 2003-06-16 08:55:19
I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.

As for FAAC vs mp3 128 kbit, I haven't done a direct back to back comparison on identical samples, but some of the FAAC samples in this test were really bad, as can be seen from the charts. I'm relatively new to codec listening tests, but in some FAAC samples, the voices/instruments actually sounded different in their tonality from the others, let alone pre-echo or other more typical codec introduced artifacts. 

Den.
Title: AAC at 128kbps test - FINISHED
Post by: Ivan Dimkovic on 2003-06-16 09:24:27
Quote
I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.


There is a ongoing improvement to Ahead Nero  "streaming" preset as well,  some changes already were made (still not out) as long as some grouping improvements and bug fixes. This will probably be available in the next web version of the Nero.
Title: AAC at 128kbps test - FINISHED
Post by: Dalkus on 2003-06-16 09:32:12
Interesting results. Does this test say anything about how the decoders are doing with VBR files in that bit rate area? (i.e. the encoders that are able to produce VBR files). I'm thinking if 128 CBR isn't  good enough for portable use, maybe ~128 VBR is?
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 09:36:13
Quote
I am also interested to see how aacenc -streaming compares to QT.

I did some preliminary tests, and I compared blindly QT 6.3 to PsyTEL & Nero -streaming. Tested on 6 samples of this test.
Difference is impressive. Most audible one is the higher lowpass, which provide a richer sound than QT (appreciable on Atrain and 41_30), and less artifacts in some case. Now, PsyTEL & Ahead Nero AAC encodings are close to QT (for my taste), and sometime better than QT (41_30 for exemple). Unfortunately, I didn't have too much time last week to test it further. And I broke my headphone. The new one is here now, and I have some time the 2 next days

My results are here :

http://membres.lycos.fr/guruboolez/AUDIO/a...128/tableau.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/tableau.txt)

and for VBR :
http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/tableau_vbr.txt)
Title: AAC at 128kbps test - FINISHED
Post by: bawjaws on 2003-06-16 11:11:55
minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

-- a happy Mac user B) off to harass Apple for VBR capabities
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-16 11:32:38
What I find strange is the difference between the Sorenson (FhG Pro codec) and the QT (Dolby codec) results. I have the impression that the Dolby codec is based on the FhG codec with optimisations for speed rather than quality. Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting. Could it be that this setting disables all options which reduce the quality of the encoding in favour of speed?

Another interesting point is the following link from Dolby Labs:

http://www.dolbylabs.com/DolbyAAC/FAQ/ (http://www.dolbylabs.com/DolbyAAC/FAQ/)

which contains the following note:

Quote
How does Dolby AAC differ from AAC?
Dolby AAC consists of a standard MPEG-AAC core plus proprietary enhancements that enable a wider frequency range and better audio quality at lower bit rates.


So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 11:37:44
Quote
Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting.

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/speed.txt)
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-16 11:58:05
Quote
Quote
Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting.

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/speed.txt)

Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 12:06:41
Quote
Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-16 12:08:25
Nice test, Roberto.

A few comments:

1.  I think classical and jazz could have been better represented.

2.  It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez.  You might link to his listening results.

3.  The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test.

4.  You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used.  The similar presentation format might make people think that all the analysis was identical.  The difference is mainly one about risk.  The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs.  On the other hand, it's more sensitive than the Tukey HSD.

5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.

6.  In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result.  Also, I'd like to see some mention of the results you threw out for rating the original less than 5.


Some ideas for a future test:

1.  Perhaps another call for samples -- classical and jazz samples -- would be profitable.

2.  You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA.  This can be done with a small filesize penalty using Sox.  That would help to keep the ratings in perspective.

3.  Verifying VBR average bitrates:  I think that this task could be split up among several people, each encoding whole albums with all codecs.

ff123

Edit:  Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.

Edit2:  The next test you'll probably want to be sure to check for level (volume) differences too.
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-16 12:13:35
ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-16 12:13:37
Quote
Quote
Hmm...  I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.

This is true. The QuickTime interface does not allow the batch processing of files and therefore it is not easy to use it for mass encoding of CDs. Also note that on MacOS X the iTunes application does not allow the use of the highest quality setting of the AAC codec and therefore the only option left for someone is to use the Ovolab AAChoo front-end

http://www.ovolab.com/ (http://www.ovolab.com/)

for batch encoding at any quality setting. This is what I use for my iPod encodings and it works well. Still, I hope that the next version of iTunes will allow the setting of the encoding quality. I also hope that the next QuickTime revision will support VBR AAC encodings...
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-16 12:26:17
Quote
ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?

The problem I was having with just using the data directly to get an overall summary is the idea that one sample might have a greater influence on the overall results than another.  For example if one sample had a clear winner with a mean of 4.5, but another sample had a different clear winner with a mean of 2.5, I was thinking that the first sample would be given more weight in the overall results.

However, perhaps that's not really an issue when using a "blocked" analysis, which is supposed to take care of such things.  Different listeners have varying internal quality scales.  So I suppose that's analagous to the different samples having varying difficulty levels.

In that case, I would just try the exact same ANOVA / Fisher LSD on the data directly.

Problem with bootstrap is that it doesn't easily give you the nice 95% confidence intervals for the graphs.  Also, it's similar to the Tukey HSD in being more conservative with risk (and less sensitive).

ff123

Edit:  Roberto, don't forget to include the ATrain and Layla numbers when you do this analysis
Title: AAC at 128kbps test - FINISHED
Post by: Cobra on 2003-06-16 12:55:40
Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?
Title: AAC at 128kbps test - FINISHED
Post by: dev0 on 2003-06-16 13:01:36
Quote
Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?

Learn to read! It was mentioned in the original announcement, that the winner of the AAC test will compete against MP3, Musepack and Vorbis. I'd probably try GT2's 128kbps mode too.

dev0
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-16 13:09:09
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 13:28:39
Quote
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).
Title: AAC at 128kbps test - FINISHED
Post by: feces1223 on 2003-06-16 13:33:41
Yeah I'd Never Expect QT to come in as top cuz i thought there audio support was crap. The whole domination over movie trailers is pretty good though (QT trailers dominate WMP or REAL). Do you think Apple Store rips their albums using Quicktime? And which version and up supports AAC ripping (obviously Pro)?
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-16 13:35:52
Quote
Quote
I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).

Yes, but doesn't GT2's 128kbps mode give more than 128k on average?

GT2 128kbps is Vorbis RC2 128kbps with lossless channel coupling

I would be surprised if that's still state-of-the-art as far as Vorbis at the most common bitrate goes.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 14:07:10
Quote
Yes, but doesn't GT2's 128kbps mode give more than 128k on average ?

Don't really know. I haven't played a lot with vorbis, especially at this bitrate.
I encoded the 12 samples with GT2 (bitrate is of course not representative of a general behaviour) :

Code: [Select]
001. 41_30sec :       150 kbps
002. ATrain :         144 kbps
003. BeautySlept :    123 kbps
004. Blackwater :     130 kbps
005. death2 :         106 kbps
006. FloorEssence :   132 kbps
007. Layla :          152 kbps
008. LifeShatters :   138 kbps
009. MidnightVoyage : 148 kbps
010. thear1 :         142 kbps
011. TheSource :      128 kbps
012. Waiting :        148 kbps



==> 136.75 kbps (according to foobar2000)
Title: AAC at 128kbps test - FINISHED
Post by: dev0 on 2003-06-16 15:15:27
I can only support Guru's comment: GT2's 128kbps mode seems to be the most transparent Vorbis encoder at around 130kbps at least in the few tests I've done. Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 15:41:14
Quote
Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0

I can't imagine a big listening test in july 2003, including an early codec, dated from november 2001. Most people will use a CVS or PostCVS (never heard anything different) version for their ~128 encodings.

Nevertheless, a dedicated listening test between GT2 and 1.0 should be started. If conclusions are in favor of the old codec version, it may convinced Xiph people of some flaws in their 'final' codec. The noise issue is acceptable at low bitrate (50-90 kbps), and I prefered some hiss that metallic distorsion. But at ~120 kbps, I prefer some occasional distorsion than a continuous noise, perceptible on most music. Especially when I tested QT encondings, hiss-free and with very few artifacts.
Title: AAC at 128kbps test - FINISHED
Post by: treech on 2003-06-16 16:06:00
AACenc = Sorenson ?
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 16:08:28
Quote
AACenc = Sorenson ?

PsyTEL AACenc.exe version 2.15
Title: AAC at 128kbps test - FINISHED
Post by: treech on 2003-06-16 16:20:26
ahh, ok thanks

also, would interresting to se if/how much worse psytel fastenc is compared to the normal psytel...
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-16 16:25:47
Quick and dirty new analysis, in response to complaints about the "eyeball" method of ranking

Substituting all 4.5's for the two samples in which means are not shown, I quickly came up with the following graph for the overall results, using ANOVA / Fisher LSD instead of the "eyeball" ranking method:

http://ff123.net/export/aac.png (http://ff123.net/export/aac.png)

The rankings are about the same as before, with QT the clear winner, FAAC the clear loser, and the rest tied. However, this method has the advantage of showing how close the codecs are to the reference score of 5.0

In a future test, I recommend using an anchor (filtered version of the original) to put things in better perspective.

Edit:  the graph label should be corrected to read "Rating" instead of "Ranking"
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 16:36:53
Back to some results.

I'm really surprised to see BeautySlept notation. I tested this sample in two test, this one and a preliminary one (http://membres.lycos.fr/guruboolez/zConclusion.txt).

Code: [Select]
              General    Personal    Previous
Quicktime     4.42        3.7          4.2
Ahead Nero    3.81        1.4          2.1 *
PsyTEL        4.25        2.9          1.8
Sorenson      4.26        2.5          2.5
FAAC          3.92        2.0          --- **

[span style='font-size:8pt;line-height:100%']* Ahead MP4 Codec was an older version
** FAAC was not tested previously
[/span]
I systematically found all encoding, except QuickTime, more than annoying. And I can't bear Ahead codec distorsions ! Even with VBR (-streaming) , Ahead was rated 2.5 and PsyTEL 3.5 (QT = 5.0 on this test !).
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ? Or maybe headphone issues ? In fact, I tried to perform this test with a set of poor earbuds (Sennheiser MX-500), began with BeautySlept sample, and found all codecs, including Ahead & FAAC, to be near transparency ! I gave up on this result... With a good set of headphone, the distorsion was not only easily perceptible, but awfully annoying.

P.S. According to the general conclusion, BeautySlept is one of the two samples where FAAC provide a better sound than Nero encoding (3.81 vs 3.92  -  second one is LifeShatters, 3.88 vs 3.89)
Title: AAC at 128kbps test - FINISHED
Post by: ckjnigel on 2003-06-16 16:53:04
Thanks to Roberto and know that my admiration continues to grow.  I think this is fascinating.
1) Any speculation as to why ATrain and Layla were so easy for AAC?  Surely they were chosen because the samples were challenging to MP3, Ogg and MPC.  (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?
3) One guesses that Apple is devoting considerable resources to further development.  Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has  and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support  for MPC.  "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public.  "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.
Title: AAC at 128kbps test - FINISHED
Post by: Ivan Dimkovic on 2003-06-16 17:06:02
Guruboolez,    ask Menno for the internal NERO plug-in update - I think I found the reason for the BeautySlept problem, and it should be much better now - 

File was ended up heavily undercoded at 112 kbps - clearly a CBR bug (which is being fixed now)

Quote
1) Any speculation as to why ATrain and Layla were so easy for AAC? Surely they were chosen because the samples were challenging to MP3, Ogg and MPC. (I'm guessng the next test will address issues of what samples are handled relatively better or worse by the codecs.)


Atrain has relatively low perceptual entropy, making it easy to encode at 128 kbps for AAC - also, most passages are "noisy" which gives more masking abilities for the psych model.  Same goes for layla (noisy)
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 18:21:30
Quote
Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Fixed. Thank-you very much.

Quote
This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?)   Or is it just a licensing issue?


Only licensing issues. It's not warez, or pirate, because the copyright owner (Ivan) allowed it to be distributed.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 18:24:42
Quote
the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf (http://www.allmusic.com/cg/amg.dll?p=amg&uid=MISS70305210557&sql=A4c6tk6dxqkrf)

regards; ilikedirt

Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 18:32:28
Quote
minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

Fixed (i think)

Thanks a lot.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 18:35:17
Quote
So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?

It's a FhG codec improved for speed. And, miraculously, it also improves quality while at it. :B
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 19:03:35
Quote from: ff123,Jun 16 2003 - 08:08 AM
1.  I think classical and jazz could have been better represented.

Well, Lizst and Bach were removed because people guessed they would be too easy, and therefore not significative.

Besides, we needed to make space for problem samples, and JohnV wouldn't accept a test with more than 12 samples.

Quote
2.  It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez.  You might link to his listening results.


Addressed. Thanks.

Quote
3.  The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test.


He didn't participate in this one already. :-P

Quote
4.  You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used.  The similar presentation format might make people think that all the analysis was identical.  The difference is mainly one about risk.  The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs.  On the other hand, it's more sensitive than the Tukey HSD.


Addressed

Quote
5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.


Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

Quote
6.  In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result.  Also, I'd like to see some mention of the results you threw out for rating the original less than 5.


OK, I will do that.

Of course, I won't make public the names of the participants that rated the reference. That wouldn't be nice to them, I believe.

In your test, you had people rating the reference too?


Quote
1.  Perhaps another call for samples -- classical and jazz samples -- would be profitable.


Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

Quote
2.  You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA.  This can be done with a small filesize penalty using Sox.  That would help to keep the ratings in perspective.


OK, I think I'll do that. Thanks for the idea.

Quote
3.  Verifying VBR average bitrates:  I think that this task could be split up among several people, each encoding whole albums with all codecs.


Yes, I plan to start a taskforce on this later this month.

Quote
Edit:  Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.


Done

Quote
Edit2:  The next test you'll probably want to be sure to check for level (volume) differences too.


OK, I will.

Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: NickSD on 2003-06-16 19:06:49
Quote
Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?

I don't know who submitted it, but the style would be something along the lines of jazz-rock.  Dave Matthews Band is very hard to categorize... (Incredible live band, by the way.)

Nick
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-16 19:09:21
Quote
Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

That argument simply doesn't work at all.

If you believe another set of samples would produce a significant performance difference between the AAC codecs compared to what we've seen here, it follows directly from that the next test would be highly unfairly biased towards AAC if you did use the same samples.
Title: AAC at 128kbps test - FINISHED
Post by: niktheblak on 2003-06-16 19:36:06
Quote
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ?

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I have to ask before I lose my mind, did the BeautySlept sample originate from me (in this thread (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=9&t=372&hl=)) or is it a funny coincidence
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 19:42:15
Quote
1) Any speculation as to why ATrain and Layla were so easy for AAC?  Surely they were chosen because the samples were challenging to MP3, Ogg and MPC.  (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)

No, they were chosen because they were featured in ff123's 64kbps test.

I will probably ditch them for the extension test though (thanks for the clarification, Garf)

Quote
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?


No idea.

Quote
3) One guesses that Apple is devoting considerable resources to further development.  Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?


There's another developer, that actually offered to send me a demo encoder that I can use in some future listening test, but I don't know if I can decline informations now.

What prevents dolby from doing what?

Quote
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has  and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support  for MPC.  "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public.  "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.


There's no standardized DRM system for AAC. And no, none of these encoders offer the option of DRM'ing your AACs, differently from WMA/Windows Media Encoder.

Regards;

Roberto.

@NickSD: Thanks for the info.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-16 19:50:19
Quote
Quote
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ?

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I was usually embarassed by the synthetic consistency of this harpsichord (and by the metronomic performance of the player). Thank you for this confirmation
Here is a real harpsichord sample. Bitrate is higher than with BeautySlept (265 kbps with --standard and 1.15r). It's a real killer too for most codec, if you want to reach 128 kbps and not more. I strongly suggest anyone to test it with WMA9 (PRO), incredibly transparent at 128 kbps compared to all other format/codecs I heard.

http://membres.lycos.fr/guruboolez/AUDIO/s...chpsichord.flac (http://membres.lycos.fr/guruboolez/AUDIO/samples/Bachpsichord.flac)
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-16 20:43:48
Quote
Quote
5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.


Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

I think the best way is to just use the same procedure that you did for each sample.  But you'll have to calculate the mean scores for ATrain and Layla first (easiest to just use Excel for that).

ff123
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-16 20:49:53
Regarding 41_30sec.wav

Quote
2 more pieces of info: who submitted it (if it's known), and what's the style?

Originally submitted by Filburt

style:  rock
Title: AAC at 128kbps test - FINISHED
Post by: ErikS on 2003-06-16 21:34:55
Does ANOVA take into consideration the varying scales that people use when they grade the test samples? For example someone who thinks that the goal of the codec is to achive total transparancy may grade everything between 1 and 3 when someone else who doesn't care that much would give grades between 4 and 5 to the same set.

So could someone very briefly explain how ANOVA works (or post some good links)? Which types of data sets is it possible to use ANOVA on?
Title: AAC at 128kbps test - FINISHED
Post by: Agent86 on 2003-06-16 21:58:01
Does anyone know anything about the AAC codec that XM is using to broadcast its satellite radio?

XM AAC Information (http://www.xmradio.com/corporate_info/fast_facts_sound.html)

It apppears as though it is heavily customized, but I wonder if it originally came from anything used in this listening test.

- Agent 86
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-16 22:13:23
XM is probably using gear from Thales
http://www.thales-bm.com/ (http://www.thales-bm.com/)
Probably this (http://www-db.thomcastcom.ext.imaginet.fr/db/thomcast/webdriver?MIval=V_Prod_Detail_b.html&F_ID=2&D_ID=1&P_ID=118) baby.
It's the same gear most DRM stations are using.

Thales uses AAC encoder from FhG and SBR encoder from CT. Probably  integerized versions.
Title: AAC at 128kbps test - FINISHED
Post by: Differenciam on 2003-06-16 22:48:19
Very interesting.  I'm quite disappointed in Nero, yet another reason I just keep re-using their trial. :roll:

I'm starting to wish my player supported AAC now. 
Title: AAC at 128kbps test - FINISHED
Post by: Agent86 on 2003-06-16 22:59:25
Quote
Thales uses AAC encoder from FhG and SBR encoder from CT.


That's pretty cool stuff.  One probably costs more then what I'll make in an entire year, but its damn cool stuff.

Too bad we can't encode with those codecs.  It would be interesting to see where "satellite radio" falls in the group.

- Agent 86
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-17 04:52:41
Quote
Does ANOVA take into consideration the varying scales that people use when they grade the test samples? For example someone who thinks that the goal of the codec is to achive total transparancy may grade everything between 1 and 3 when someone else who doesn't care that much would give grades between 4 and 5 to the same set.

So could someone very briefly explain how ANOVA works (or post some good links)? Which types of data sets is it possible to use ANOVA on?

This particular type of ANOVA does take into account the varying scales.  That's because I treat each listener as a "block."  Here is a general description of blocked ANOVA:

http://www.basic.nwu.edu/statguidefiles/on...ay_b_anova.html (http://www.basic.nwu.edu/statguidefiles/oneway_b_anova.html)

ff123
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-17 04:58:43
I updated the overall graph, following a suggestion by ff123 and Garf. Now, there's no more results being "eyeballed"

As you will notice, Faac got a little worse in this updated overall, and Nero is now almost perfectly tied to Psytel AACenc. Sorenson also got closer to QuickTime (or maybe QuickTime got ranked lower).

For those feeling nostalgic of the old plot:
http://rarewares.hydrogenaudio.org/test/aa...t/plot10old.png (http://rarewares.hydrogenaudio.org/test/aac128test/plot10old.png)

Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: den on 2003-06-17 05:39:39
Thanks again for organising this most useful test Roberto, and thanks to the others who helped with the setup, and post analysis.

Based on these results, which AAC encoder will be the "champion" to go up against the next round of contenders, namely vorbis, mpc and mp3?

It concerns me a little, that a particular AAC encoder will be chosen as the representative, but in reality, it may not be the most logical choice for the average user. For example, I don't think QT will be used as often as some of the others by Mr/Ms Joe Windows-User in its current state, due to the crappy "one file at a time" user interface. Even Sorenson may be a great encoder, but its cost may keep people away? 

I for one will always favour an encoder that allows me to access it from the command line, and feed it from a range of front ends of my choosing depending on what I need at the time, ie foobar, EAC, CDex, Frontah, home made batch files, "insert you favourite media player/front end/ripping/compression package here"... 

If all the command line optioned encoders sound like complete crap, I would be forced to use one of the superior encoders as the difference will be worth it, but I suspect from these results, Sorenson, Quicktime, Nero and Psytel all have some potential to compete against the others at ~ 128 kbit, especially if they are allowed to go VBR. If you choose QT to be the AAC representative, I will still be keen to be involved, and see the results, but if it is close to the other non-AAC formats in quality, I'll have this nagging question in the back of my head. "I wonder how Psytel would have gone..."

I appreciate that these listening tests can not accomodate everyone's wishes, and I suppose ultimately you need to pick the best performing encoder, regardless of its expense or convenience...

Just my 2 Australian cents.

Den.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-17 07:14:37
Quote
Based on these results, which AAC encoder will be the "champion" to go up against the next round of contenders, namely vorbis, mpc and mp3?

Add WMA to that list.

And atrac3 is another possibility, but that's only a speculation right now. I don't even plan to take this decision now, I'd rather go take a looong nap after I finish dealing with the test results. Preferably at the beach.

After I'm sufficiently rested, I'll start thinking about codecs, bitrates, settings, etc.

Anyway, ATM there are two options for the AAC: Nero and QuickTime

QuickTime because it kicks ass. Nero because Ivan reportedly has been improving his codec, spurred on by this test results. If he can deliver an improved codec on time, it might be considered for the extension test.

Quote
It concerns me a little, that a particular AAC encoder will be chosen as the representative, but in reality, it may not be the most logical choice for the average user. For example, I don't think QT will be used as often as some of the others by Mr/Ms Joe Windows-User in its current state, due to the crappy "one file at a time" user interface. Even Sorenson may be a great encoder, but its cost may keep people away? 


Right, I don't think people will be letting go 400 hard earned smackers just so that they can encode their audio tunes to AAC.

Besides, I think Sorenson suffers from the same problem as QuickTime, of "one file at a time". Gotta check that later.

Quote
I for one will always favour an encoder that allows me to access it from the command line, and feed it from a range of front ends of my choosing depending on what I need at the time, ie foobar, EAC, CDex, Frontah, home made batch files, "insert you favourite media player/front end/ripping/compression package here"...


I understand you are talking about AACenc, but including AACenc wouldn't be a good idea. From the development point of view, AACenc is now dead, since it can't be developed furthermore. Nero has the downside of being GUI-only, but at least it's being actively developed, more so these days.

Quote
I appreciate that these listening tests can not accomodate everyone's wishes, and I suppose ultimately you need to pick the best performing encoder, regardless of its expense or convenience...


Well, by far the most important aspect is the quality, but price and convenience also have their importance. For instance, I would hardly pick Sorenson for the extension test even if it had won, because it's an encoder few people have access to, it's damn expensive and you can only encode one file at a time.

QuickTime, on the other hand, is cheaper, widely available and reportedly you don't even need to register it if you access the encoding component directly through another program. (maybe a CLI program  )

Thanks a lot for your remarks.

Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: lazyn00b on 2003-06-17 09:09:22
Quote
For instance, I would hardly pick Sorenson for the extension test even if it had won, because it's an encoder few people have access to, it's damn expensive and you can only encode one file at a time.

If you use Squeeze 3.5, you can "Open Watch Folder" to do a batch encode. It's very odd name for batch encoding until you realize that Squeeze will remain open and "watch" for any new WAV files you drop into the Watch Folder and immediately start encoding them according to your previous settings. It's weird, but kinda cool in the right situation. Squeeze will also separate the WAV files and MP4 files into separate sub-folders called "Work" and "Done", which may or may not drive you insane.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-17 10:10:03
We can consider the possibility, for the next test, to include two AAC codecs :

- VBR one, in fast development (don't forget HE-AAC), easy to use, pretty GUI (I can transcode my MPC to MP4 with Nero)
- Winner of this test : CBR QT 128, for people that want a very good quality, without consideration of time encoding, GUI annoyance, etc...


Other codecs may be mpc, vorbis (official), wma (standard, pro ?), mp3 (lame). Maybe ATRAC3...
But that's maybe too much challengers (12 samples x 6-7 codecs = 72...84 files).
(don't forget that you need a CLI decoder, if you want to reduce the package size with a .bat file : is there any WMA9 or ATRAC3 separate decoder available ?)


Problem is : what's the purpose of this test ?
Is it a practical one : what is the best audio format for portable use ? MPC is actually useless here...
Is it a theorical one : what is the best audio format at 128 kbps, regardless of any portable available, or GUI, or encoding speed ?
Is it a mix of the two ?

We need to clearly answer to these questions in order to make the codec/setting choices.
Title: AAC at 128kbps test - FINISHED
Post by: JohnV on 2003-06-17 11:26:21
Quote
We can consider the possibility, for the next test, to include two AAC codecs :

- VBR one, in fast development (don't forget HE-AAC), easy to use, pretty GUI (I can transcode my MPC to MP4 with Nero)
- Winner of this test : CBR QT 128, for people that want a very good quality, without consideration of time encoding, GUI annoyance, etc...


Other codecs may be mpc, vorbis (official), wma (standard, pro ?), mp3 (lame). Maybe ATRAC3...
But that's maybe too much challengers (12 samples x 6-7 codecs = 72...84 files).
(don't forget that you need a CLI decoder, if you want to reduce the package size with a .bat file : is there any WMA9 or ATRAC3 separate decoder available ?)


Problem is : what's the purpose of this test ?
Is it a practical one : what is the best audio format for portable use ? MPC is actually useless here...
Is it a theorical one : what is the best audio format at 128 kbps, regardless of any portable available, or GUI, or encoding speed ?
Is it a mix of the two ?

We need to clearly answer to these questions in order to make the codec/setting choices.

I was talking with Roberto earlier today exactly about this. We have a plan how to proceed, but I don't know when Roberto wants to make it public.

I'm also wondering if MPC is necessary for low-medium (around 128kbps) vbr test. But Roberto claims that Musepack users will be very upset if it's not included. I'm not so sure about this, because people already know that Musepack is very good at higher bitrates, and I'm not sure if MPC users are even interested in the bitrates this low.
Title: AAC at 128kbps test - FINISHED
Post by: dev0 on 2003-06-17 11:40:33
Quote
I'm also wondering if MPC is necessary for low-medium (around 128kbps) vbr test. But Roberto claims that Musepack users will be very upset if it's not included. I'm not so sure about this, because people already know that Musepack is very good at higher bitrates, and I'm not sure if MPC users are even interested in the bitrates this low.


Most current Musepack users (audiophiles/HA-whores ) are definetly not interested in anything below --standard, but it has been mentioned before that it will probably do a lot better than people would expect it to at around 128kbps, so this might bring some clarity to the situation and possibly make Musepack an option for a new group of users.

dev0
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-17 12:01:45
Excluding MPC makes no sense - it's definetely a contender at 128kbps.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-17 12:20:28
Quote
Excluding MPC makes no sense - it's definetely a contender at 128kbps.

I'm partially agree with you. I did some tests long time ago, and I found mpc --radio to be the most pleasant format to my ears : PsyTEL -streaming was second, followed by LAME ABR. I didn't like Vorbis (rank #5 if I remember correctly).
Nevertheless, if the new test has for purpose PORTABLE abilities, musepack is a contestable choice... It may be a better choice to include ATRAC3 (MiniDisc portable) or WMA9, even if theses competitor are worse than mpc.

There are some problems with mpc too. Accoding to my experience, the amplitude of musepack is the higher I saw on 'common' music. A piano Goldberg Variations at --radio is ~100 kbps (Gould 81 for exemple) ; on harpsichord, it's ~170-180 kbps. Some isolated tracks are ~90 kbps with piano, and ~200 with harpsichord. Difference isn't so big with other codecs : it's easier to reach an approx. 130 kbps, or fit them to an exact 128 kbps without losing too much in quality.
Here are the bitrate for each samples of the previous test :
http://membres.lycos.fr/guruboolez/AUDIO/a...pack_report.htm (http://membres.lycos.fr/guruboolez/AUDIO/aac128/musepack_report.htm)
(--radio, but don't remember if it's 1.14 or 1.15r)
Title: AAC at 128kbps test - FINISHED
Post by: JohnV on 2003-06-17 12:42:23
Quote
Excluding MPC makes no sense - it's definetely a contender at 128kbps.

Do you know anybody who uses MPC at these bitrates? Do you think thousands of new people will start using MPC even if MPC wins that test? Imo MPC is clearly a niche format, not even targeted at masses. The (sad) fact is MPC does not have mass appeal, starting from its website...

I think many people here already know that MPC is surprisingly good at average 128kbps. Not sure what would it serve to include it, simply because even though MPC is very good, it is generally unknown format used by small niche group and lacks the appeal ouside hifi quality high bitrate use.

Sure, maybe MPC gets few dozen of new users who will start using it at higher bitrates, but I'm not sure if it makes any difference regarding MPC's position as a format.

I don't have anything against the inclusion of MPC, on the contrary, but if we have to chop some codecs, imo we should start from those codecs which have overall the least interestingness globally.
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-17 13:10:14
I don't get it. Why should we include Vorbis then? What's the point of the test?
Title: AAC at 128kbps test - FINISHED
Post by: JohnV on 2003-06-17 13:23:57
Quote
I don't get it. Why should we include Vorbis then? What's the point of the test?

Xiph tries to make Vorbis appeal to masses. It's relatively good at all bitrates, it has multichannel encoding, it has directshow filters, it is used in movie encoding, it has an order of magnitude bigger userbase than MPC, Vorbis has had quite a lot of media coverage (MPC once introduced in c't), there are several broadcasters streaming vorbis etc. etc.
In short, Vorbis is trying to be in the same league with the big formats, when MPC is not even trying and it isn't in the same league, except quality wise. I'm not sure if the knowledge that MPC is good at 128kbps is very relevant for the masses outside HA, and most HA people interested in MPC know that MPC is good at near 128kbps already, so why include MPC?
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-17 13:27:05
Quote
and most HA people interested in MPC know that MPC is good at near 128kbps already, so why include MPC?

Well, I for one would like to know how good or bad it is.

But you did not really answer my question, what is the use of the second test? Is it to compare AAC to it's direct competitors, and see who has the best quality? In that case, MPC would not be a competitor indeed.
Title: AAC at 128kbps test - FINISHED
Post by: JohnV on 2003-06-17 13:56:21
Quote
But you did not really answer my question, what is the use of the second test? Is it to compare AAC to it's direct competitors, and see who has the best quality? In that case, MPC would not be a competitor indeed.

Umm, I don't think it's the idea that the cross-format test is AAC vs others. Rather I'd see it as comparison of the big players against each others - the formats which are the most widely used in PC, so that the test has as much significance as possible for as many people as possible.

I'm not deciding anything here. Besides, I sure will be talking a lot about this with Roberto. I just think we should consider carefully which formats will be included.
Title: AAC at 128kbps test - FINISHED
Post by: ErikS on 2003-06-17 14:56:52
Just to chime in with Garf... MPC is used by exactly as many portable devices as Vorbis. There is only those who say they are working on it and that it will be released later - that goes for Vorbis as well as MPC. So from that is definately not an argument for excluding one from the test and keeping the other.

If the test is setup to answer which is the best choice for portables both WMA8 and ATRAC3 have to be included. Neither Vorbis nor MPC has anything to do in such test.
Title: AAC at 128kbps test - FINISHED
Post by: JohnV on 2003-06-17 15:03:15
Quote
Just to chime in with Garf... MPC is used by exactly as many portable devices as Vorbis. There is only those who say they are working on it and that it will be released later - that goes for Vorbis as well as MPC. So from that is definately not an argument for excluding one from the test and keeping the other.

If the test is setup to answer which is the best choice for portables both WMA8 and ATRAC3 have to be included. Neither Vorbis nor MPC has anything to do in such test.

As far as I can see I didn't say anything about portables at all.
Portable usage is one aspect which should be considered also though.
Title: AAC at 128kbps test - FINISHED
Post by: ErikS on 2003-06-17 15:10:15
Quote
As far as I can see I didn't say anything about portables at all.
Portable usage is one point of view also.

guruboolez did.

In case it will be narrowed down to portable use you could afford to have two AAC codecs in the test as well.
- QT
- Nero VBR
- LAME ABR
- WMA
- ATRAC3

More than five codecs is too much IMO.
Title: AAC at 128kbps test - FINISHED
Post by: den on 2003-06-17 15:58:42
The objectives of this test are not that clear, at least to me anyway. 

Despite being an ATRAC3 user (dodges rotten tomatoes)    , I'm not sure if it belongs here. I hate to generalise, but most if not all ATRAC3 users do so because they have no other choice, ie they own an ATRAC3 only Sony/Minidisc product. OK, so Sony now have these Discmans with ATRAC and MP3, but I'd bet my left one    that savvy owners would use MP3/LAME alt presets. You simply have more control over your encode...

I would expect in the longer term that ATRAC will be left to die when Minidisc eventually goes away...

I suggest we drop ATRAC3 from the list, unless you really need another codec as a control, or comparison for the test.

I think WMA, Vorbis, Musepack and the omnipresent MP3 should stay in. In fact LAME and FHG should probably be tested as some here swear (substantiated or otherwise) about FHG being better at 128 kbits than LAME. They all offer creditable, albeit sometimes unused performance at this bitrate, even if they lack of portability. While Vorbis and MPC have similar, virtually non-existant portable support, it will probably come in the future, even if only on PocketPCs, etc (Vorbis is there now.)

With Musepack in particular, I came away pleasantly surprised with mpc -radio during recent tests. It isn't regularly used or recommended by HA, but that could be from habit rather than the facts. (In my tests, I actually preferred it over Vorbis...)

Just another 2 Australian cents wirth (which currently buys a lot more than it did 12 months ago...)  B)


Den.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-17 17:20:32
Wow. I wake up, and there's almost a format war going on.

Well, people, first, I'd like to repeat that I'm not willing to make decisions ATM. We have a full month ahead until the next test starts, it's now time to chill out.  B)

Some misconceptions that happened here:

1) Where the heck did you guys came from that this test is intened to address formats with portable support? I never wrote it, either at posts or at the test pages. The main purpose of these tests is verify what format performs best at quality. And only that. Other issues like usability, popularity and availability might be taken in consideration, but quality weights in about 90% in my decisions. Only when two formats/codecs are tied in quality that I will consider other features.

2) I definitely don't plan to allow a format to be represented twice in the extension test (I.E, QT and Nero). I remember how ff123 suffered from his decision of representing vorbis twice on his 64 test, and I don't plan to go the same way.

3) atrac3 is only speculation ATM. There'll be lots of things to consider.

4) It's likelier that MPC will be at the test instead of atrac3

5) No Lame vs. FhG MP3 at this time. I decided that this can be handled in a separate test.

Now, the idea JohnV and I had was to conduce a fast listening test (5 days, 8 samples, 2/3 codecs) at the beginning of July, QuickTime 128 vs. Nero -streaming (and maybe a 3rd codec kept in secrecy ATM). If Nero wins, it'll be used on the extension test. If QT wins, it'll be used and God help us all.

Again, it's only an idea. It's not definitive

And this is my planned schedule:
June 28th to July 3rd: The quick AAC test
July 16th to July 27th: The extension test

And it's only a plan. Don't go planning your schedule based on that, and don't start complaining about the dates now. You'll have time to do that later.

As a bottom line, the formats that are already guaranteed at this test are Vorbis (official CVS version), AAC (QT or Nero), MP3 and VQF. I mean, WMA v9.

Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: pierugg on 2003-06-17 20:05:53
What about MP3Pro ??
Title: AAC at 128kbps test - FINISHED
Post by: floyd on 2003-06-17 20:30:34
There isn't much point testing mp3pro at 128kpbs...
Title: AAC at 128kbps test - FINISHED
Post by: SometimesWarrior on 2003-06-17 20:56:19
I'd like to better understand the purpose of this test, too. I know Roberto said "the purpose is to find what encoder performs best encoding AAC at 128kbps", but that still leaves me with some questions.

CBR was chosen because some codecs are incapable of making VBR audio, and a level playing field was desired, correct? Well, this isn't a friendly, sporting match between AAC implementations, this is a codec war! Let every codec employ its best tools for the job, I say, and that would mean using VBR wherever possible. I know there was a discussion back with the 64kbps test about setting the Vorbis q-value so that the samples' bitrates averaged to 64kbps, and I think ff123's conclusion was that q0 should be used simply because it's the most likely setting for a typical user to choose. But that decision was not a quality-based one, and since this test is all about quality, the question should be re-visited. Or maybe you've already dealt with the issue by planning a quick AAC test at the end of the month. In that case... never mind.

Another question: I don't recall reading why 128kbps was chosen. Is it just a convenient number chosen for historical reasons? Are the codecs optimized for this specific bitrate? Different bitrates can be used for streaming, file distribution, digital radio, portable playback, and archival. Is 128kbps considered to be the best compromise for all these applications? Or are people just more likely to pay attention to the results of a 128kbps test?

Roberto, I'm not trying to discount your listening test. I still think it was (and will be) very well-designed. It has already given us valuable information, and I know it will continue to do so.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-17 21:33:25
MP3pro isn't considered good above 96kbps. The SBR codec starts degrading the audio quality more than mp3 alone would do.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-18 03:53:56
Here's a comment from the masses:

http://groups.google.com/groups?dq=&hl=en&...ob.news.rcn.net (http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=bcnnt2%24p1d%241%40bob.news.rcn.net)

It warms my heart.

ff123
Title: AAC at 128kbps test - FINISHED
Post by: chf on 2003-06-18 04:29:52
 
    Sorry, Is there any one try to compare ODG of these AAC Codec ?

    I want to do this. but I dont know how to let QT output as wav file ?

    By the way, where can I download the samples ?

    I just see samples**.zip.torrent  but I dont have bitTorrent..


      Thanks a lot
Title: AAC at 128kbps test - FINISHED
Post by: tangent on 2003-06-18 05:47:33
It would be nice to see MPC @128kbps added to the test, so that we can validate those comments saying that "subband can rule at 128kbps too!". 2 main problems I see:

1. Have we already hit our rule-of-thumb limit of maximum 6 test examples?
2. How do you control the MPC bitrate? In previous listening tests @128kbps, the performance of MPC depended on the resulting bitrate. MPC clearly won the first dogies test , but that can be attributed to bitrate bloat to 143kbps. For rawhiles and fossiles, MPC was at 136 and 138 and there were no significant results. For wayitis, MPC was tied at the top with Ogg Vorbis, but needed 140kbps to do that.
Title: AAC at 128kbps test - FINISHED
Post by: ajp on 2003-06-18 08:31:36
The result of the test has been posted here as well...

http://www.cdfreaks.com/news/7355 (http://www.cdfreaks.com/news/7355)

I like some of the comments below....
Title: AAC at 128kbps test - FINISHED
Post by: SometimesWarrior on 2003-06-18 09:01:15
Quote
Sorry, Is there any one try to compare ODG of these AAC Codec ?

I want to do this. but I dont know how to let QT output as wav file ?

By the way, where can I download the samples ?

I just see samples**.zip.torrent  but I dont have bitTorrent..

You can use FAAD to decode the QT files, I believe. I haven't used QT, but I think that's how Roberto decompressed the test samples. I think FAAD can be downloaded at RareWares.

The test files are no longer available for download, since the test is over. You can still get the names of the samples, and then search for available copies. HA and ff123 still host some, if not all, of the samples used in the test.

As for ODG... isn't that the EAQUAL measurement standard? An EAQUAL comparison would be academic, not practical, since previous discussions have shown that its quality rankings are not very precise, nor are they appropriate for codec comparison.
Title: AAC at 128kbps test - FINISHED
Post by: SometimesWarrior on 2003-06-18 09:12:18
Quote
The result of the test has been posted here as well...

http://www.cdfreaks.com/news/7355 (http://www.cdfreaks.com/news/7355)

I like some of the comments below....

"I hope they are talking about Astrid/Quartex AAC, others AAC sux..."

Ladies and gentlemen: Hydrogen Audio, the ivory tower of psychoacoustic compression.
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-18 10:24:25
Quote
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?

Radio@AOL is using the now famous Dolby AAC encoder. More information can be found in the following link:

http://www.dolbylabs.com/DolbyAAC/ (http://www.dolbylabs.com/DolbyAAC/)

Given Dolby's comment:

Quote
Dolby® AAC is an enhanced version of MPEG-2 and MPEG-4 AAC that adds proprietary Dolby intellectual property and supports data rates from 14 kb/s (mono) to 128 kb/s (stereo) and up.

How does Dolby AAC differ from AAC?
Dolby AAC consists of a standard MPEG-AAC core plus proprietary enhancements that enable a wider frequency range and better audio quality at lower bit rates.


it seems that they don't simply use the FhG codec modified for speed but they must have done some work in order to improve the quality. This makes sense after the listening test results.
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-18 10:30:04
Quote
The result of the test has been posted here as well...

http://www.cdfreaks.com/news/7355 (http://www.cdfreaks.com/news/7355)

I like some of the comments below....

This article is not entirely correct because the Sorenson encoder was voted second in quality and not the one in Nero.
Title: AAC at 128kbps test - FINISHED
Post by: loophole on 2003-06-18 13:50:23
Nice board

I would just like to point out to anybody interested, is that iTunes 4 does use the same encoder as QuickTime 6.3, it's just that in the 4.0 release it was accidently hardcoded to the fastest setting. 4.01 has corrected this and it is now hard coded to highest quaiity (as seen by the extreme decrease in encoding speed on my powerbook (13.9x down to 7.2x)).

I am 99.99999% sure on this, but maybe someone with better ears than me could verify it.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-18 15:03:08
Quote
You can use FAAD to decode the QT files, I believe. I haven't used QT, but I think that's how Roberto decompressed the test samples. I think FAAD can be downloaded at RareWares.

I used FAAD.

QT has some issues and can't decode AAC files wrapped with mp4creator (Psytel and FAAC, in the test)

Quote
"I hope they are talking about Astrid/Quartex AAC, others AAC sux..."

Ladies and gentlemen: Hydrogen Audio, the ivory tower of psychoacoustic compression.


I know I should reply, talking about Pac and about stolen libraries. But I really don't feel like it. >_<

I mean, what the heck, since the days of the VQF forum people know Astrid was stolen from CelestialTech's AudioLib 1.0
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-18 15:04:59
In order to clarify the challengers for the second (part of the) test, I did a quick blind test, and opposed WMA9 called 'PRO' to WMA9 standard. Results are here (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=32&t=10551).

WMA9 PRO seems to be the best encoder, and should be prefered to the standard (and portable-compatible) one.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-18 15:07:38
Quote
I would just like to point out to anybody interested, is that iTunes 4 does use the same encoder as QuickTime 6.3, it's just that in the 4.0 release it was accidently hardcoded to the fastest setting. 4.01 has corrected this and it is now hard coded to highest quaiity (as seen by the extreme decrease in encoding speed on my powerbook (13.9x down to 7.2x)).

I am 99.99999% sure on this, but maybe someone with better ears than me could verify it.

I head about that at the 3ivX forum too. But they aren't sure either.

I'm waiting for some confirmation before I update the results page.

One way to confirm would be to encode a file with iTunes, then with QT in high mode, and compare them.

Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-18 15:27:37
Quote
Quote
I would just like to point out to anybody interested, is that iTunes 4 does use the same encoder as QuickTime 6.3, it's just that in the 4.0 release it was accidently hardcoded to the fastest setting. 4.01 has corrected this and it is now hard coded to highest quaiity (as seen by the extreme decrease in encoding speed on my powerbook (13.9x down to 7.2x)).

I am 99.99999% sure on this, but maybe someone with better ears than me could verify it.

I head about that at the 3ivX forum too. But they aren't sure either.

I'm waiting for some confirmation before I update the results page.

One way to confirm would be to encode a file with iTunes, then with QT in high mode, and compare them.

Regards;

Roberto.

I will encode a file using QT6.3 (Highest) and iTunes 4.01 and compare the encoded files using a hex editor. I will report the results back to this thread.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-18 16:24:57
Tested (quickly) GT2 vs Post CVS release of ogg vorbis, here (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=9&t=10555).
I changed my mind about GT2 virtue : final version is maybe more 'dirty' (hiss, noise added), there are less annoying artifacts (and size is a bit smaller), at least, on difficult samples.
Title: AAC at 128kbps test - FINISHED
Post by: loophole on 2003-06-18 16:38:38
Quote
Quote
Quote
I would just like to point out to anybody interested, is that iTunes 4 does use the same encoder as QuickTime 6.3, it's just that in the 4.0 release it was accidently hardcoded to the fastest setting. 4.01 has corrected this and it is now hard coded to highest quaiity (as seen by the extreme decrease in encoding speed on my powerbook (13.9x down to 7.2x)).

I am 99.99999% sure on this, but maybe someone with better ears than me could verify it.

I head about that at the 3ivX forum too. But they aren't sure either.

I'm waiting for some confirmation before I update the results page.

One way to confirm would be to encode a file with iTunes, then with QT in high mode, and compare them.

Regards;

Roberto.

I will encode a file using QT6.3 (Highest) and iTunes 4.01 and compare the encoded files using a hex editor. I will report the results back to this thread.

That won't work because iTunes adds extra metadata. The files will be slightly different sizes. Someone will have to do a listening test I think.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-18 16:53:18
Quote
That won't work because iTunes adds extra metadata. The files will be slightly different sizes. Someone will have to do a listening test I think.

Hrm... maybe extracting the AAC tracks with MP4creator for Mac (http://rarewares.hydrogenaudio.org/files/MPEG4IP.Server.Utils-0.9.8.for.OSX.tar.bz2), and then comparing the AACs?
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-18 16:59:51
Quote
Here's a comment from the masses:

http://groups.google.com/groups?dq=&hl=en&...ob.news.rcn.net (http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=bcnnt2%24p1d%241%40bob.news.rcn.net)

It warms my heart.

ff123

Your reply appeared on google groups:
http://groups.google.com/groups?dq=&hl=en&...52dfa%404ax.com (http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=qgivev8juueb6r2ode852eehvg1s152dfa%404ax.com)


(http://pessoal.onda.com.br/rjamorim/HPL_roflmaom.jpeg)
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-18 17:12:34
Quote
Quote
That won't work because iTunes adds extra metadata. The files will be slightly different sizes. Someone will have to do a listening test I think.

Hrm... maybe extracting the AAC tracks with MP4creator for Mac (http://rarewares.hydrogenaudio.org/files/MPEG4IP.Server.Utils-0.9.8.for.OSX.tar.bz2), and then comparing the AACs?

Ok, I'll do that then and report back.
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-18 18:49:07
Ok, I encoded an AIFF file with QuickTime 6.3 and then with iTunes 4.01 at 160Kbps. I then used mp4creator to extract the audio track and... the iTunes encoded file does not match any QuickTime 6.3 encoded file at any quality setting 

I also checked the tracks in a hex editor and they are clearly different. The sizes are also different i.e.

QT6.3 High  - 1994056 bytes
QT6.3 Better - 1994053 bytes
QT6.3 Good  - 1994087 bytes
iTunes 4.01  - 1995001 bytes

I then tried AAChoo (front-end for QT) and the encoded tracks were the same to those produced by QT for each quality setting, although the .mp4 files were slightly different in size because of the addition of extra tags.

I am confused... Is iTunes passing some special parameters to QT?
Title: AAC at 128kbps test - FINISHED
Post by: brett on 2003-06-18 20:46:54
i did a quick encode of one 6+ minute track in itunes 4.0.1 @ 160 and qt 6.3 @ 160 "best" -- itunes took 24 seconds, qt took 44 seconds. dunno if itnues is more altivec or mp savvy, tho. so . . . fwiw . . . might not be worth much.
Title: AAC at 128kbps test - FINISHED
Post by: bawjaws on 2003-06-18 22:28:21
Quote
Ok, I encoded an AIFF file with QuickTime 6.3 and then with iTunes 4.01 at 160Kbps. I then used mp4creator to extract the audio track and... the iTunes encoded file does not match any QuickTime 6.3 encoded file at any quality setting 

What options did you select with QT?

iTunes only lets you set bitrate but QT has a load of other options. Surely the setting of those would have an effect? In particular there are options for optimizing the file for streaming.

Now I would, on first thought, switch them all off but one of the major features of iTunes 4 is LAN streaming (with internet streaming to specific clients due to return in a future update according to rumour) so it makes sense for iTunes to use these settings if it doesn't massively affect encoding time.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-18 23:42:43
Quote
In particular there are options for optimizing the file for streaming.

Optimize for streaming only affects the MP4 container. If you strip the AAC file from inside, there should be no difference.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-18 23:52:36
It might make it easier to compare if you decode to WAV.

ff123
Title: AAC at 128kbps test - FINISHED
Post by: Paspro on 2003-06-18 23:55:36
Quote
Quote
Ok, I encoded an AIFF file with QuickTime 6.3 and then with iTunes 4.01 at 160Kbps. I then used mp4creator to extract the audio track and... the iTunes encoded file does not match any QuickTime 6.3 encoded file at any quality setting 

What options did you select with QT?

iTunes only lets you set bitrate but QT has a load of other options. Surely the setting of those would have an effect? In particular there are options for optimizing the file for streaming.

Now I would, on first thought, switch them all off but one of the major features of iTunes 4 is LAN streaming (with internet streaming to specific clients due to return in a future update according to rumour) so it makes sense for iTunes to use these settings if it doesn't massively affect encoding time.

I encoded the file using QT and each quality setting without any other special parameters. The option to hint for streaming was disabled but that does not affect the encoding. I then extracted the tracks using mp4creator and I could not match any of them with the extracted track from the iTunes 4.01 encoded file. I managed to match them with encodings using AAChoo which is a front-end for QT (like iTunes for AAC). The iTunes 4.01 speed of encoding was faster than the QT one in high quality.

My guess is that iTunes calls the QT encoder with some special parameters to optimise the speed resulting in a file which cannot be generated by simply using the QT interface.
Title: AAC at 128kbps test - FINISHED
Post by: loophole on 2003-06-19 01:57:48
Quote
Quote
Quote
Ok, I encoded an AIFF file with QuickTime 6.3 and then with iTunes 4.01 at 160Kbps. I then used mp4creator to extract the audio track and... the iTunes encoded file does not match any QuickTime 6.3 encoded file at any quality setting 

What options did you select with QT?

iTunes only lets you set bitrate but QT has a load of other options. Surely the setting of those would have an effect? In particular there are options for optimizing the file for streaming.

Now I would, on first thought, switch them all off but one of the major features of iTunes 4 is LAN streaming (with internet streaming to specific clients due to return in a future update according to rumour) so it makes sense for iTunes to use these settings if it doesn't massively affect encoding time.

I encoded the file using QT and each quality setting without any other special parameters. The option to hint for streaming was disabled but that does not affect the encoding. I then extracted the tracks using mp4creator and I could not match any of them with the extracted track from the iTunes 4.01 encoded file. I managed to match them with encodings using AAChoo which is a front-end for QT (like iTunes for AAC). The iTunes 4.01 speed of encoding was faster than the QT one in high quality.

My guess is that iTunes calls the QT encoder with some special parameters to optimise the speed resulting in a file which cannot be generated by simply using the QT interface.

Hmm, I wonder what the quality is like tho. Maybe in the next test have an iTunes encoded AAC file and a QuickTime encoded one. Who knows. I just wish iTunes had a quality option so I could be sure.
Title: AAC at 128kbps test - FINISHED
Post by: jweeks on 2003-06-19 06:36:53
From reading this thread:

http://discussions.info.apple.com/WebX?128...Hk.18@.3bc2a4c8 (http://discussions.info.apple.com/WebX?128@51.dggFaSEimHk.18@.3bc2a4c8)

...it appears that iTunes 4.0.1 uses the "BETTER" QT quality (not "BEST") -- it sacrifices quality for higher encoding speed.

-Jonathan
Title: AAC at 128kbps test - FINISHED
Post by: brett on 2003-06-19 08:15:53
i for one have already sent apple feedback asking for the qt "quality" settings to be brought into itunes so the user can choose to sacrifice speed for quality -- like i want to -- and i linked this thread to them to show how their own slight quality "win" (quicktime) can be easily discounted because their consumer encoder (itunes) does not carry the same quality. others concerned should provide feedback to apple, too. sometimes, they listen.
Title: AAC at 128kbps test - FINISHED
Post by: 6545608403 on 2003-06-19 19:20:36
Quote
(jweeks @ Jun 18 2003 - 09:36 PM)

From reading this thread:

http://discussions.info.apple.com/WebX?128...Hk.18@.3bc2a4c8 (http://discussions.info.apple.com/WebX?128...Hk.18@.3bc2a4c8)

...it appears that iTunes 4.0.1 uses the "BETTER" QT quality (not "BEST") -- it sacrifices quality for higher encoding speed.


A fairly knowledgable (they are hit and miss) Applecare representative told me (after confirming with an Apple programmer) that iTunes uses "Good" quality for its encoding, and iTunes encoding did not change between 4.0 and 4.0.1.  The main quality/encoding difference since iTunes 4.0's release is the upgrade of the QT encoder used (from 6.2 to 6.3).

I cannot verify this information quantitatively since 4.0 and 4.0.1 encoded files are different post the installation of QT 6.3.  (I have both on my system.)
Title: AAC at 128kbps test - FINISHED
Post by: loophole on 2003-06-20 12:41:19
Strange, my encoding speed dropped dramatically with the 4.01 upgrade. I wonder if Good is like q3 for lame, better is like q2 and best is like q0.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-21 02:31:27
Quote
Strange, my encoding speed dropped dramatically with the 4.01 upgrade. I wonder if Good is like q3 for lame, better is like q2 and best is like q0.

I've been having a very fruitful e-mail swap with Apple's head AAC developer,  Mr. Stanley Kuo. Here's his clarification on this specific subject:

Quote
iTunes does use the SAME encoder as QT, and iTunes does give the user the ability to change the encoder's mode. iTunes uses the "Better" quality mode (not the fastest) which is optimized to perform best with 16bit source material (ie. CD source). "Best" quality mode is targetted at 24bit source material (eg. DVD source) and there should be NO discernible difference in quality for CD source between these two modes of the encoder. Generally it's just a waste of CPU to use best quality for 16 bit source, but there's no harm done by doing this of course.


Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: SometimesWarrior on 2003-06-21 02:40:23
Roberto, do you still plan on adding the individual user comments and sample bitrates?
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-21 03:23:55
Quote
Roberto, do you still plan on adding the individual user comments and sample bitrates?

I do. Will probably do that this weekend.

I'm really sorry about the delay. This week has been terrible. Finals, listening test (and dealing with lots of people claiming my test is useless), parents came visit me (fortunately they were in route to another town),...
Title: AAC at 128kbps test - FINISHED
Post by: ddrawley on 2003-06-21 04:48:15
Some say a monument will never be erected for a critic.

I now erect ( uhhh, pile up ) one. It looks, and smells like cow feces.

I for one appreciate your hard work. I have been wondering for some time how the AAC codecs stand against each other.

Thank you
Title: AAC at 128kbps test - FINISHED
Post by: 6545608403 on 2003-06-21 07:25:00
Quote
I've been having a very fruitful e-mail swap with Apple's head AAC developer,  Mr. Stanley Kuo. Here's his clarification on this specific subject:


QUOTE

iTunes does use the SAME encoder as QT, and iTunes does give the user the ability to change the encoder's mode. iTunes uses the "Better" quality mode (not the fastest) which is optimized to perform best with 16bit source material (ie. CD source). "Best" quality mode is targetted at 24bit source material (eg. DVD source) and there should be NO discernible difference in quality for CD source between these two modes of the encoder. Generally it's just a waste of CPU to use best quality for 16 bit source, but there's no harm done by doing this of course.


Roberto, thanks for clearing that up; it's been a subject of so many posts on many boards. 

(So much for the "knowledgable Applecare".    )  But, where is iTunes' option to "give the user the ability to change the encoder's mode"?  I can't find it...
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-22 08:44:24
Quote
But, where is iTunes' option to "give the user the ability to change the encoder's mode"?  I can't find it...

I am not a Mac user, so I am unable to discuss that subject further with Mr. Kuo, sorry.


The test comments have been uploaded.
http://rarewares.hydrogenaudio.org/test/aa...t/comments.html (http://rarewares.hydrogenaudio.org/test/aac128test/comments.html)

Sorry for the sucky way I made them available. I'll try to make it better soon, it's 4:45 AM here...

Regards;

Roberto.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-22 08:54:21
2 points about the results:

1 - If you submitted the results and don't see them there, it's because you didn't mention in the e-mail that you allowed me to publish them. If you actually forgot to mention and what them to be published (either with your real name/nickname or anonimously), please mail me at the same address.

2 - I would like to request people to avoid making fun of any "dumb" comments they might find. Remember there was a time you weren't experts.
Title: AAC at 128kbps test - FINISHED
Post by: ErikS on 2003-06-22 09:45:12
Funny that everybody hears different things. If we add together all flaws that people heard there won't be much left of the original music.

One of the codecs got these comments but was still ranked second for this sample... the others must sound horrible then. 

"Bad pre-echo, clicks, cut-off, warble, 'discrete' freq sweep, dirtier, noisy, tones merging, distortion or clipping, noisy attacks, rumbling, muffled, post echo tones, smeared attacks, faser sound, gurgling, noisy hi-hats"
Title: AAC at 128kbps test - FINISHED
Post by: [proxima] on 2003-06-22 15:34:59
Quote
Funny that everybody hears different things.

I think this is normal, these results might not respect the personal preferences (and i'm not talking about zealots  ).
Quote
One of the codecs got these comments but was still ranked second for this sample... the others must sound horrible then.

According to my personal preferences, for example, Sorenson scored the same as FAAC 
I'm quite surprised, for this. I remember a previous thread where Ivan Dimkovic stated this codec (AACdemo) "as the state of art" around 128 kbps. But i'm not here to start a flame, i respect the other's impressions and the tests conclusions.
Nevertheless, If you're speaking about Flooressence, i think that this sample should be considered a real problematic sample for Sorenson: very annoying, expecially when an obvious "scratch" occur at 2.2 - 3.3.
Quote
"Bad pre-echo, clicks, cut-off, warble, 'discrete' freq sweep, dirtier, noisy, tones merging, distortion or clipping, noisy attacks, rumbling, muffled, post echo tones, smeared attacks, faser sound, gurgling, noisy hi-hats"

IMHO this abundance of artifacts is due to the different way people DESCRIBE problems. Here there are a lot of people for whom English is not the native language and there are objective difficulties with comments, expecially for newbies/untrained people (myself included  ).  If you read the comments in a "critical" manner you should see that some people named the same artifacts with different terms.

@Roberto: What about to publish anymously the scores for those people who don't want to be named ?
I think that the raw scores and the comments (with or without names) should be published. IIRC this was the method adopted by ff123 for the 64 kbps test.
Title: AAC at 128kbps test - FINISHED
Post by: Ivan Dimkovic on 2003-06-22 16:18:02
Quote
I'm quite surprised, for this. I remember a previous thread where Ivan Dimkovic stated this codec (AACdemo) "as the state of art" around 128 kbps. But i'm not here to start a flame, i respect the other's impressions and the tests conclusions.


IMHO, Sorenson and AACDemo make different streams - similar to some extent, but different - I can ABX these two on many impulse clips.

It might seem that encoder used in Sorenson has TNS artifacts (especially audible on clips like Fatboy) that are not present in similar products sharing the FhG's code (FhG's professional AAC, Dolby's professional AAC, etc..)

Also, what is more important - some versions of the FhG's AAC codec use low-pass which is approx. 0.5 kHz lower - making sound a little bit dull at 128 kbps.  Sorenson is one of those.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-22 19:41:29
I see a couple of results that confuse me:

1)

http://rarewares.hydrogenaudio.org/test/aa...Z-results03.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results03/JAZ-results03.txt)

The rating is 5.0.  How is this possible?  ABC/HR should not have recorded a value of 5.0, or is there some sort of bug somewhere?


2)

http://rarewares.hydrogenaudio.org/test/aa...alf_results.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results12/Gecko_sample12_first_half_results.txt)
http://rarewares.hydrogenaudio.org/test/aa...alf_results.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results12/Gecko_sample12_second_half_results.txt)

Hopefully, the two separate ratings were combined afterwards by the listener to yield a composite rating.

ff123
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-22 20:04:11
Quote
I see a couple of results that confuse me:

1)

http://rarewares.hydrogenaudio.org/test/aa...Z-results03.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results03/JAZ-results03.txt)

The rating is 5.0.  How is this possible?  ABC/HR should not have recorded a value of 5.0, or is there some sort of bug somewhere?

BUSTED!!!!

Quote
2)

http://rarewares.hydrogenaudio.org/test/aa...alf_results.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results12/Gecko_sample12_first_half_results.txt)
http://rarewares.hydrogenaudio.org/test/aa...alf_results.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results12/Gecko_sample12_second_half_results.txt)

Hopefully, the two separate ratings were combined afterwards by the listener to yield a composite rating.


I merged the results together (sum/2). That should produce a fair overall result for that test.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-22 20:09:11
Quote
,Jun 22 2003 - 11:34 AM] @Roberto: What about to publish anymously the scores for those people who don't want to be named ?
I think that the raw scores and the comments (with or without names) should be published. IIRC this was the method adopted by ff123 for the 64 kbps test.

Hrm.. the problem is the way I put it in the readme.

Quote
If you want to allow me to publish your comments and/or scores,
  please make that clear in the e-mail body. If there's no
  mention of it, I WILL NOT publish the comments, and the scores
  will only be mentioned in the final results, anonimously.


Next test, I'll make the publish thing differently, saying that comments will be published anyway and if you don't want to be anonymous, say so.

BTW, added the results of JohnV and ff123.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-22 20:24:10
I think the next test should mention something about artifact training (not just training
about how to use ABC/HR).  It looks like artifact training helped at least one person.

Experienced listeners should be able to skip over the training.  It probably wasn't a big
issue for this test, because it looks like most people who participated probably had some
experience with listening for artifacts.

ff123
Title: AAC at 128kbps test - FINISHED
Post by: SometimesWarrior on 2003-06-22 21:36:39
Man, Gecko tossed all the codecs into the rubbish bin!  I wish I had spread my scores out, to better differentiate the scores and make my results a bit more statistically significant.

I'm glad that some comments were pretty extensive. I'll go back to the samples and listen for the mentioned artifacts. IMO, after ff123's "artifact basic training", listening for artifacts in the 64kbps and 128kbps tests is the obvious next step... like real-world artifact combat, I guess.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-22 21:50:02
Quote
Man, Gecko tossed all the codecs into the rubbish bin!  I wish I had spread my scores out, to better differentiate the scores and make my results a bit more statistically significant.

It's possible to come up with a crude "listener sensitivity" index, just by adding up the total scores on all samples.  In general, the lower a listener scored, the more sensitive he must be to artifacting.  Of course, people don't necessarily use the same part of the rating scale -- that's why I said the index is crude.

I've noticed that I'm pretty much "middle of the road" as far as hearing artifacts is concerned -- I hear some things, but I know I don't hear as much as others do.

For small listening tests, it would be useful to call on those people who are especially sensitive.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-23 00:19:26
Here's several more discrepant results:

http://rarewares.hydrogenaudio.org/test/aa...Z-results02.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results02/JAZ-results02.txt)
http://rarewares.hydrogenaudio.org/test/aa...Z-results04.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results04/JAZ-results04.txt)
http://rarewares.hydrogenaudio.org/test/aa...Z-results07.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results07/JAZ-results07.txt)
http://rarewares.hydrogenaudio.org/test/aa...Z-results08.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results08/JAZ-results08.txt)
http://rarewares.hydrogenaudio.org/test/aa...Z-results09.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results09/JAZ-results09.txt)

Obviously, the results files should not be edited after they have been saved, which is what it looks like happened here.

ff123

Edit:  Oops.  Looks like the test proctor did the editing.
Title: AAC at 128kbps test - FINISHED
Post by: rjamorim on 2003-06-23 00:24:47
Quote
Here's another discrepant result:

http://rarewares.hydrogenaudio.org/test/aa...Z-results08.txt (http://rarewares.hydrogenaudio.org/test/aac128test/results08/JAZ-results08.txt)

The 3L on the left of the bottom portion doesn't match up with the name of the file on the top portion.  Again, a rating of 5.0 was assigned.

ff123

Ah. Duh. That one was me. So that your process program would add all the 5.0 results to the list and not output errors to the listing file.

I will revert the mess uploading the original results files (empty) that he sent me.
Title: AAC at 128kbps test - FINISHED
Post by: ff123 on 2003-06-23 00:28:48
Quote
Ah. Duh. That one was me. So that your process program would add all the 5.0 results to the list and not output errors to the listing file.

I will revert the mess uploading the original results files (empty) that he sent me.

Ah, that explains everything.  Thanks.  The process program can be easily modified to not output warnings for results files with all 5.0's
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-23 00:36:28
Here are my own results :
http://membres.lycos.fr/guruboolez/AUDIO/a...128/tableau.txt (http://membres.lycos.fr/guruboolez/AUDIO/aac128/tableau.txt)

Quote
In general, the lower a listener scored, the more sensitive he must be to artifacting.  Of course, people don't necessarily use the same part of the rating scale -- that's why I said the index is crude.


I'm totally agree with you. Exemple : beautylept.
Proxima, JohnV and Gecko are severe (as me), and others people seem to appreciate a lot the results of the different encoder on this harpsichord sample.
We can't consider garf as an inexperienced listener, but his rating was really high (3.8 - 4.3 & others > 4.5....). For comparison, Gecko is the most disapointed (average : ~2.0 / max = 2.5)

Gecko (http://rarewares.hydrogenaudio.org/test/aac128test/results03/Gecko_sample03_results.txt)
Garf (http://rarewares.hydrogenaudio.org/test/aac128test/results03/garf03.txt)
Title: AAC at 128kbps test - FINISHED
Post by: Garf on 2003-06-23 00:47:21
There was another case in which I'd considered all codecs unlistenable (2.0 highest score), but I saw other people giving high grades. All a matter of preference.
Title: AAC at 128kbps test - FINISHED
Post by: guruboolez on 2003-06-23 01:03:32
Quote
There was another case in which I'd considered all codecs unlistenable (2.0 highest score), but I saw other people giving high grades. All a matter of preference.

Lifeshatters (http://rarewares.hydrogenaudio.org/test/aac128test/results08/garf08.txt) I suppose. I have at contrairy some difficulties to ABX similar samples : they are deafening and boring me at the same time. I can understand, now, some reaction with harpsichord music
Title: AAC at 128kbps test - FINISHED
Post by: Gecko on 2003-06-25 21:47:36
Quote
Man, Gecko tossed all the codecs into the rubbish bin!  I wish I had spread my scores out, to better differentiate the scores and make my results a bit more statistically significant.

I didn't know I was such a "rigorous" judge.  I find it difficult to give consistent ratings. It's all very subjective and possibly different from sample to sample. If I'm grudgy, I might give lower scores etc. (Allthough I don't remember being in any extreme mood when I did these tests). I try hard to use the verbal scale for rating. Notice that 3.0 is still only slightly annoying. I quickly skimmed my results and found that I did rate LifeShatters_2.wav a solid 4.5. That's better than "perceptible but not annoying"! Maybe more emphasis should be put on the verbal scale rather than the pure numbers. On a scale from 1 to 5 a three is only mediocre, while the verbal description is better than that imo. Maybe that confuses people.

I didn't give the encoders a "128k bonus". What I want from a codec/format is transparency. I guess (at 128k) that is also part of the design goal and for some listeners it has been met on some samples, so I'm asking much, but not too much, imo. Also, I don't have any other reference than the original, so what else should I rate against? I would have given higher scores if the question would have been: considering the bitrate, how do you think these encoders perform? I do admit that "annoying" is also a very subjective rating and personally, I'm annoyed pretty easily by artifacts. Basically, if it doesn't sound like the original, then it's not acceptable for me (allthough some things like a slight lowpass can be OK).

Two more things to consider: My equipment is mediocre at best. (I recently upgraded to an Aureon Sky though. ). I'm pretty familiar with the test samples from previous testing.