HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: rjamorim on 2004-02-07 23:59:49

Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-07 23:59:49
Hello.

I'm creating this thread so that we can discuss how the upcoming AAC @ 128kbps listening test will be handled.

The encoders I plan to feature are:

-Nero AAC encoder VBR profile Streaming :: Medium  (@Ivan: Fast or HQ mode?)
-Apple iTunes 4.2 128kbps
-FAAC "whatever VBR setting comes close to 128kbps"
-Compaact! "same thing as above"
-Winamp AAC encoder 128kbps
-FhG l3enc 1.0 as anchor

Some codecs that might be considered to replace another:
-Real Producer
-NCTU AAC
-Emuzed (http://www.emuzed.com)
-Imagine Technology (http://www.imaginetechnology.net/)


I am personally very fond of the idea of using l3enc as anchor. It'll be a good insight on how perceptual audio coding developed since the first MP3 implementation. It'll be like "the oldest vs. the newest".

Of course, as usual, EVERYTHING is still open to discussion. Remember I am conducing these tests for you, not for me, so please make your opinions heard.

Actually, only one thing is not open for discussion: the amount of codecs. There won't be more than 6 (including anchor), no matter what.

Some other topics I want to discuss:

-Samples tested. IMO we should definitely replace some samples. Specifically Polonaise (I would suggest Fossiles as replacement) and Illinois (I would replace it with our old friend Fatboy). Do you believe something else should be replaced?

I wouldn't like to replace Waiting. I love that sample for it's wacky behaviour, so I want to force you guys to listen to it over and over again until your ears pop out of your skull :B


-Should only LC be tested? Or maybe test other (better?) profiles when the encoder supports them, like FAAC and Compaact. I think an argument for this would be that it's like VBR vs. CBR - you shouldn't penalize some codecs because others lack that feature. A very good argument against this would be that LC is waaaay more supported now and probably in the future. Opinions?


-Do you think there is any need for encryption (it will definitely be used on the Multiformat tests)? If so, we'll only use Schnofler's comparator, since ff123's doesn't support encryption.


These are the subjects I can think about ATM. I'll post more ideas/questions as I come up with them.

Thanks for your help.

Best regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: Bonzi on 2004-02-08 00:12:53
Well, isn't winamp AAC practically the same thing as quicktime AAC except an older version?  I would be in favour of NCTU AACenc instead if this is the case.  It would be interesting to see how it would fare in comparison to other codecs which rely less on ODG and other objective measures in development.
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-08 00:18:33
Quote
Well, I am personally very fond of the idea of using l3enc as anchor. It'll be a good insight on how perceptual audio coding developed since the first MP3 implementation. It'll be like "the oldest vs. the newest".


Yeah, but what will you say when it comes in third?

P.S.: Come on, somebody had to say it.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 00:19:01
Quote
It would be interesting to see how it would fare in comparison to other codecs which rely less on ODG and other objective measures in development.

That is a very good point, indeed.

Quote
Yeah, but what will you say when it comes in third?


I will give up all this mess and become a hermit
Title: AAC @ 128kbps listening test discussion
Post by: sony666 on 2004-02-08 00:23:16
6 encoders is the absolute maximum imho, I'd rather drop the anchor to be honest.

128kbit AAC will likely be even harder than the mp3 test, and listening gets a little tedious after the 3rd or 4th sample, so adding exotics like NCTU is a big nono
Title: AAC @ 128kbps listening test discussion
Post by: guest0101 on 2004-02-08 00:24:05
I would like to see you keep the WinAmp encoder due to its widespread use among users. Would be nice to know how well it encodes. Also I would like to know of Imagine Technology's codec as I use that with their Cool Edit Pro/Adobe Audition plug-in. Thanks rjamorim for all the kind advice your post on HA. You're very helpful!
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 00:25:53
Quote
6 encoders is the absolute maximum imho, I'd rather drop the anchor to be honest.

No, there must be a bottom anchor. It doesn't have to be l3enc, but it's a good way to keep things in perspective.
Title: AAC @ 128kbps listening test discussion
Post by: harashin on 2004-02-08 00:25:55
How about VQF as anchor? According to AudioCodingWiki (http://www.audiocoding.com/wiki/index.php?page=VQF) , it is(was?) a part of MPEG-4 Audio version 1.
Title: AAC @ 128kbps listening test discussion
Post by: Cygnus X1 on 2004-02-08 00:30:37
Re: the profile issue: I'd say just stick with the LC profile, for the simple fact that I'm not personally aware of any hardware player that can handle higher profiles like Main. Plus, I also question whether using the more complex profiles drastically improves sound quality (i.e., a statistically significant improvement over LC). I haven't seen any data comparing the different profiles of popular encoders (FAAC, Nero, Dolby, et al), so while higher profiles should sound better in theory, to what extent (if any) has yet to be studied.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 00:31:28
Quote
How about VQF as anchor? According to AudioCodingWiki (http://www.audiocoding.com/wiki/index.php?page=VQF) , it is(was?) a part of MPEG-4 Audio version 1.

Actually, TwinVQ is part of MPEG4 audio. It's slightly different from VQF, and in MPEG4 audio it's limited to veeeery low bitrates - around 8-16kbps

That said, I was planning to use it as an anchor in the multiformat test that will come after the AAC test.
Title: AAC @ 128kbps listening test discussion
Post by: sony666 on 2004-02-08 00:39:13
Quote
Quote
6 encoders is the absolute maximum imho, I'd rather drop the anchor to be honest.

No, there must be a bottom anchor. It doesn't have to be l3enc, but it's a good way to keep things in perspective.

I trust in your greater wisdom
Also, thank you for going through the trouble once again.

For the anchor, FhG 1.0 would surely be funny, but I persoanlly would prefer something that is used more widely in old files and has a bad name to it. (hint: Blade 128k) Maybe it is not so bad as many ppl like to call it?

For the profile, only plain LC please

For samples: I would surely like to see "mybloodrusts" and "daFunk" from the mp3 test
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 00:41:26
Quote
but I persoanlly would prefer something that is used more widely in old files and has a bad name to it. (hint: Blade 128k)

I already ruined Blade's reputation

Title: AAC @ 128kbps listening test discussion
Post by: harashin on 2004-02-08 00:45:29
Quote
Actually, TwinVQ is part of MPEG4 audio. It's slightly different from VQF, and in MPEG4 audio it's limited to veeeery low bitrates - around 8-16kbps

That said, I was planning to use it as an anchor in the multiformat test that will come after the AAC test.

Oh. Thanks for the clarification.
Title: AAC @ 128kbps listening test discussion
Post by: sony666 on 2004-02-08 00:59:37
Quote
<scientific graph that proves Blade 128k, indeed, stinks>

Ack.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-08 01:10:11
Quote
Actually, only one thing is not open for discussion: the amount of codecs. There won't be more than 6 (including anchor), no matter what.

dont get that...
one codec more would make sense to me!   

and i would choose real for this extra codec, as i dont like their 192kbps is better than 128kbps marketing blabla...
first of all i want to know how their codec does compared to itunes
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 01:16:45
Quote
dont get that...
one codec more would make sense to me!   

I guess you have no idea the utter pain in testing several (6+) encoders, most of them getting pretty close to transparency :B

hehe

BTW, I would expect Real to be similar to Winamp.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-08 01:24:19
Quote
I guess you have no idea the utter pain in testing several (6+) encoders, most of them getting pretty close to transparency
well as i attended every test till now, also the first one, i think i have a small idea of what this means

but when it will be finished i think most will say "it was worth it"

anyways you said the codec amount is not to be discussed and i respect this 
and i also dont want that real with its .ra sh*t gets popular
Title: AAC @ 128kbps listening test discussion
Post by: karl_lillevold on 2004-02-08 01:49:02
Quote
i also dont want that real with its .ra sh*t gets popular

RM is just another container format, playable in many players these days, but I have already mentioned that AAC wrapped in RM is just a temporary solution. Our alternative file writers were not ready in time for Beta. RealPlayer 10 Gold and RealProducer (Helix Producer) 10 Gold will write AAC to another format...

The Real (HE-)AAC encoder was licensed from Coding Technologies, is the only HE capable encoder in addition to Nero (CT did invent aacPlus/HE by the way), and their AAC base encoder probably started out from a similar codebase as the FhG AAC encoder. Personally I would be very curious how it compares to the rest of the AAC encoders out there. "Installing" Helix Producer is as simple as unzipping the files, and if included in the test, I will make sure Roberto has no trouble whatsoever with the encoding or installation (don't want to repeat trying to install RealPlayer.. ), as well as losslessly convert the .ra files to whichever format is preferred, .aac, or .m4a.

As a sidenote, I think it is a good idea to be very clear about which is the actual codec provider, for example for WinAmp, who did they license the encoder from? For Real, like I mentioned, this would be Coding Technologies.
Title: AAC @ 128kbps listening test discussion
Post by: ssjkakaroto on 2004-02-08 02:12:48
e ai Roberto

would it make any sense using LAME instead of FhG l3enc? that way we could see how much better (if better at all ) AAC is when compared with the best MP3 encoder
I wouldn't be surprised at all if it came out in first
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 02:37:38
Quote
As a sidenote, I think it is a good idea to be very clear about which is the actual codec provider, for example for WinAmp, who did they license the encoder from? For Real, like I mentioned, this would be Coding Technologies.

Nullsoft (AOL actually) licensed from Dolby.

CodingTechnologies licensed their base AAC encoder from FhG (FhG consumer AAC encoder, actually)

FhG and Dolby share a lot of codebase, according to an AAC developer.

That's why I guess they would be similar

Maybe I should start a poll with AAC codec options? What do you think?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 02:39:12
Quote
would it make any sense using LAME instead of FhG l3enc? that way we could see how much better (if better at all ) AAC is when compared with the best MP3 encoder
I wouldn't be surprised at all if it came out in first

Dude, you pointed out the exact problem in your suggestion. How can I use an anchor that might even surpass some of the actual competitors? :B
Title: AAC @ 128kbps listening test discussion
Post by: kwanbis on 2004-02-08 03:19:19
Quote
Quote
would it make any sense using LAME instead of FhG l3enc? that way we could see how much better (if better at all ) AAC is when compared with the best MP3 encoder
I wouldn't be surprised at all if it came out in first

Dude, you pointed out the exact problem in your suggestion. How can I use an anchor that might even surpass some of the actual competitors? :B

well ... you mean it to be an anchor ... but what happened to xing? ... i second the opinion of using LAME ...
Title: AAC @ 128kbps listening test discussion
Post by: guest0101 on 2004-02-08 03:20:56
Well I think Apple iTunes (Quicktime 6.5) and Nero 6 should be in for sure since they are 2 quite different codecs and 2 of the most popular AAC encoder apps. Apple is based on Dolby Consumer I believe with heavy tweaking and Nero is heavily tweaked and is based on the Coding Technoligies codec I believe. I thought WinAmp would make a good baseline Dolby Consumer (without tweaking) comparison against the iTunes/Quicktime 6.5 encoder.

That would take care of 3 of your 6 choices in my opinion.

RealNetworks would be nice, to see a mostly untweaked Coding Technologies codec compared against Nero (heavily tweaked) one. Hopefully RealPlayer 10 will play .m4a and .mp4 (including HE AAC) natively when it gets out of beta
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 03:33:26
Quote
well ... you mean it to be an anchor ... but what happened to xing? ...

Blah!!!

Please consider all the bad comments on Xing done here and elsewhere over the years, and then come tell me I did a bad choice!
Title: AAC @ 128kbps listening test discussion
Post by: bidz on 2004-02-08 03:34:50
No more than 6 codecs, as 6 is already enough. I also second the latest LAME version for anchor. Atleast keep iTunes, Nero and Winamp in the test, as these are the most widespread codecs. RA's AAC codec might also be more spread than some of the others (NCTU, etc) once it's out of beta, so i'd probably want to include that one also, even though all these codecs may be based on the same codebase - optimizations might be different.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 03:36:37
Quote
Nero is heavily tweaked and is based on the Coding Technoligies codec I believe.

Nero is 100% in-house development

Quote
That would take care of 3 of your 6 choices in my opinion.


FAAC must be definitely in, because of implications of open source/yadda yadda.

I actually think Faac is more essential in this test than Winamp

Quote
RealNetworks would be nice, to see a mostly untweaked Coding Technologies codec compared against Nero (heavily tweaked) one.


Real codec is absolutely unrelated to Nero, as I explained above. So, no, it won't work to compare them in these grounds.
Title: AAC @ 128kbps listening test discussion
Post by: kwanbis on 2004-02-08 03:36:54
Quote
Quote
well ... you mean it to be an anchor ... but what happened to xing? ...

Blah!!!

Please consider all the bad comments on Xing done here and elsewhere over the years, and then come tell me I did a bad choice!

it sure it was a good choice, but it, turning not into the anchor, didn't harm the test ...
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 03:42:37
Quote
it sure it was a good choice, but it, turning not into the anchor, didn't harm the test ...

It harm in the aspect that there's no perspective. Codecs dipped too low on scores. I wanted Xing to be like Blade was in the first 128kbps multiformat test. Always at the bottom, to avoid some otherwise reasonable codec being labeled very bad because others were better.
Title: AAC @ 128kbps listening test discussion
Post by: guest0101 on 2004-02-08 03:58:18
I really like you original choices for the 5 AAC encoders (not including the anchor which appears to be open to much debate):

-Nero AAC encoder VBR profile Streaming :: Medium (@Ivan: Fast or HQ mode?)
-Apple iTunes 4.2 128kbps
-FAAC "whatever VBR setting comes close to 128kbps"
-Compaact! "same thing as above"
-Winamp AAC encoder 128kbps

Perhaps you will stick with them, as that selection should give you a smattering of the various AAC encoders. I trust your judgment on this rjamorim
Title: AAC @ 128kbps listening test discussion
Post by: westgroveg on 2004-02-08 04:11:54
What about FhG AAC (http://www.iis.fraunhofer.de/amm/download/) ?

I'm sure we can get permission / find a way to encode samples for test purposes
Title: AAC @ 128kbps listening test discussion
Post by: MGuti on 2004-02-08 04:16:37
nero is very important
i would like to see real as compared to itunes
FAAC is a must (NCTU-AAC was based on FAAC i believe)
throw in winamp if you really want to
for the anchor use the itunes MP3 (we know it sucks and will act as a comparison between itunes AAC and MP3)

just my opinion, as long as nero, itunes, and faac are in there i'll be happy.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 04:21:16
Quote
What about FhG AAC (http://www.iis.fraunhofer.de/amm/download/) ?

I'm sure we can get permission / find a way to encode samples for test purposes

Well, I agree that would be interesting, but I am not fond of testing codecs noone else has access to.

I mean, so what that it won, I can't use it anyway... :/
Title: AAC @ 128kbps listening test discussion
Post by: Dologan on 2004-02-08 06:05:54
I support Roberto's initial codec choices fully, including l3enc as lower anchor. 
More than six codecs is absolutely insane and those that ask for it have probably never participated fully in one of Roberto's tests, or have an unhealthy amount of free time and patience.
None of the other choices IMO merits the place of any of the initially chosen contestants, since they are either 1) In beta or not-quite-final state 2) not really popular or relevant enough.
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-08 11:49:03
I also think Roberto's choice of codecs is nice as it is. The only other codec I'd be interested in, is the NCTU codec, for the same reason Bonzi stated in the second post of this thread (proving that listening tests are superior to ODG results). But none of the codecs of the original line-up seems debatable to me, with the possible exception of compaact (but I'd prefer testing a serious contender instead of just slapping some encoder which isn't really used anyway).
Lame as an anchor is a bad idea in my opinion. Not only because it might repeat the "Xing disaster", but because it would considerably increase the difficulty of the test and it's difficult enough as it is right now. With Lame there'd be six real codecs instead of five codecs and one ... well ... joke. And it will help general motivation if there's at least one codec everyone should be able to identify.

Now to an interesting question
Quote
-Should only LC be tested? Or maybe test other (better?) profiles when the encoder supports them, like FAAC and Compaact. I think an argument for this would be that it's like VBR vs. CBR - you shouldn't penalize some codecs because others lack that feature. A very good argument against this would be that LC is waaaay more supported now and probably in the future. Opinions?

I haven't really made up my mind about that one. The problem, of course, is that a comparison between an LC codec and one which uses additional tools will be completely useless to people who plan on using AAC on their portables. But it might also provide a very interesting outlook into the future. The latter, though, only holds if we can really expect considerable advantages from using the additional features. I'd really like to hear some comments from the developers on this. If the advantages are only slim, there's no point in using them, in my opinion. But if we can really expect a definite increase in quality, maybe it would be interesting to have FAAC use them, to see how it compares to the supposedly better, but then somewhat handicapped, commercial encoders.
But these are only some thoughts, I haven't really decided for myself yet whether that's a good idea.
Title: AAC @ 128kbps listening test discussion
Post by: Gabriel on 2004-02-08 12:42:18
About the anchor:

I would very much like to see Lame in this test, as this would allow us (and me) to see how it performs against current AAC codecs.

However, it would make the test harder, and with less defined scale. Using something that we think will really be a low anchor is a better idea. This way the scale of notation will probably be more uniform.
Using l3enc 1.0 is, from an "historical" point of view very interesting.

So even if I would be interested to see Lame compared to AAC codecs, I think that it would be better for the purpose of this test (which is to compare AAC codecs) to use another anchor. I like the idea of l3enc 1.0.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-08 12:45:00
doesnt faac maybe do well as "anchor" (maybe bringing the same or worse quality as lame)?
Title: AAC @ 128kbps listening test discussion
Post by: kl33per on 2004-02-08 12:53:33
Obviously both the Apple and Nero codecs should be included, to see how they've each progressed since the last test.  Furthermore, I think compaact should definately been included, and as it seems to be undergoing fairly regular development as well as trying to establish an active user base here at HA, I think it's time we got an evaluation of it's quality.  I would also like to see how far FAAC has come being a free alternative.  Finaly, I'm probably most interested in the NCTU codec for the reasons Bonzi has stated.  As Winamp should be similar to iTunes (and apparantly Real as well) I think it should be dropped.  I think the anchor is also good choice and is deffinately a requirement.

On the issue of Lame.  Clearly in the last Multiformat test, LAME was behind by a fair margin.  Furthermore, this is specifically a AAC test and should not feature other formats.
Title: AAC @ 128kbps listening test discussion
Post by: dev0 on 2004-02-08 13:06:23
Quote
doesnt faac maybe do well as "anchor" (maybe bringing the same or worse quality as lame)?

FAAC has improved a lot recently and a lot of people will be suprised to hear how well it performs at 128kbps.
I've already found samples on which it performs better than LAME and I wouldn't be suprised, if their overall performace would turn out to be similiar. This is pure speculation though and not backed up by any tests (yet).
Gabriel's idea of including LAME seems very interesting to me, but LAME's performance is probably too good to serve as an anchor.

dev0
Title: AAC @ 128kbps listening test discussion
Post by: Gabriel on 2004-02-08 13:09:43
Quote
Gabriel's idea of including LAME seems very interesting to me


Did you read my post? Perhaps I did not managed to properly expose my opinion?
Title: AAC @ 128kbps listening test discussion
Post by: ErikS on 2004-02-08 13:41:11
1. Who uses Compaact? What's the reason behind including it in this test? (Not intending to sound demeaning in any way - just curious since I never seen it being used anywhere...)

2. Please replace the Rite of Spring sample! It drives me mad... 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 13:58:19
Quote
2. Please replace the Rite of Spring sample! It drives me mad... 

No problem. But please suggest another sample to replace it

Edit: Woot! 4444 posts
Title: AAC @ 128kbps listening test discussion
Post by: Alexander Lerch on 2004-02-08 14:01:53
Hi,

you all know I am biased. That said, I would really like to see compaact! in this test. Surely it has not so much users as Nero, but it is only 3 months old, and we are a very small company, not being able to make so much marketing. I personally think that compaact! has potential for being a popular encoder.

Regarding different profiles:
The main profile is not useless, it gives higher quality than the LC profile. However, the quality increase is not great (but of course greater than with the silly LTP profile).
I personally think every competing encoder should give its best, regarding settings and profiles in this test. But again, I'm biased...

Regards,
Alexander
Title: AAC @ 128kbps listening test discussion
Post by: Sebastian Mares on 2004-02-08 14:18:18
Well, I think the following codecs should be included:I would either use l3enc or LAME for the 6th encoder.
Title: AAC @ 128kbps listening test discussion
Post by: Latexxx on 2004-02-08 14:22:49
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-08 14:31:51
Quote
I am personally very fond of the idea of using l3enc as anchor. It'll be a good insight on how perceptual audio coding developed since the first MP3 implementation. It'll be like "the oldest vs. the newest".

I would vote to use FAAC1.17 as an anchor since it's much worse than latest FAAC and it could be interesting to compare both versions.
If you don't like FAAC1.17 then I would vote for LAME to be an anchor
(even if it turns out not to be the worst it's still very interesting comparison)

Quote
I wouldn't like to replace Waiting. I love that sample for it's wacky behaviour, so I want to force you guys to listen to it over and over again until your ears pop out of your skull :B

Yes, Waiting is definitely not to be removed.

Quote
-Should only LC be tested? Or maybe test other (better?) profiles when the encoder supports them, like FAAC and Compaact. I think an argument for this would be that it's like VBR vs. CBR - you shouldn't penalize some codecs because others lack that feature. A very good argument against this would be that LC is waaaay more supported now and probably in the future. Opinions?

Those other profiles in FAAC are likely to be broken so it shouldn't be used.
Title: AAC @ 128kbps listening test discussion
Post by: MGuti on 2004-02-08 15:40:59
since this will be a 128 test HE for nero won't help. im not sure what the main profile for AAC really is (and no idea what the difference is for LC). i think that all codecs should use LC simply because it is the setting that can be used on portables.  i don't know about anyone else, but the only time i would encode to 128 is if i had limited space on a portable. otherwise i would aim for a transparancy setting.

i think an early version of FAAC is a good idea. i would like to see how the open source encoder has progressed. besides won't lame be against AAC and the other codecs in the next multiformat test? there is no point in using it as an anchor since is might end up like Xing. pick a truely poor encoder.

Nero
itunes (real and winamp are assumed to be similar)
FAAC
compaact
NCTU-AAC (real or winamp could replace, but since NCTU doesn't use short blocks - or something to that effect - i would like to see how it performs)
FAAC 1.17 - anchor

oh, and will there be an vorbis test before the multiformat test to compare the various tunes?
Title: AAC @ 128kbps listening test discussion
Post by: dev0 on 2004-02-08 15:48:05
I'm completely opposed to including NCTU, until they have sorted out their licensing issues.
rjamorim's original selection seems to be the most sensible to me and l3enc would probably do fine as an anchor.

dev0
Title: AAC @ 128kbps listening test discussion
Post by: magic75 on 2004-02-08 15:57:00
Quote
Quote
Quote
well ... you mean it to be an anchor ... but what happened to xing? ...

Blah!!!

Please consider all the bad comments on Xing done here and elsewhere over the years, and then come tell me I did a bad choice!

it sure it was a good choice, but it, turning not into the anchor, didn't harm the test ... 

Well, I don't think you can compare it like that. Xing was chosen because pretty much everybody thought it would fail miserably. You can't say the same thing about Lame in this test. Using Lame here is to much risk messing up the point of the test: Which is the best AAC encoder?

In the MP3 case I think we were lucky to have iTunes serving as a "light" anchor.

And I guess we will see Lame against the AAC winner in the next multi-format test?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 16:07:51
Quote
I'm completely opposed to including NCTU, until they have sorted out their licensing issues.

According to them, they already sorted it out (I.E, started developing an encoder from scratch and gave up developing from Faac)

But still, due to their quite immoral past behaviour, I'm not very inclined to take them seriously and test their encoder.

About Faac 1.17: It has already been tested in the first AAC@128kbps test, so you can use that test to compare how it fared against others. I don't see much point testing an old version again.

But, in MP3's case, I would really like to know how it sounded when it was premiered

Quote
And I guess we will see Lame against the AAC winner in the next multi-format test?


Of course. I hoped to test another encoder since Lame was already tested in the former multiformat test. But since it won the MP3 test... heh

Actually, I think that's another good reason not to test Lame here. It has been tested in the 128kbps extension test, the 64kbps test, the MP3 test, will be tested in the next multiformat test... :B
Title: AAC @ 128kbps listening test discussion
Post by: Tomb on 2004-02-08 16:27:18
This may seem a stupid question but does the Quick Time Professional AAC Codec differ from the I-Tunes one or are they one and the same? I thought I read somewhere that there were slight differences but I cannot remember where I read this!

Depending on the answer I would like to see:
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 16:42:20
Quote
This may seem a stupid question but does the Quick Time Professional AAC Codec differ from the I-Tunes one or are they one and the same? I thought I read somewhere that there were slight differences but I cannot remember where I read this!

They are the same encoder. The difference is that in QuickTime you have the option to encode in three quality modes: Good, Better and Best. iTunes is hard-coded at Better. Best supposedly is the same as Better in 16bit streams, and only shows any improvement in 24bit streams.
Title: AAC @ 128kbps listening test discussion
Post by: mmortal03 on 2004-02-08 17:19:11
Quote
Quote
would it make any sense using LAME instead of FhG l3enc? that way we could see how much better (if better at all ) AAC is when compared with the best MP3 encoder
I wouldn't be surprised at all if it came out in first

Dude, you pointed out the exact problem in your suggestion. How can I use an anchor that might even surpass some of the actual competitors? :B

I doubt LAME at 128kbps would surpass AAC, it was the lowest in the previous 128kbps multi-format test (besides Blade).  What has changed at 128kbps since then that woukld put it over AAC now?  LAME is a good choice as anchor.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 17:22:54
Quote
I doubt LAME at 128kbps would surpass AAC, it was the lowest in the previous 128kbps multi-format test (besides Blade).  What has changed at 128kbps since then that woukld put it over AAC now?

In the multi-format is was worse than QuickTime. But I can't predict how it will behave against Compaact or Faac.

Besides, as I recently posted, Lame has been featured in too many tests. Enough is enough, IMO.
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-08 17:26:37
So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.
Title: AAC @ 128kbps listening test discussion
Post by: emtee on 2004-02-08 17:26:56
@rjamorim:
Maybe you should start a poll to decide what will the anchor be, reminding users that LAME as already been tested in the previous listening tests and will be tested again in the multiformat listening test, after this one.

I would like to see these in the test:For the anchor, i vote for a MPEG-4 codec. Maybe an early version of FAAC or Psytel.
Title: AAC @ 128kbps listening test discussion
Post by: mmortal03 on 2004-02-08 17:30:07
I re-read some of the posts, and I have changed my mind.  The next multi-format will pit LAME against AAC, so let's keep this one an all AAC test, including anchor.

Quote
So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.


I would wait till after this test, as once we see the results, it might not even matter any more to you how it compares to LAME.  These tests have definately had interesting results.  Yours might win


Edit:  BTW, knik, was that entire quote yours, or were you answering someone elses question?  It seems as if you are FOR LAME in the first sentence, but then you say there is no need for it in the second sentence...
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-08 17:34:17
Quote
I would wait till after this test, as once we see the results, it might not even matter any more to you how it compares to LAME.  These tests have definately had interesting results.  Yours might win

I'm sure the results will be very interesting with or without LAME but including LAME would make it even more interesting.
Title: AAC @ 128kbps listening test discussion
Post by: mmortal03 on 2004-02-08 17:36:37
Gotcha, you want to know how LAME does in comparison to the OTHER AAC codecs, not just the winner.  I can understand that.
Title: AAC @ 128kbps listening test discussion
Post by: Continuum on 2004-02-08 17:40:37
Quote
So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.

I second that! The AAC winner might not be useable for all situations, so it's quite interesting to compare every AAC competitor with the best MP3 solution available.

Besides, what's wrong with using Lame in many tests? It allows (to an admittedly small amount) inter-test comparability (Another meaning of the word anchor?), while FhG l3enc 1.0 will only make the used rating range smaller.
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-08 17:55:30
Quote
Quote
So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.


Edit:  BTW, knik, was that entire quote yours, or were you answering someone elses question?  It seems as if you are FOR LAME in the first sentence, but then you say there is no need for it in the second sentence...

The whole point is that including LAME here is at least as useful as including it in the multiformat test.
Title: AAC @ 128kbps listening test discussion
Post by: bidz on 2004-02-08 17:56:37
Quote
Quote
I would wait till after this test, as once we see the results, it might not even matter any more to you how it compares to LAME.  These tests have definately had interesting results.  Yours might win

I'm sure the results will be very interesting with or without LAME but including LAME would make it even more interesting.

Agreed. I really want to know how the latest version of LAME stacks up against most AAC encoders, not just the winner.

Using L3enc as anchor would be "funny", but would it be of any use? i don't think so, but having LAME as anchor would be of use for many people. This way they can see how all AAC encoders, not just the winner (currently Quicktime/iTunes) performs against the best MP3 solution.
Title: AAC @ 128kbps listening test discussion
Post by: MGuti on 2004-02-08 18:11:29
IMO, this is a test to determine the best AAC codec, not compare MP3 and AAC. if you really want to know how LAME (a free mp3 encoder) fairs against FAAC (a free AAC encoder) run asimple test. with 2 encoders it is much easier than with 6 and LAME included.

if early FAAC is out of the question than the early MP3 encoder sounds best. it will act as a good anchor.

in truth, if we REALLY wanted to be able to compare every codec in relation to each other we would have to run a test including every codec at once. that is far too many to do in a single test (time and patience being the limiting factors).

stick with AAC encoders and leave LAME out. you never really know how the test will turn out anyway.
Title: AAC @ 128kbps listening test discussion
Post by: SirGrey on 2004-02-08 19:35:24
I also like the idea to use l3enc as an anchor, but may be it will be better to use it's latest version (2.71 or whatever was the last) ?
To compare to the first mp3 codec in it's best, not the buggy first version ?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 20:44:22
Well, the anchor rating isn't supposed to be useful to users. It's supposed to be bad (Lame doesn't fit that criteria) and put things into perspective. That's why formal tests would rather use lowpass as anchor

Let's be realistic: If Lame is included, it won't be an anchor. It will just be another competitor. An MP3 competitor in an AAC test. You probably notice the mess here...

What people seem to forget is that the bottom anchor is supposed to fail! It's not there to compare against the actual competitors.

Anyway, that's my point of view. If the majority of users want, I'll feature lame. But I'll be sincere: I don't like the idea of adding a MP3 competitor to an AAC test.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: kwanbis on 2004-02-08 21:23:40
Quote
Well, the anchor rating isn't supposed to be useful to users. It's supposed to be bad (Lame doesn't fit that criteria) and put things into perspective. That's why formal tests would rather use lowpass as anchor

...

What people seem to forget is that the bottom anchor is supposed to fail! It's not there to compare against the actual competitors.

roberto, i respect your idea, but i don't share it ... i mean, and even you said it i think, the test are for the community, as such, for me at least, and a poll could be well be used, it is very interesting to know if the "actual champ", is still up to the task, compared to the "title challenger" ... for me including LAME would be very nice for the test ... it would mean something like "should i bother to re-rip into AAC, or can i still keep my LAME files"? ... lets do a poll

PS: i hoppe we can still go next week to picaso's display at san pablo

EDIT: this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-08 21:48:56
Quote
So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

Including Lame simply to have a comparison Lame/FAAC is not very convincing to me. There are lots of comparisons you have to leave out in every listening test (why not include Xing to see how it fares against compaact?), and I can't see why Lame/FAAC should be so interesting that it would merit an inclusion of Lame in this test (but I'm open for arguments).

Quote
Besides, what's wrong with using Lame in many tests? It allows (to an admittedly small amount) inter-test comparability

Inlcuding Lame for inter-test comparability could be interesting but unless results are really clear-cut, you couldn't draw many safe conclusions. For example, if Lame finishes last in the AAC test you could claim with some amount of certainty that all the AAC encoders are better than all the mp3 encoders from the mp3 test. But if, say, Lame finishes last in both the AAC test and the multiformat test (excluding anchors, the latter is to be expected), you will only provoke senseless claims like "Vorbis finished 1.2 points ahead of Lame in the multiformat test, but FAAC only managed an advantage of 0.9 in the AAC test, so Vorbis must be better than FAAC".

Quote
this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

That's what the upcoming multiformat test is for.

I'm still against the inclusion of Lame. In my opinion, the disadvantage of increasing the difficulty level of the test outweighs any advantages. You have to be realistic about this. There were on average 16 participants in the mp3 test, and I'm convinced the number of participants depends very much on the difficulty, so if we add another serious contender to this test, we can probably expect even fewer participants (unless Roberto includes a bunch of problem samples). So, even if the results get a bit less interesting without Lame, in my opinion significance beats interestingness any day of the week.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-08 21:56:53
Quote
EDIT: this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

The first multiformat test already proved that. The best AAC implementation (QuickTime) was quite better than the best MP3 implementation (Lame)
Title: AAC @ 128kbps listening test discussion
Post by: QuantumKnot on 2004-02-09 00:27:49
I second the idea of using Xing (ver 1.5) as an anchor.  It seems to be one of the most popular mp3 encoders used by consumers out there.  It's kinda outdated, not tuned since 1999, doesn't use block switching, etc.  So it's not going to compete with any of the AAC coders, yet represents the consumer standard
Title: AAC @ 128kbps listening test discussion
Post by: kwanbis on 2004-02-09 01:21:37
Quote
Quote
EDIT: this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

The first multiformat test already proved that. The best AAC implementation (QuickTime) was quite better than the best MP3 implementation (Lame)

right ... i didn't remembered that test ...
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 01:54:45
Well, I see there is too much discussion about what codecs to support and what anchor to use. I think the best and most fair way to settle the issues is creating a poll.

A poll to select the anchor would be easy, but codec will prove complicated since there's no way to set IPB to allow users to choose 5 options.

So, I will use codec packs as options. Something like this:

1. - iTunes, Nero, Faac, Compaact and Winamp
2. - iTunes, Nero, Faac, Real and NCTU
...

Since there would be several different possibilities involved, I would like to see if everyone agrees on a list of codecs that MUST be present.

1 - iTunes: Because it was the winner of the first AAC test and it's hugely popular
2 - Nero: Because it's Nero (heh) and it's popular.
3 - Faac: Because it's the only open source and really multiplatform alternative, and because I believe it's quite popular as well.

From there, I would create different packs featuring Compaact and/or Winamp and/or NCTU and/or Real.

As an added bonus, I will get rid of accusations of favouring this or that codec.

Do you agree?

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: paradynamic on 2004-02-09 02:10:34
Why not use Fhg AAC encoder (Sorenson?) for low anchor?  Would be interesting to see progress of recent aac development.
Title: AAC @ 128kbps listening test discussion
Post by: quackquack on 2004-02-09 02:18:05
I think Compaact is essential as right now it is the only VBR-able competition to Nero.  I know it has undergone a number of quality improvements as well, and it would be interesting to see how it stacks up.

- Matt
Title: AAC @ 128kbps listening test discussion
Post by: kwanbis on 2004-02-09 02:25:21
only question is ... haw -APS would compare agains AAC?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 02:37:37
Quote
Why not use Fhg AAC encoder (Sorenson?) for low anchor?  Would be interesting to see progress of recent aac development.

Because it might be better than other codecs.

Quote
I think Compaact is essential as right now it is the only VBR-able competition to Nero.


Ehm... FAAC?

Quote
only question is ... haw -APS would compare agains AAC?


 

Please, don't tell me you are suggesting I test different bitrates...
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-09 03:05:54
Probably the choice of anchor could be a separate poll in itself.  But to reiterate a couple of points made by others in this thread, the anchor

1) should be significantly worse than any of the other codecs.

2) should sound bad enough to be readily identifiable.  Yes, that will compress the ratings, but it will take a lot of the burden off the listener, who now need concentrate on one less codec.  Also, it may encourage more people to participate, who might otherwise be scared off.  I think lame and audioactive are too good, but the other mp3 codecs might work.

l3enc might work, but I would probably encode the samples with it to listen to how bad it is.  It might be too good as well.  In fact, it's probably a good idea to listen to the encodes of any prospective anchor.

Small listening pre-tests are a good idea, in my opinion.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: AtaqueEG on 2004-02-09 03:07:26
Quote
Well, I see there is too much discussion about what codecs to support and what anchor to use. I think the best and most fair way to settle the issues is creating a poll.
(...)
Do you agree?

Yes, absolutely.
I think this would be the best way to settle this issue.

But I think that everybody agrees on the 3 codecs you mention, so why don't you have a list with only the remaining codecs. Let every one vote and select the three with the most votes.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 03:10:11
Quote
Let every one vote and select the three with the most votes.

Two, actually
(The anchor would be handled in another poll)

I think that might work too, although that would limit votes to one per person, and the ideal would be two. Ideas, comments?
Title: AAC @ 128kbps listening test discussion
Post by: AtaqueEG on 2004-02-09 03:14:46
Hmm?
Two separate polls?
Monday through wednesday, choice of first contender.
Separate poll, thursday through saturday, remaining candidates minus first poll winner.

It's the only thing that comes to my mind right now.
Title: AAC @ 128kbps listening test discussion
Post by: ssjkakaroto on 2004-02-09 03:22:50
that's a good idea, having a separate poll for the anchor i think it's the best choice right now

after reading this thread i've been convinced that LAME would not be a good anchor, now my vote would go for iTunes since it had the lowest score in the last test
Title: AAC @ 128kbps listening test discussion
Post by: MGuti on 2004-02-09 03:37:07
go for the polls separated by time.

please just pick an anchor. it doesn't really matter as long as it sucks. it isn't in there to compare to the other codecs, its there to lose, and keep the other ratings up.
Title: AAC @ 128kbps listening test discussion
Post by: eagleray on 2004-02-09 03:44:13
My vote is to use the old FAAC as the anchor.  After all, this test is about AAC.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 03:50:59
Quote
go for the polls separated by time.

Yes, I will do that.

Now, choosing the codecs:
http://www.hydrogenaudio.org/show.php/showtopic/18523 (http://www.hydrogenaudio.org/show.php/showtopic/18523)

Some time next week, choosing the anchor.

Please vote.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: ExUser on 2004-02-09 05:18:47
What about the Homeboy AAC encoder as an anchor? Wasn't it really bad?
Title: AAC @ 128kbps listening test discussion
Post by: Ivan Dimkovic on 2004-02-09 08:17:49
Regarding Nero AAC - Fast mode shouldn't be used, as it is still not tuned.
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-09 08:36:04
Quote
Besides, what's wrong with using Lame in many tests? It allows (to an admittedly small amount) inter-test comparability (Another meaning of the word anchor?), while FhG l3enc 1.0 will only make the used rating range smaller.

I like the idea.
Using a stable high quality encoder as an anchor could indeed allow some inter-test comparison and IMO such kind of anchor would be much more valuable than any low quality old encoder. An ancient encoder is just a waste of listening slot.


Quote
Let's be realistic: If Lame is included, it won't be an anchor. It will just be another competitor. An MP3 competitor in an AAC test. You probably notice the mess here...

How can you know LAME won't lose? IMHO LAME at a last place is a realistic scenario.
I really think comparing the best MP3 to the worst AAC encoder would be fascinating and I can't see any mess here.
Title: AAC @ 128kbps listening test discussion
Post by: menno on 2004-02-09 08:47:53
EDIT: Winamp problem is a decoder-only problem. Nothing seems wrong with the encoder.

EDIT: Hmm, actually I can't be sure if this is an encoder problem too. I guess some testing should be done whether a Winamp file decoded with Winamp sounds better then with another decoder. The difference between 2 decoded samples (winamp decoder versus another) is quite difficult to hear, but in a test like this it will probably be noticed.

Besides this, I think NCTU-AAC should be added to the test, just to show how much it sucks. As low anchor also the new Adobe Audition filter could be used.

Menno
Title: AAC @ 128kbps listening test discussion
Post by: eltoder on 2004-02-09 09:19:51
If everyone knows that NCTU-AAC sucks badly may be it could be used as anchor? Just a thought.

-Eugene
Title: AAC @ 128kbps listening test discussion
Post by: plonk420 on 2004-02-09 12:33:30
Quote
Quote
How about VQF as anchor? According to AudioCodingWiki (http://www.audiocoding.com/wiki/index.php?page=VQF) , it is(was?) a part of MPEG-4 Audio version 1.

Actually, TwinVQ is part of MPEG4 audio. It's slightly different from VQF, and in MPEG4 audio it's limited to veeeery low bitrates - around 8-16kbps

That said, I was planning to use it as an anchor in the multiformat test that will come after the AAC test.

VQF ran at around 80-96kbps. i should know, i was the one that was so happy about it intially (with my 128kbps mp3 vs 96kbps vqf) i spammed the mp3 newsgroups and mp3.d newsgroup about it when i first started playing with it. (i wouldn't even touch it now, altho i think it's in my audio tools collection somewhere X)

can't find it or anything else that's helpful .. seems to have fallen off google 

oh, here we go.. here's one at least... http://www.mp3-tech.org/tests/pm/VQF-96k.htm (http://www.mp3-tech.org/tests/pm/VQF-96k.htm)
Title: AAC @ 128kbps listening test discussion
Post by: danchr on 2004-02-09 12:53:29
Quote
Besides this, I think NCTU-AAC should be added to the test, just to show how much it sucks. As low anchor also the new Adobe Audition filter could be used.

A question comes to mind: Is NCTU AAC bad enough to be used as low anchor?
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-09 13:00:52
Quote
But if, say, Lame finishes last in both the AAC test and the multiformat test (excluding anchors, the latter is to be expected), you will only provoke senseless claims like "Vorbis finished 1.2 points ahead of Lame in the multiformat test, but FAAC only managed an advantage of 0.9 in the AAC test, so Vorbis must be better than FAAC".

Well, such claim is better than pure guess when you can't compare faac to vorbis directly.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 13:57:28
@ Menno: Thanks for bringing this issue to our attention.

Discussion here:
http://www.hydrogenaudio.org/forums/index....ndpost&p=182562 (http://www.hydrogenaudio.org/forums/index.php?showtopic=18523&view=findpost&p=182562)

About anchors: I'm taking note of all your suggestions, and will later create a poll so that people can vote on what they prefer. Thank-you.
Title: AAC @ 128kbps listening test discussion
Post by: askoff on 2004-02-09 14:51:00
rjamorim: Are you going to use VBR streaming profile with Nero? I have done some encoding with it, and the average bitrate is too high for this "128kbps" test. VBR Internet profile produces closer to 128kbps on average.

I think that more encoders should be included or the anchor should be replaced with some AAC encoder. As i did the MP3 test, the anchor was useless to me, and this time it make out quite well against some other encoders.
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-09 15:46:51
Quote
I think that more encoders should be included or the anchor should be replaced with some AAC encoder. As i did the MP3 test, the anchor was useless to me, and this time it make out quite well against some other encoders.

Seconded. Same here.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-09 16:22:36
Quote
Quote
I think that more encoders should be included or the anchor should be replaced with some AAC encoder. As i did the MP3 test, the anchor was useless to me, and this time it make out quite well against some other encoders.

Seconded. Same here.

I personally would prefer that fewer codecs be tested rather than more.  If an anchor is used it should sound bad, IMO.  Otherwise, I think it's better to just drop it.  Especially on a test in which all the codecs should approach transparency.  6 codecs was almost too much for me in the mp3 test.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: Latexxx on 2004-02-09 16:25:03
Evil Psytel as anchor. 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 19:02:43
OK, now a fucker decided to mess my poll.

My patience is growing too thin, so I'll tell you what I plan to do:

-Compaact and Real are in
-Winamp is out because it seems to be broken, so it would be unfair to test it now.
-NCTU is in as an anchor.
-The test will be encrypted and will require Java ABC-HR.

Flames? Opinions? Comments?

BTW: Everything is discusseable, except encryption. It is definitely going to happen.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: Alexander Lerch on 2004-02-09 19:06:52
It may sound a bit unfair to newer members, but to remove any doubts about manipulating the result in favour of compaact!, I suggest to only take hydrogenaudio members into account, that were members *before* the release of compaact!, i.e. end of october 03.

If the participation at the listening test would be without restriction, and compaact! would rate good, nobody would believe it now anyway.

Alexander
Title: AAC @ 128kbps listening test discussion
Post by: menno on 2004-02-09 19:07:28
*deleted*
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-02-09 19:10:49
Quote
-The test will be encrypted and will require Java ABC-HR.

How strong this encryption is? What crypto system it uses?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 19:12:15
Quote
If the participation at the listening test would be without restriction, and compaact! would rate good, nobody would believe it now anyway.

Ph34r my m4d scr33ning skillz!
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-02-09 19:13:48
Quote
If the participation at the listening test would be without restriction, and compaact! would rate good, nobody would believe it now anyway.

It's possible to make a comparison between 'trusted' users and outside users. If the results disagree significantly, you'll know something is amiss.
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-02-09 19:14:21
Quote
-NCTU is in as an anchor.

I'm not sure NCTU is a very good anchor. It may be too good for anchor in samples which don't have strong attacks.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 19:16:56
Quote
How strong this encryption is? What crypto system it uses?

ElGamal and BlowFish, in a Public Key environment (the test admin keeps the private key, as proper)

I believe you need to be a very good hacker to be able to fool the system.

Edit: More info:
http://www.hydrogenaudio.org/forums/index....ndpost&p=135744 (http://www.hydrogenaudio.org/forums/index.php?showtopic=11691&view=findpost&p=135744)
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-02-09 19:17:14
Quote
Quote
If the participation at the listening test would be without restriction, and compaact! would rate good, nobody would believe it now anyway.

It's possible to make a comparison between 'trusted' users and outside users. If the results disagree significantly, you'll know something is amiss.

Good point. And with strong enough ecryption implementation, nobody won't even try any foul play..

Heh.. funny, but maybe this troll incident brought something good after all: It's definitely good that the encryption will be in use now!
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-09 19:19:38
Well, sucks that the poll didn't work out.

Quote
-Compaact and Real are in
-Winamp is out because it seems to be broken, so it would be unfair to test it now.

OK by me. I would have liked to see Winamp in the test (instead of Real), but if it's broken, well, can't change it.

Quote
-NCTU is in as an anchor.

I wouldn't mind that, if it's really clear that it is considerably inferior to all the other codecs. Otherwise, I'd prefer keeping the field of real contenders down to 5.

Quote
-The test will be encrypted and will require Java ABC-HR.

When were you planning on starting the test, again? I'm sure you mentioned it earlier, but I can't find it now. If you're not planning on starting it this weekend, I'll rework some of the encryption code and keep that source closed, so cheating will get even harder (I said it before, it's impossible to completely eliminate the possibility to cheat, simply because you need to give the user the right to encode his own results).
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 19:26:40
Quote
OK by me. I would have liked to see Winamp in the test (instead of Real), but if it's broken, well, can't change it.

Actually, I think it can be changed. A friend of mine contacted a developer inside nullsoft that will try to fix that. He has been told the deadline is tuesday afternoon next week :B

Quote
I wouldn't mind that, if it's really clear that it is considerably inferior to all the other codecs. Otherwise, I'd prefer keeping the field of real contenders down to 5.


I think I'll create a poll for anchor neverthless.

Quote
When were you planning on starting the test, again? I'm sure you mentioned it earlier, but I can't find it now. If you're not planning on starting it this weekend, I'll rework some of the encryption code and keep that source closed, so cheating will get even harder (I said it before, it's impossible to completely eliminate the possibility to cheat, simply because you need to give the user the right to encode his own results).


Perfect with me. The test start the next Wednesday (the 18th), so I need it ready by Wednesday morning.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-09 19:30:48
Quote
-Winamp is out because it seems to be broken, so it would be unfair to test it now.

Flames? Opinions? Comments?

we could wait (like for vorbis 1.0.1  )
Title: AAC @ 128kbps listening test discussion
Post by: guest0101 on 2004-02-09 19:35:57
Would like to see WinAmp in the test (if it is fixed by then). Perhaps that could be the "anchor"?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-09 19:36:48
Quote
we could wait (like for vorbis 1.0.1  )

Vorbis was a f***ing scam that promised the release for "next week", but took months instead.

Nullsoft isn't promising anything, so it's even more risky. So, no. If they don't deliver on time, they're out.
Title: AAC @ 128kbps listening test discussion
Post by: menno on 2004-02-09 20:48:37
I did some tests with FAAD2 on Winamp AAC encoder generated files. And I can exactly reproduce the Winamp decoder output in FAAD2, by limiting the TNS filtering to 5 kHz. So this is surely a decoder only problem, the encoder is safe to use.

Menno
Title: AAC @ 128kbps listening test discussion
Post by: MGuti on 2004-02-09 21:34:49
stupid troll....and please don't exclude members by time on the board. i've only been on here since early january but i would hate to be excluded.

that haveing been said, i'm glad to see compaact is in, its a shame that winamp is out (in out in out where is it now?), and real is iffy IMO. because they have a music store, i feel that it should be tested (a reason to buy from real instead of apple) but unless the container issue is fixed i won't end up using them.

NCTU might not be a good idea for anchor. it might be another Xing. it would make me more comfortable to have a piss-poor encoder as anchor instead of a possible contender. if the god's have chosen NCTU, so be it.
Title: AAC @ 128kbps listening test discussion
Post by: askoff on 2004-02-09 22:34:42
I'm fine with the selected codeks. I was just wondering how is this encryption efekting us testers? I mean how is the test canged comparing to mp3 test?
Title: AAC @ 128kbps listening test discussion
Post by: ancl on 2004-02-09 22:51:06
Quote
I'm fine with the selected codeks. I was just wondering how is this encryption efekting us testers? I mean how is the test canged comparing to mp3 test?

The only difference for the testers is that you have to use another ABX program. (And that you can't modify your result after testing
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-10 00:45:17
Quote
The only difference for the testers is that you have to use another ABX program. (And that you can't modify your result after testing

Right. When you load the .ecf (Encrypted Config File) in Java ABC/HR, the files are randomized and loaded in proper places like it would happen if you loaded a .txt file in Win32 ABC/HR

Then, you save the results in an encrypted results file (I don't remember the extension right now), that you attach to a mail (preferably all results zipped into one) and send to me. At home I have the decrypting key.

Of course, once the test is finished, I'll make the decrypting key available so that you can check out your own results.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-10 04:10:42
I will personally paint the town red with the corpse of the first guy messing with my new poll (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=3&t=18560).
Title: AAC @ 128kbps listening test discussion
Post by: kwanbis on 2004-02-10 11:54:41
Quote
I will personally paint the town red with the corpse of the first guy messing with my new poll (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=3&t=18560).

you are being to violent latelly
Title: AAC @ 128kbps listening test discussion
Post by: askoff on 2004-02-10 14:35:02
Quote
Right. When you load the .ecf (Encrypted Config File) in Java ABC/HR, the files are randomized and loaded in proper places like it would happen if you loaded a .txt file in Win32 ABC/HR

So if I open the config file twice, samples are in different scorebar?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-10 14:39:04
Quote
So if I open the config file twice, samples are in different scorebar?

Yes. And the same happens with Win32 ABC/HR
Title: AAC @ 128kbps listening test discussion
Post by: duartix on 2004-02-11 19:13:20
Quote
Of course, once the test is finished, I'll make the decrypting key available so that you can check out your own results.

Kudos Roberto for providing a way to see our own tests. (For me, my own test is as important as all other users tests together). Last test I took, all formats were transparent for me, except of course for the anchor and the format I was using for archives, so even though the encryption is unfortunately needed, so is the ability to check our own limitations.
Thanks! 

Now I guess there is only one question to be answered. LC or HE?
I guess this is like going to a XVid forum and ask people wether they use motion precision 5 or 6.
If we can get higher quality from HE why not use it? Because our portable won't suport it? Most players don't even support AAC, period.
New players will surely do, but what will you do then? Re-archive your 300(+) CD collection with HE? The only time I archived, it involved bringing around 30 CDs to my job everyday, and it took me over 2 weeks.
That is probably why they are still in OGG with just the most recent ones being backed up to Musepack.

I vote for HE, but I guess you can roll your eyes out 720º, and start another poll.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-11 19:16:46
Quote
Now I guess there is only one question to be answered. LC or HE?

HE is for low bitrates only (80kbps and lower). This test will be at 128kbps
Title: AAC @ 128kbps listening test discussion
Post by: duartix on 2004-02-11 19:33:53
Thanks for clearing that, I had no idea! 
I thought is was something like a higher quality mode...
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-12 15:41:01
2 Things:

1. From the 64kbps test and the 128kbps mp3 test I got the impression that my "experiencia" sample is too easy too encode / doesn't cause enough problems. I'd like to provide a similiar sample (same genre, female voice) that's harder to encode. In case you don't agree, please tell me - than I can stop spending time on this.

2. It has been said before here and in another thread - and I noticed it with the files I encoded for testing: Latest Nero encoder seems to give too high bitrate at medium setting. (135... 145-155...165 kbps here). Is your decision about use of medium fixed already?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-12 15:44:58
Quote
1. From the 64kbps test and the 128kbps mp3 test I got the impression that my "experiencia" sample is too easy too encode / doesn't cause enough problems. I'd like to provide a similiar sample (same genre, female voice) that's harder to encode. In case you don't agree, please tell me - than I can stop spending time on this.

Oh, yes, please. It would be good to replace part of the samples, before people start getting tired of them.

Quote
2. It has been said before here and in another thread - and I noticed it with the files I encoded for testing: Latest Nero encoder seems to give too high bitrate at medium setting. (135... 145-155...165 kbps here). Is your decision about use of medium fixed already?


No, I didn't do the bitrate deviation analysis yet. I'm planning to do that this weeked. I'll probably settle for the bitrate that comes closer to 128.
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-12 20:21:58
OT discussion about results of multiformat tests split here (http://www.hydrogenaudio.org/forums/index.php?showtopic=18656&). No more related discussion in this thread, please. Thank you.
____________________________________________

I've uploaded a sample to replace "experiencia" sample here (http://www.hydrogenaudio.org/forums/index.php?showtopic=18632&st=0&#entry184033) .

Edit: I've tested it with 3 of the AAC codecs used in the test. This sample is comparable to the better ones in 128kbps lame test, I'd say. So I recommend to replace 'experiencia' by it.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 06:07:49
Thank-you tigre.

This is what I planned for sample replacements:

polonaise -> fossiles (http://ff123.net/samples/fossiles.flac). Because everybody considered polonaise too damn hard in the MP3 test, let alone in this test.

Illinois -> fatboy (http://lame.sourceforge.net/download/samples/fatboy.wav). Illinois it a cool song, but it's also quite easy, and makes you sick of it too soon. Also, I want to see codecs bleed, that's why I want to replace it with fatboy  (although fatboy will probably lead to bitrate issues)

experiencia -> Quizás (read above posts) - again, a too easy sample.

EnolaGay -> OrdinaryWorld (http://pessoal.onda.com.br/rjamorim/OrdinaryWorld.flac) - by Duran Duran (a New Wave replacement) because in the MP3 test codecs got nearly all tied, I.E, the sample isn't good to diferentiate codecs, it seems (http://www.hydrogenaudio.org/forums/index.php?showtopic=18560&view=findpost&p=183539). I hope the guitar play in the intro of Ordinary World will bring problems to some codecs.

@ErikS: I will look at the samples Guruboolez sent me to replace riteofspring. I'll upload it and post here as soon as I find something suitable.

Do I suck? Do I rule? Please post.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: ViPER1313 on 2004-02-16 07:01:01
WOOT! The return of fatboy  - I don't want to hear anyone complaining about how all the codecs sound transparent on that one  .
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-16 08:49:40
Quote
Quote
i also dont want that real with its .ra sh*t gets popular

RM is just another container format, playable in many players these days, but I have already mentioned that AAC wrapped in RM is just a temporary solution. Our alternative file writers were not ready in time for Beta. RealPlayer 10 Gold and RealProducer (Helix Producer) 10 Gold will write AAC to another format...

The Real (HE-)AAC encoder was licensed from Coding Technologies, is the only HE capable encoder in addition to Nero (CT did invent aacPlus/HE by the way), and their AAC base encoder probably started out from a similar codebase as the FhG AAC encoder. Personally I would be very curious how it compares to the rest of the AAC encoders out there. "Installing" Helix Producer is as simple as unzipping the files, and if included in the test, I will make sure Roberto has no trouble whatsoever with the encoding or installation (don't want to repeat trying to install RealPlayer.. ), as well as losslessly convert the .ra files to whichever format is preferred, .aac, or .m4a.

As a sidenote, I think it is a good idea to be very clear about which is the actual codec provider, for example for WinAmp, who did they license the encoder from? For Real, like I mentioned, this would be Coding Technologies.

I would *really* like to see Coding Technologies' AAC encoder in the listening test.

Either in the form of Real's implementation, or using CT's actual stand-alone encoder.

(I believe you can use Real Splitter+3ivx Muxer to convert from RM to MP4)
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-16 09:01:24
Quote
  • Nero
  • iTunes
  • Real
  • FAAC
  • Compaact
  • Anchor

I agree
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-16 09:24:28
Quote
OK, now a fucker decided to mess my poll.

My patience is growing too thin, so I'll tell you what I plan to do:

-Compaact and Real are in
-Winamp is out because it seems to be broken, so it would be unfair to test it now.
-NCTU is in as an anchor.
-The test will be encrypted and will require Java ABC-HR.

Flames? Opinions? Comments?

BTW: Everything is discusseable, except encryption. It is definitely going to happen.

Regards;

Roberto.

Looking forward to the test... sorry for not reading through all the drama before initially replying
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-16 09:51:41
Re: Anchors,

I'm actually partial to a very simple lowpass or something, ie something to just make it at least *easy* to place the last contender...

Oh, that horid crackling sound... that's the bad bad one
Title: AAC @ 128kbps listening test discussion
Post by: Atlantis on 2004-02-16 11:48:04
Hi Roberto,
I'd like to see featured in this test a live recording sample like this one (http://www.hydrogenaudio.org/forums/index.php?showtopic=18632&view=findpost&p=183818)
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-16 13:03:29
what about including a speech sample with low volume music in the background (like a movie background)

this would satisfy the video nerds and would maybe make the comparsion between the codecs easier?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 15:10:04
Quote
what about including a speech sample with low volume music in the background (like a movie background)

this would satisfy the video nerds and would maybe make the comparsion between the codecs easier?

No problem. But please provide the sample and suggest what sample should be replaced.

@Atlantis: Please suggest what should be replaced too.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-16 17:47:54
Quote
Quote
what about including a speech sample with low volume music in the background (like a movie background)

this would satisfy the video nerds and would maybe make the comparsion between the codecs easier?

No problem. But please provide the sample and suggest what sample should be replaced.

hm best would be a sample from a dts source (and maybe with english language  )

i can only provide a german ac3 source, if thats also ok?

what to replace? hm dunno, maybe waiting cause it was already used in the first aac test too? 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 17:53:11
Quote
i can only provide a german ac3 source, if thats also ok?

I would personally prefer LPCM...

Quote
what to replace? hm dunno, maybe waiting cause it was already used in the first aac test too? 


No, Waiting is perfect for killing samples tenderly. It must stay.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-16 18:48:09
Quote
Quote
i can only provide a german ac3 source, if thats also ok?

I would personally prefer LPCM...

hm lpcm is used for dvd movie sound? i thought only ac3 and dts?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 18:59:20
Quote
hm lpcm is used for dvd movie sound?

I don't know. But I would prefer LPCM neverthless

Of course, if it's not available, DTS would be the way to go. AC3 isn't a good idea considering it's already quite compressed.
Title: AAC @ 128kbps listening test discussion
Post by: Continuum on 2004-02-16 19:59:55
Quote
hm lpcm is used for dvd movie sound? i thought only ac3 and dts?

It is for Music DVDs. Maybe some opera recitative? 
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-16 20:18:32
Quote
Maybe some opera recitative? 



but serious, anyone having a nice dts source from a movie dvd?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 20:30:03
Quote
but serious, anyone having a nice dts source from a movie dvd?

I only have from an Eagles concert. My only DVD with DTS
Title: AAC @ 128kbps listening test discussion
Post by: Atlantis on 2004-02-16 20:32:48
Quote
@Atlantis: Please suggest what should be replaced too.

Uhm, I'm unsure about this...

Anyway, I think BigYellow could be replaced, in the 128 MP3 test all codecd were tied (except itunes implementation).

All is IMO, obviously 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 21:57:20
Hrm... your sample is interesting, but from hearing to it, it seems to be potentially very easy on codecs :/
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 22:10:09
OK, some more replacement suggestions:

- Polonaise will no longer be replaced with Fossiles, but with Guruboolez' Brahms - Danse Hongroise 6 (http://pessoal.onda.com.br/rjamorim/Hongroise.flac). So it gets replaced by another piano sample.

- riteofspring will be replaced with Mahler - Symphonie 3 (http://pessoal.onda.com.br/rjamorim/Mahler.flac), also submitted by Guruboolez.

Any comments? Tonight I'll freeze the samples list and start encoding/packing/uploading.

Thanks to Guruboolez for providing those samples.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-16 22:11:32
Quote
Hrm... your sample is interesting, but from hearing to it, it seems to be potentially very easy on codecs :/

I've tested this sample with 2 of the codecs that will be in the test at (- if VBR - settings aiming at) 128kbps.
There were more then one for me easily abx'able differences in both cases but nothing really obvious/noticable to me without comparing to the original. So ABC/HR ratings would have been between ~4 and ~4.5. IMO there are some sounds in this sample that might cause more problems for other contenders (I won't talk about details publicly before the test) I'm not sure if this sample will help to get significant differences between codecs.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 22:14:44
Quote
I'm not sure if this sample will help to get significant differences between codecs.

Good point. Can anyone else suggest live recordings?

Edit: So here is the samples list as of now:

BigYellow
DaFunk
fatboy
gone
Hongroise
Mahler
mybloodrusts
NewYorkCity
OrdinaryWorld
Quizas
Scars
Waiting
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-16 22:30:07
Quote
Can anyone else suggest live recordings?

I can have a look if I find something quickly...

BTW: I would test a bit before suggesting something - so can you please tell what settings will be used with comaact and nero?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-16 22:48:02
Quote
BTW: I would test a bit before suggesting something - so can you please tell what settings will be used with comaact and nero?

Nero: -internet
Keep in mind the test will use a DLL that officially will only be released with next Nero version:
http://pessoal.onda.com.br/rjamorim/aacenc_v2620.zip (http://pessoal.onda.com.br/rjamorim/aacenc_v2620.zip)

Compaact: VBR 5
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-17 00:44:10
I had no luck finding a hard to encode live sample. What about using Layla (http://www.ff123.net/samples/Layla.flac) sample, or drop using a live sample and use this (http://www.hydrogenaudio.org/forums/index.php?showtopic=18397&) instead?
Title: AAC @ 128kbps listening test discussion
Post by: KpeX on 2004-02-17 02:32:25
Quote
Quote
hm lpcm is used for dvd movie sound?

I don't know. But I would prefer LPCM neverthless

Of course, if it's not available, DTS would be the way to go. AC3 isn't a good idea considering it's already quite compressed.

LPCM is definitely within DVD specs, but the only DVDs I know of that use it are music DVDs - defeating the purpose of finding a movie soundtrack sample.

I would be just as wary to use a DTS source as an AC3 source.  IMO the compression quality isn't that much different from AC3 despite the fact that DTS uses ridiculously high bitrates.
Title: AAC @ 128kbps listening test discussion
Post by: askoff on 2004-02-17 04:31:00
How about true killer sample, what Guruboolez uploaded once IIRC. biniou.flac (http://personal.inet.fi/koti/askoff/biniou.flac)
Title: AAC @ 128kbps listening test discussion
Post by: plonk420 on 2004-02-17 05:12:07
Quote
Quote
hm lpcm is used for dvd movie sound?

I don't know. But I would prefer LPCM neverthless

Of course, if it's not available, DTS would be the way to go. AC3 isn't a good idea considering it's already quite compressed.

obviously, AC3 is lossy, but DTS is as well. however, the problems in theatrical AC3 is not present in DVD AC-3 because higher bitrates can and usually are used, as far as the stuff i've encountered (i wish i could borrow/rent Kings of Comedy to triple check that). well, the Garbage music video, TWINE @ 96kbps being an sadly grieved exception to that. if high enough bitrates are used in AC3, it should at least be "on par" with DTS (which i thought i read was not the most efficient codec).

i guess what i'm saying is, is that AC3 can be just as good of a contender as DTS, since they're both lossy.


oh, and i wouldn't suggest a straight music video DVD for LPCM, unless they mastered the sound specifically for the DVD. they seem to use 2nd generation audio from an analog-seeming transfer or something as most don't even compare to the CD mix.
Title: AAC @ 128kbps listening test discussion
Post by: plonk420 on 2004-02-17 05:17:21
Quote
How about true killer sample, what Guruboolez uploaded once IIRC. biniou.flac (http://personal.inet.fi/koti/askoff/biniou.flac)

that sounds like a good sample, at least based on my experience that killer samples seem to have synthy or distortion related attributes to choke the codec.
Title: AAC @ 128kbps listening test discussion
Post by: dev0 on 2004-02-17 05:21:48
I really like my caffeinebattlecry sample and it would probably cause problems with some of the encoders (haven't tested yet):
caffeinebattlecry.sample20sec.flac (http://www.rc55.com/friends/tobias/files/audio/samples/caffeinebattlecry.sample20sec.flac)
caffeinebattlecry.sample5sec.flac (http://www.rc55.com/friends/tobias/files/audio/samples/caffeinebattlecry.sample5sec.flac)

It did really well as a sample in my private (F)AAC tests, but I guess people could complain about the test becoming too muck rock and not enough classical stuff. I will look for a good live sample later today.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 06:54:49
Quote
Quote
How about true killer sample, what Guruboolez uploaded once IIRC. biniou.flac (http://personal.inet.fi/koti/askoff/biniou.flac)

that sounds like a good sample, at least based on my experience that killer samples seem to have synthy or distortion related attributes to choke the codec.

Hrm... I can already hear people complaining that sample gives them headaches

Also, I don't think it represents any specific style well. I can't even identify what instrument is playing that!

Quote
I really like my caffeinebattlecry sample and it would probably cause problems with some of the encoders (haven't tested yet)


Do you think that sample would be good to replace the other sample you submited? (mybloodrusts). I think it's kinda too agressive and "messy" (for lack of a better word), I already got comments from several people complaining they don't know where to focus to try to detect artifacts on such kinds of samples (gone's final part is very difficult in that aspect too)
Title: AAC @ 128kbps listening test discussion
Post by: dev0 on 2004-02-17 08:36:12
Identifying artifacts in mybloodrusts is really easy in my oppinion (and according to the ratings from the prior tests most people seem to think similiar), but I wouldn't have a problem with replacing it with caffeinebattlecry, which might represent punkrock/emocore/hardcore music more accurately, since it's less noisy and features more "classical" artifact spots (hard attacks etc.).

I'd really like to have them both featured, but I see that you can't do that. There are some interesting spots on albums I bought recently, which I haven't looked into any further yet, but I promise there'll be a shitload of new heavy samples for the next test set ready.
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-02-17 10:37:02
http://sjeng.org/ftp/work/Peccatum.flac (http://sjeng.org/ftp/work/Peccatum.flac)

Old sample from Dibrom. IMHO very useful for codec testing. (Posted in other thread)
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-17 12:07:09
In case it's not too late yet:

I've finally found a good/hard to encode live sample. It's from
James Brown - Greatest Hits - Body Heat (live)

Uploaded here (http://www.hydrogenaudio.org/show.php/showtopic/18632).
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-17 12:17:32
I don't think that biniou.wav is a killer sample. Nero AAC have problems with it in the past - but other encoders sound find, close to transparency.
Brass instruments in Mahler3 are more terrific, for all encoder at this bitrate
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-17 13:33:21
still interest in a movie sample (speech will low volume background music)?

if yes, how should i treat the multichannel dts source? it should be possible to downsample with azid to stereo...
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 16:05:41
Quote
http://sjeng.org/ftp/work/Peccatum.flac (http://sjeng.org/ftp/work/Peccatum.flac)

Old sample from Dibrom. IMHO very useful for codec testing. (Posted in other thread)

Indeed, very interesting. Do you think it can be used to replace gone maybe?

@tigre: Thank-you very much. I'll replace Scars with it, since that seems to be another sample listeners don't like much. Also, I didn't have any funk/soul sample

@bond: I probably can accept, but you will need to hurry. I'll start encoding the samples very soon. Also, tell what sample should be replaced :B
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-17 16:31:32
Quote
@bond: I probably can accept, but you will need to hurry. I'll start encoding the samples very soon. Also, tell what sample should be replaced :B

thanks to gam3r and fr4nz i now have 2 sources:

1) speech only
2) speech + some pumping sound (from drums?) in the background

which of the two is prefered?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 16:34:50
Quote
1) speech only
2) speech + some pumping sound in the background

Speech can usually be compressed down to 16kbps while keeping a quite good quality (look at Speex, ACELP, etc.), so I think the version with pumping will be better. I'll test it to see if it is too obviously easy to encode, and I'm afraid it might be.

Downmixing with DPL would do, no problem.

And resampling to 44.1kHz would also be very important

Also, again: What sample will be replaced?
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-17 17:16:24
ok here it is: http://8ung.at/bond/NinjaScroll.flac (http://8ung.at/bond/NinjaScroll.flac)

i created it that way:
1) decoding the source .dts (from the anime Ninja Scroll, thanks gam3r!  ) with the Intervideo Audio dshow decoder (which is the only stable DTS decoder i know) to .wav (default settings: 2 speaker mode - dolby surround compatible, vocal options: both)
2) cutting the .wav in cooledit
3) saving as .fla with case's flac plugin (default setting: compression level 5)
4) rename to .flac (i assume thats correct)

i hope its interesting

Quote
Also, again: What sample will be replaced?

hm also again, no clue as i dont really know the samples which will be used atm
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 17:20:26
Quote
ok here it is: http://8ung.at/bond/NinjaScroll.flac (http://8ung.at/bond/NinjaScroll.flac)

Server down?

I can't get it from here nor from the 3 SSH servers I have access to.

Can you please put it at the upload thread?
http://www.hydrogenaudio.org/show.php/showtopic/18632 (http://www.hydrogenaudio.org/show.php/showtopic/18632)

Quote
hm also again, no clue as i dont really know the samples which will be used atm


http://www.hydrogenaudio.org/forums/index....ndpost&p=185286 (http://www.hydrogenaudio.org/forums/index.php?showtopic=18474&view=findpost&p=185286)
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 17:35:37
Downloaded, k-tnx

Some problems I find:

-The volume is too low
-I couldn't find any part of it that would stress codecs. My guess it that most of them would reach transparency there.

Opinions from others?
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-17 17:57:27
Quote
-The volume is too low

well the volume is always that low on movie dvds (normally someone would gain the whole thing, but i avoided that of course to not change the source)

Quote
-I couldn't find any part of it that would stress codecs. My guess it that most of them would reach transparency there.

hm if you think so 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 18:39:57
Last minute change: Fatboy will be replaced with Roel's velvet.wav (http://lame.sourceforge.net/download/samples/velvet.wav). Compaact behaved too badly on Fatboy (270kbps!). And velvet is pretty evil neverthless.

So, the final frozen list will be:

BigYellow
bodyheat
DaFunk
gone
Hongroise
Mahler
mybloodrusts
NewYorkCity
OrdinaryWorld
Quizas
velvet
Waiting

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: music_man_mpc on 2004-02-17 18:51:47
Quote
Last minute change: Fatboy will be replaced with Roel's velvet.wav (http://lame.sourceforge.net/download/samples/velvet.wav).

Ugh!  I can't do ABXing with velvet, one constant beat with melody tends to get to me pretty fast, one constant beat on its own . . . . my ears just can't take that. 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 21:29:14
hello.

I'd like to ask a favour from someone that understands well the unix shell.

This is one of the .bat files I'll use for this listening test:

Code: [Select]
@echo off

rem Reference decoding
flac -d -o ..\Sample01\BigYellow.wav ..\Sample01\BigYellow.flac

rem Samples decoding
faad -o ..\Sample01\BigYellow_1.wav ..\Sample01\BigYellow_1.mp4
faad -o ..\Sample01\BigYellow_2.wav ..\Sample01\BigYellow_2.mp4
faad -o ..\Sample01\BigYellow_3.wav ..\Sample01\BigYellow_3.mp4
faad -o ..\Sample01\BigYellow_4.wav ..\Sample01\BigYellow_4.mp4
faad -o ..\Sample01\BigYellow_5.wav ..\Sample01\BigYellow_5.mp4

rem Cleanup
del ..\Sample01\BigYellow.flac
del ..\Sample01\BigYellow_1.mp4
del ..\Sample01\BigYellow_2.mp4
del ..\Sample01\BigYellow_3.mp4
del ..\Sample01\BigYellow_4.mp4
del ..\Sample01\BigYellow_5.mp4


Can someone please provile a .sh script that would perform the same?

I will expect the test participants to have FLAC an FAAD somewhere in the path. There's no way I can provide binaries for Linux, Solaris, MacOS, BSD... specially considering all the dependency issues.

The echo off isn't needed.

Thank-you for your help.

Regards;

Roberto.


Also, so that critics can start complaining right away that Nero bitrates are too high:
http://pessoal.onda.com.br/rjamorim/Bitrates.txt (http://pessoal.onda.com.br/rjamorim/Bitrates.txt)
Title: AAC @ 128kbps listening test discussion
Post by: rpop on 2004-02-17 21:35:29
Omg, they're too high! I understand that bitrates tend to rise for problem samples and for overall music they stay closer to their target (as I once argued), but I doubt this is the case here....I'll encode a few CDs to check.
Title: AAC @ 128kbps listening test discussion
Post by: Continuum on 2004-02-17 21:46:19
Code: [Select]
Nero:  
Hongroise    142
Mahler  142

They really made a special Guru-optimized build, after his classical test results. 
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-02-17 21:52:00
Quote
Quote
http://sjeng.org/ftp/work/Peccatum.flac (http://sjeng.org/ftp/work/Peccatum.flac)

Old sample from Dibrom. IMHO very useful for codec testing. (Posted in other thread)

Indeed, very interesting. Do you think it can be used to replace gone maybe?

I'd use it to replace whatever sample had the least discriminating power/highest scores and that's still in. Qua genre it has a little of a lot, if you know what I mean.
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-02-17 22:16:03
Quote
Omg, they're too high! I understand that bitrates tend to rise for problem samples and for overall music they stay closer to their target (as I once argued), but I doubt this is the case here....I'll encode a few CDs to check.

I tested 272 samples which are a mixture of completely different music styles. I have 132.5kbps average for Nero, which is actually lower than for Compaact1.2b2 VBR5 which was 1 kbps higher average.
It depends on the music style pretty much.
Title: AAC @ 128kbps listening test discussion
Post by: guest0101 on 2004-02-17 22:31:03
JohnV:

Compaact 1.2 beta 3 has been out for almost a week I believe. Alexander informed me of it. Perhaps he forgot to update his beta thread with that link. I think it has various optimizations, so you might want to get a copy instead of using beta 2 for testing.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 22:47:42
Quote
Compaact 1.2 beta 3 has been out for almost a week I believe. Alexander informed me of it. Perhpas he forgot to updatre his beta thread with that link. I think it has various optimizations, so you might want to get a copy instead of using beta 2 for testing.

Do you really think he didn't send me an e-mail already about beta 3?

Edit: Erm? That post is directed at JohnV?


So, anyone willing to take a look at my batch script to convert it to bash-friendly?
Title: AAC @ 128kbps listening test discussion
Post by: rpop on 2004-02-17 22:51:07
Quote
I tested 272 samples which are a mixture of completely different music styles. I have 132.5kbps average for Nero, which is actually lower than for Compaact1.2b2 VBR5 which was 1 kbps higher average.
It depends on the music style pretty much.

If Nero and Compaact were so close, maybe the fatboy sample where Compaact reached 270 kbps should be reconsidered? :B
Title: AAC @ 128kbps listening test discussion
Post by: Ivan Dimkovic on 2004-02-17 22:52:45
Hold a sec...

BigYellow   143   123   130   124   111


ok..> I encoded BigYellow...

Original:  4320784

MP4:  419946  - RATIO 10.288 - BIT RATE - 137.1 Kbit/s
Audio Track Only: 413488 - RATIO 10.4495 - BIT RATE - 135.02 Kbit/s

(Audio track only is the actual encoder bit rate measured in bit allocation, you could get it by extracting track)

So I think claim 143 + 5-6 kb/s is little bit out of bounds
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-17 23:01:32
Indeed, partially my fault. foobar randomized the input, and I forgot to sort it properly.

These are the bitrates foobar reports for the sample suite, encoded with the AAC plugin v. 2.6.2.0, high quality, -internet, LC:

(http://pessoal.onda.com.br/rjamorim/screen4.png)

The average is still 141kbps.

The final proof:

All 12 wav files summed: 52.895.248 bytes  divided by
All 12 aac files summed: 5.264.357 bytes  equals 10,0478

Final bitrate = 1411 (CD bitrate) / 10,0478 = 140,428

So there you have it.
Title: AAC @ 128kbps listening test discussion
Post by: guest0101 on 2004-02-17 23:22:34
Quote
Quote
Compaact 1.2 beta 3 has been out for almost a week I believe. Alexander informed me of it. Perhpas he forgot to updatre his beta thread with that link. I think it has various optimizations, so you might want to get a copy instead of using beta 2 for testing.

Do you really think he didn't send me an e-mail already about beta 3?

Edit: Erm? That post is directed at JohnV?


So, anyone willing to take a look at my batch script to convert it to bash-friendly?

rjamorim,

Yes that reply was directed at JohnV replying to his post mentioning beta 2 of Compaact 1.2... Sorry if you missed that.
Title: AAC @ 128kbps listening test discussion
Post by: plonk420 on 2004-02-18 09:21:02
Quote from: rjamorim,Feb 17 2004, 09:05 AM
Quote from: Garf,Feb 17 2004, 08:37 AM

@tigre: Thank-you very much. I'll replace Scars with it, since that seems to be another sample listeners don't like much. Also, I didn't have any funk/soul sample 

nooooo! i liked scars! and i liked it even more once i found out it was from Chrono Cross
Title: AAC @ 128kbps listening test discussion
Post by: spoon on 2004-02-18 09:31:52
Nero on Internet :: Medium (High quality) is fair, over 500 full audio tracks (2G encoded mp4) it came out 2.5% above 128Kbps.

Compaact comression is running right now, in about 7 hours it will be completed and I will post the results all together.
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-18 09:43:18
Quote
Quote
Compaact 1.2 beta 3 has been out for almost a week I believe. Alexander informed me of it. Perhpas he forgot to updatre his beta thread with that link. I think it has various optimizations, so you might want to get a copy instead of using beta 2 for testing.

Do you really think he didn't send me an e-mail already about beta 3?

Edit: Erm? That post is directed at JohnV?


So, anyone willing to take a look at my batch script to convert it to bash-friendly?

well, here's a quick translation, I haven't tested it

Simply copy this into a blank file with LF for newlines, and set it to be +x

Code: [Select]
#!/bin/sh

# !!!UNTESTED!!!

# Reference decoding
flac -d -o ../Sample01/BigYellow.wav ../Sample01/BigYellow.flac

# Samples decoding
faad -o ../Sample01/BigYellow_1.wav ../Sample01/BigYellow_1.mp4
faad -o ../Sample01/BigYellow_2.wav ../Sample01/BigYellow_2.mp4
faad -o ../Sample01/BigYellow_3.wav ../Sample01/BigYellow_3.mp4
faad -o ../Sample01/BigYellow_4.wav ../Sample01/BigYellow_4.mp4
faad -o ../Sample01/BigYellow_5.wav ../Sample01/BigYellow_5.mp4

# Cleanup
rm -f ../Sample01/BigYellow.flac
rm -f ../Sample01/BigYellow_1.mp4
rm -f ../Sample01/BigYellow_2.mp4
rm -f ../Sample01/BigYellow_3.mp4
rm -f ../Sample01/BigYellow_4.mp4
rm -f ../Sample01/BigYellow_5.mp4
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-18 12:48:45
What faac setting did you use? Default seems to give considerably less than 128 on average sample.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-18 13:15:25
Quote
What faac setting did you use? Default seems to give considerably less than 128 on average sample.

-q 115. The average bitrate ended at +-132kbps

I didn't use TNS, since it is disabled by default and not recommeneded at the Audiocoding Wiki page.
Title: AAC @ 128kbps listening test discussion
Post by: spoon on 2004-02-18 18:57:38
Mammouth encoding task is over, 3 days worth - lets just say central heating was not required

Here are the raw figures for final file sizes:

500 lossless Ape files converted to Mp3 @128Kbps came to 1,980,037,004 bytes


FAAC (1.23.5) Quality 125    2,074,364,607 bytes                 5% over
FAAC (1.23.5) Quality 100    1,881,780,853 bytes      5% under

Quality 115 should be spot on



Nero Internet::Medium  high quality    2,029,931,171 bytes            2.5% over



Compaact q5    1,922,327,404 bytes      3% under

Guessing q6 would be 15% over (would have liked to try q6 compression, but we ran out of time).

Best of luck Roberto!
Title: AAC @ 128kbps listening test discussion
Post by: rpop on 2004-02-18 19:01:36
These bitrates sound fair
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-18 20:20:48
Thank-you very much for your help, Spoon

The test will start any minute now.
Title: AAC @ 128kbps listening test discussion
Post by: knik on 2004-02-18 20:25:27
Quote
Quote
What faac setting did you use? Default seems to give considerably less than 128 on average sample.

-q 115. The average bitrate ended at +-132kbps

I didn't use TNS, since it is disabled by default and not recommeneded at the Audiocoding Wiki page.

Thanx, I didn't even know it's so high.
Edit: I mean I didnt know the requred quality setting is so high.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-18 21:00:13
Quote
Thanx, I didn't even know it's so high.
Edit: I mean I didnt know the requred quality setting is so high.

Well, the required bitrate is 128. That's it


Test is now open:
http://www.hydrogenaudio.org/forums/index....ST&f=22&t=18863 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=22&t=18863)

Please participate
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-19 07:32:50
I'm coming a bit late.
~250 discs are currently stored on my HDD in lossless format. Classical music only. I've loaded them with foobar2000, randomized the playlist, and converted them with VBR streaming (aacenc32 2620 HIGH). I've stoped after 40 hours of processing.
1495 files were converted. Musical length: 105 hours of music.

Code: [Select]
• 1495 files

• TOTAL DURATION (fb2k 0.77c) : 4d 9:24:44 <=> 105 hours 24 min 44 sec <=> 379484 seconds

• Total size (without tags) : 6,14 Go (6 602 644 072 octets) [Go=GB; octets=bytes] <=> 52 821 152 576 bits


• average bitrate = 52 821 152 576 / 379484 / 1024 = 135,929 kbps [136 kbps]
or
• average bitrate = 52 821 152 576 / 379484 / 1000 = 139,192 kbps [139 kbps]




< 100 kbps   =    7 tracks*         52min 38sec
100-109 kbps =    0 track
110-119 kbps =   35 tracks       2h 44min 37sec
120-129 kbps =  237 tracks      12h 44min 54sec
130-139 kbps =  643 tracks   1d 19h 22min 34sec
140-149 kbps =  487 tracks   1d 14h 52min 53sec
150-159 kbps =   79 tracks       6h 36min 40sec
> 160 kbps   =    7 tracks          14min 23sec



* On 7 tracks, 6 are mono (or close to be mono) and the last one is a Nirvana joke, with a short musical surprise occuring afer 8 minutes of complete silence.


=> Bitrate is superior to 128 kbps with classical music. But the deviation isn't too high.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-19 16:33:38
Quote
I've loaded them with foobar2000, randomized the playlist, and converted them with VBR streaming (aacenc32 2620 HIGH).

I'm using the Internet VBR profile in the test... :/
Title: AAC @ 128kbps listening test discussion
Post by: jido on 2004-02-19 22:32:57
Stux, you should guard these "rm -f" with a condition so that the users don't lose their samples. Something like:
Code: [Select]
[ -e ../Sample01/BigYellow.wav ] && rm -f ../Sample01/BigYellow.flac

Would be safer, Roberto will appreciate 
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-20 09:23:00
Quote
Stux, you should guard these "rm -f" with a condition so that the users don't lose their samples. Something like:
Code: [Select]
[ -e ../Sample01/BigYellow.wav ] && rm -f ../Sample01/BigYellow.flac

Would be safer, Roberto will appreciate 

Actually, I would just get rid of the rm's
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-20 09:23:51
Quote
Quote
I've loaded them with foobar2000, randomized the playlist, and converted them with VBR streaming (aacenc32 2620 HIGH).

I'm using the Internet VBR profile in the test... :/

oups... typo error.

I used VBR internet too (streaming would be clearly > 150 kbps).
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-20 16:03:53
Quote
oups... typo error.

I used VBR internet too (streaming would be clearly > 150 kbps).

Ah, OK.

Thank-you very much for testing bitrates.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-20 16:33:12
http://lists.mpegif.org/pipermail/mp4-tech...ary/003312.html (http://lists.mpegif.org/pipermail/mp4-tech/2004-February/003312.html)

Anyone wanting to conduce listening tests?
Title: AAC @ 128kbps listening test discussion
Post by: duartix on 2004-02-20 16:45:20
Sample 7 is next up. 
Most of the samples have been transparent to me.
Some very few, have been almost transparent. But since the 128 multiformat test, where everything but mp3 and ogg were transparent, I didn't expect to actually hear diferences.
Using onboard sound and walkman headphones at the job, but hey, it's the only way I can test... 
Title: AAC @ 128kbps listening test discussion
Post by: puntloos on 2004-02-25 00:14:06
Is this a bad time to mention my PuntAAC codec (which says it encodes at 128kbit but in fact runs losslessly while using 5000kbit) and my normalisation rule

Code: [Select]
S = F / A


where:

S = Final rating
F = Rating given by listening tests
A = Normalized bitrate quotient = b / B (where b = track bitrate and B = 128)

example
If a track with the chosen quality setting (aimed at 128kbit, obviously) turns out to sound 'transparent' (5 out of 5) to everyone, but has a bitrate of 700, then without my normalisation rule it would've won with the perfect score, while with my rule, the final score would be:

S = 5 / (700/128) = 0.9

which sounds fairer to me...

yes, yes, I know that the quality settings are chosen such that the SAMPLE SET averages near the intended bitrate, but:

- It still doesnt paint an absolutely fair picture to encoders that consistently end up slightly below the target bitrate
- It doesn't mean that nothing can be learned from the INDIVIDUAL SAMPLE statistics, especially if a sample is more representative of the type of sound you usually encode.

*ducks chairs thrown by absolutists*
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-25 01:22:20
Here I come to bitch again.

So far, I got samples from 6 (six) different listeners. And most of them didn't submit complete result sets.

Obviously, that's not enough to generate statistically valid results. Specially considering I didn't screen the results yet for ranked references.

So, I'd like to know, from the ones that tried to participate, the reason of such low response:

-Having problems with ABC/HR Java?
-The samples are too transparent?
-Getting tired of so many tests in a row?
-Another reason?

I don't plan to extend this test. If I don't get enough results by sunday night, I'll probably upload whatever results I got, and give up.

Regards;

Roberto.

PS: No troll feeding, please ^^
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-25 01:30:39
Here: I'm doing 1 or 2 samples a day. When I'm finished I'll submit the results.  Besides this, I should save my results more often - I've lost results when I've been almost finished with a test 2 times already, because my PC crashed (not related to ABC/HR)
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-25 02:46:39
For me, it takes much more time and effort to do this test compared to the mp3 test. I completed the mp3 test in three days, but with this one, I can't do more than two samples in a row before my concentration and motivation slip. Thus, like tigre, I usually stick to doing one or two a day.
I have completed 9 samples so far, and I'm optimistic that I'll be able to produce useful results with two of the remaining.
Title: AAC @ 128kbps listening test discussion
Post by: ErikS on 2004-02-25 05:53:05
Quote
So, I'd like to know, from the ones that tried to participate, the reason of such low response:

-Having problems with ABC/HR Java?
-The samples are too transparent?
-Getting tired of so many tests in a row?
-Another reason?

Simple reason. My computer died  I don't want to sit at a public computer taking the test... Looks a bit...
Title: AAC @ 128kbps listening test discussion
Post by: Bonzi on 2004-02-25 06:02:55
Quote
So, I'd like to know, from the ones that tried to participate, the reason of such low response:

-Having problems with ABC/HR Java?
-The samples are too transparent?
-Getting tired of so many tests in a row?
-Another reason?

I'm sorry I just cannot seem to ABX these with any kind of consistency.  I can do one sample at most before I lose all my concentration and the results become less than useful.  I'm going to try again on Saturday since I am really busy this week and maybe I can at least give you some results.  New headphones might help me too, since cheap earbuds get uncomfortable after a while.  Sorry again.
Title: AAC @ 128kbps listening test discussion
Post by: westgroveg on 2004-02-25 06:11:31
Quote
So, I'd like to know, from the ones that tried to participate, the reason of such low response

I had problems d/ling the Sun Java Runtime v1.4.
Title: AAC @ 128kbps listening test discussion
Post by: kl33per on 2004-02-25 06:59:46
I'm still completing the test.  Hopefully you'll have a flood of results right before the test closes.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-25 07:19:30
The test is pretty tough compared to the mp3 test.  And that one was already difficult.  Test fatigue is a factor for me.  Still, I've managed to mail in 5 results (one of them all 5's, though).

ff123
Title: AAC @ 128kbps listening test discussion
Post by: jido on 2004-02-25 10:15:29
Quote
Here I come to bitch again.

So far, I got samples from 6 (six) different listeners. And most of them didn't submit complete result sets.

Obviously, that's not enough to generate statistically valid results. Specially considering I didn't screen the results yet for ranked references.

So, I'd like to know, from the ones that tried to participate, the reason of such low response:

-Having problems with ABC/HR Java?
-The samples are too transparent?
-Getting tired of so many tests in a row?
-Another reason?

That would be my first listening test, and it seems very difficult. Maybe I need training but I am not able to consistently ABX the samples.

Also I didn't find the much time to work on it.
Title: AAC @ 128kbps listening test discussion
Post by: robUx4 on 2004-02-25 10:31:16
Quote
So far, I got samples from 6 (six) different listeners. And most of them didn't submit complete result sets.

Obviously, that's not enough to generate statistically valid results. Specially considering I didn't screen the results yet for ranked references.

Hi Roberto,

I've downloaded all the necessary stuff yesterday to proceed with the (complete) test. I will only have time to make it on saturday evening or sunday. I hope it won't be too late for you 

edit: BTW that's also the first time I make such a test too
Title: AAC @ 128kbps listening test discussion
Post by: askoff on 2004-02-25 11:03:04
Finaly it's over. This test was much harder than MP3 test. With couple of first samples I got realy frustrated, becaus I just couln't find a spot where to ABX. I almost gave up, until first ABXable sample was found.
I send couple of results where i mentioned that i couln't ABX it, but i guess that is one result also.
So the transparency was almost the issue for me.
Title: AAC @ 128kbps listening test discussion
Post by: smok3 on 2004-02-25 11:20:36
i vote for frustration as well, the problem is that for this test i have to abx almost all the samples to actually be sure that things are as they seems to be..., in any case my ratings are like all from 4-5, so i dont see much point on sending such results in.

java abchr is a great app thought.
Title: AAC @ 128kbps listening test discussion
Post by: Continuum on 2004-02-25 13:53:19
No real problem with Java, the save session feature is great.

The test is really difficult though. So far I have done samples 2, 5, 6 and 12. So far, I could find differences everywhere (I hope!), but this takes a lot of time. Two tests in a row is too much, it's more half test - wait - half test for me.
Title: AAC @ 128kbps listening test discussion
Post by: duartix on 2004-02-25 15:55:15
Quote
in any case my ratings are like all from 4-5, so i dont see much point on sending such results in

I believe it makes sense and that there are many reasons to still send your results in.
Even if the results say that all codecs are transparent. (My lowest rating so far is 4.7) I'm not worrying too much if I can't spot any diferences. It means the codecs are better than  my ears. 
But at least we can reach a conclusion. Without results we can't reach it!
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-25 16:58:29
Quote
Quote
in any case my ratings are like all from 4-5, so i dont see much point on sending such results in

I believe it makes sense and that there are many reasons to still send your results in.

That's right. It's important to know if all codecs are transparent for you. That means they did very well.


Thank-you very much for the explanations. I hope to get lots of results during the weekend

So that you know: the test ends Sunday midnight, Brazilian time.
http://www.timeanddate.com/worldclock/ (http://www.timeanddate.com/worldclock/)
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-02-25 18:56:53
Yes, the test is quite hard. I completed the previous one in 2-3 hours, but doing all samples for this one took considerably longer, and I had to ABX much more. That's good: it shows that AAC has been improving.

Still, with enough effort it should be quite possible to distinguish the codecs still. Practise makes perfect!
Title: AAC @ 128kbps listening test discussion
Post by: music_man_mpc on 2004-02-25 19:15:05
Quote
Still, with enough effort it should be quite possible to distinguish the codecs still. Practise makes perfect!

I agree, so far I have had to ABX one sample up to 67/102 before I was sure I heard a difference.  I am quite sure I did, though.
Title: AAC @ 128kbps listening test discussion
Post by: eagleray on 2004-02-25 20:04:15
@Roberto

I saw that you used q 115 for faac.

Did you use a "c" setting or was it the default?

Thanks.
Title: AAC @ 128kbps listening test discussion
Post by: smok3 on 2004-02-25 21:48:40
Quote
That's right. It's important to know if all codecs are transparent for you. That means they did very well.

or maybe one just got bored enough with a specific sample? 
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-26 02:46:29
So far I've only managed to ABX one of the samples

BUT I've been running a test a day or so and planning to send in the results when they're due

That and trying to rustle up a better pair of headphones
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-26 11:55:36
I'll send you my results in some hours.
I didn't find the test too difficult. It seems that some AAC encoders still have serious problems. I was more annoyed by some samples, which sounded distorted even with the reference.
I've noticed the same problems with ABC/HR for Java, especially with the piano sample. It didn't really disturbed me during this test (but for higher bitrate, which need a lot of concentration, this software is unfortunately not usable on my computer).

Anyway, thank you for conducting the test (and to schofler, for his software).
Title: AAC @ 128kbps listening test discussion
Post by: kl33per on 2004-02-26 12:48:46
Funny, I find this test much easier then the MP3 test.

Firstly, I'm not sure how many samples the MP3 test had, but having five to six samples is so much easier then having seven or eight, particularly when it comes to scoring them.

Secondly, I know that I personally have much more interest in AAC devlopment then I do in MP3 devlopment.  Thus I gave the MP3 test away without really giving it fair go, mainly because it didn't interest me.  However, considering AAC is what I currently encode music in (Nero AACEnc to be exact) the outcome of this test thouroughly interests me, and therefore I wish to do my part to make it better.  So while technically this test may be much harder, subconciously I find it much easier to do as I'm motivated to do it.

Edit: Of course this could all be rubbish if my test results show that I consistantly picked the wrong sample.
Title: AAC @ 128kbps listening test discussion
Post by: bidz on 2004-02-26 14:53:48
This test is difficult. Atleast on my equipment and with my ears. I guess 128kbps (aac) really is enough for me 
Title: AAC @ 128kbps listening test discussion
Post by: [proxima] on 2004-02-26 15:55:18
Test completed and results mailed.
I found the test not so diffcult, some AAC encoders have still annoying artifacts. For some codecs ringing is a bad issue, quite annoying for me. Nevertheless, apart some killer samples, a well tuned AAC encoder at 128 kbps is acceptable...
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-26 16:08:03
I agree. And one of them suffers from excessive lowpass to my ears (~15000 hertz, maybe less). Maybe not as bad as ringing or other distortions in daily playback, but on ABA/ABX comparisons, the consequences of the lowpass are really unpleasant and immediately betray the encoding.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-26 16:51:26
Quote
,Feb 26 2004, 12:55 PM] Nevertheless, apart some killer samples, a well tuned AAC encoder at 128 kbps is acceptable...

I agree. I guess several listeners will notice 128kbps is the transparency thresold for them...
Title: AAC @ 128kbps listening test discussion
Post by: Deimos on 2004-02-26 20:28:46
Quote
I was more annoyed by some samples, which sounded distorted even with the reference.

I'm interested to know which reference samples do you consider distorted 

Thanks in advance,

Deimos


I'm sending my results on Saturday.  A hard test for me, most ratings at 4.5-5 stage.
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-27 02:18:31
Can't remember their name or number in the list. I've reported my feelings in the ABC/HR comments; I just need the decrypted files to answer you.

P.S. The first one troubles me, and waiting.wav is heavily distorted (or at least very unatural voice) too. But there are some other files too.
It's probably personal troubles, linked to musical genre and mastering I'm not used to listen to (some people are troubled with classical: maybe too much fidelity )
Title: AAC @ 128kbps listening test discussion
Post by: kl33per on 2004-02-27 02:57:18
Quote
I agree. And one of them suffers from excessive lowpass to my ears (~15000 hertz, maybe less). Maybe not as bad as ringing or other distortions in daily playback, but on ABA/ABX comparisons, the consequences of the lowpass are really unpleasant and immediately betray the encoding.

I noticed this also (I think I mentioned it in the results), particularly on one of the samples.
Title: AAC @ 128kbps listening test discussion
Post by: Raptus on 2004-02-28 14:22:40
Results send for the 12 samples.

After some warming up I didn't have much trouble with the test. Most samples were easily picked because of obvious lowpass (my hearing goes up to 19khz) and ringing/smearing. Sample 5 was a bitch though, could't pick any until I found a spot were preecho occurred with one sample. Samples 1, 2, 6 and 10 were hard, too.

gear: Terratec DMX6Fire 24/96, oldschool Toshiba HR-80 Headphones 

Among the encoders are two very good ones and two quite bad.

Looking forward to the results. Will there be an extension?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-28 16:35:40
Quote
Looking forward to the results. Will there be an extension?

Yes. AAC winner vs. MP3 winner (Lame) vs. WMA Std., MPC, Vorbis and anchor.
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-28 16:42:36
Are they enough results for the moment? I really hope to see the final results in 48 hours, and not in 10 days...
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-28 16:56:16
Quote
Are they enough results for the moment?

Hard to say. I surely received enough results, but I'm getting a big amount of ranked references. Some of them can still be used (because the listener sucessfully ABXd the ranked reference), some must be dropped because no ABX was performed.

I'm still screening them to see what can be used and what can't. Hopefully there will be enough by tomorrow night.
Title: AAC @ 128kbps listening test discussion
Post by: elmar3rd on 2004-02-28 17:38:42
Quote
Quote
Are they enough results for the moment?

Hard to say. I surely received enough results, but I'm getting a big amount of ranked references. Some of them can still be used (because the listener sucessfully ABXd the ranked reference), some must be dropped because no ABX was performed.

I'm still screening them to see what can be used and what can't. Hopefully there will be enough by tomorrow night.

Can ranked references be considered as transparent?
A ranked reference shows, that the listener "fails" at this sample and other effects (psychological, fitness) are responsible for the ranking.
I isn't this in some way the same as transparent?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-28 17:41:11
Quote
Can ranked references be considered as transparent?

Yes, they can.

But I can't accept all ranked references, or I would be acepting files where the participant just randomly moved some sliders and sent me the results (like a very famous result set (http://pessoal.onda.com.br/rjamorim/Results.zip) I received in the 64kbps test)

That's why I'm going to use ABX results to decide what will be used and what will be dropped.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-28 18:10:57
i sent in my results now (i hope i dont have too much ranked references, its hard to avoid with encrypted results  )
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-02-28 18:41:52
I'm really impatient to see my results, and the possible mistakes I did 
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-28 21:24:36
Quote
That's why I'm going to use ABX results to decide what will be used and what will be dropped.

What happens in this case:

I successfully abxed (p < 0.01) but afterwards I'm exausted and can't hear the difference anymore reliably and rank the reference - Will this cout as 5.0?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 00:32:33
Quote
I successfully abxed (p < 0.01) but afterwards I'm exausted and can't hear the difference anymore reliably and rank the reference - Will this cout as 5.0?

Yes. I discussed the subject with Garf and ff123 (that are the two other Hydrogenaudio members with experience in public listening tests), and they agree that scoring 5 for ranked references successfully ABXd is OK.
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-02-29 06:07:39
results sent... phew
Title: AAC @ 128kbps listening test discussion
Post by: Dologan on 2004-02-29 06:21:18
Hmm... strikes me as a little unfair. If someone went through the hassle of ABXing all the way to p < 0.01, a mere slip or lapse of concentration may make all that effort worthless, since s/he might as well ranked it 5.0 right away with the first difficult sample and not ABXed at all. If I were in the situation of having to decide, I would 'switch' the rank of the reference to the sample, provided that the (misplaced) rank is above 4.0 (or maybe even a little higher).
An encrypted ABX result of p < 0.01 is a very strong evidence that the person IS detecting a difference and therefore, you are discarding valuable and valid information by assigning a 5.0 to an obvious mistake in the ranking. In the case of subtle, neutral artifacts (which would be expected to be ranked at >4.0), it would not be too difficult for a fatigued or distracted person to confound the encoding with the original, considering that none sounds annoying to him/her. Ranking a reference as already annoying, however, well...  ... we simply can't take the person's word for it and hence should be taken as 5.0, even in view of a successful ABX.
Title: AAC @ 128kbps listening test discussion
Post by: Continuum on 2004-02-29 07:08:44
The best solution IMHO would be, to reduce the two sliders to one after a significant ABX-result.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 08:18:34
well what if i rank the reference without abxing?

i mean i didnt do abx on all samples or till a result < 0.01 (yes i am lazy) when i thought i hear a difference right away (well who knows if i was right with these)

i mean i would think it would be ok to use these ranked references as 5.0 too (i think most people are reliable here to not send in randomly choosen results, like me    )
Title: AAC @ 128kbps listening test discussion
Post by: robUx4 on 2004-02-29 16:14:03
I'm conducting the test right now, but sometimes the applet crashes
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-29 16:39:59
Quote
I'm conducting the test right now, but sometimes the applet crashes

Well, if you want me to do something about it, you have to be a bit more precise in your error description... Either tell me how to reproduce the crash or start the application from a console and send me the error message. Thanks.
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-02-29 17:13:55
Quote
Quote
I'm conducting the test right now, but sometimes the applet crashes

Well, if you want me to do something about it, you have to be a bit more precise in your error description... Either tell me how to reproduce the crash or start the application from a console and send me the error message. Thanks.

Crashed here too occasionally. Since I wanted to finish the test without unnecessary repetitions I didn't try to reproduce the problem. I'll try to describe as good as I can though:

- Crash means that it becomes unresponsive, e.g. ABX window can't be closed anymore, when moving another window in foreground, things disappear, only possibility to close it down is Task manager (Win2ksp4 here).

- These crashs happened when I did too much with the mouse, e.g. changing playback range before playback is finished (normally playback just stops, but sometimes crashs happen) or when changing between A,B and X very often during short time (Fast switching disabled here).

I can try to reproduce the problem if you tell what exactly "start the application from a console" means - and what I have to do.


BTW: Results sent. Was a hard piece of work...
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-29 18:14:39
Quote
I can try to reproduce the problem if you tell what exactly "start the application from a console" means - and what I have to do.

By this I mean opening a command prompt and using "java -jar abchr.jar" (in the directory where you unzipped the files) to start the application, instead of just clicking on abchr.jar in Explorer. If there's a fatal crash, Java will print debug information which can be useful to find the error.
Won't be necessary in this case, however. I'm pretty sure what the cause of these effects is. Thanks for your description, I'll try to fix it in the next version.
Title: AAC @ 128kbps listening test discussion
Post by: AstralStorm on 2004-02-29 18:30:24
It may be the same crash, but well...

ABC-HR Java 0.4b3 SE crashes when I press stop at the end of the file (less than buffer length).
It bugs out most often with buffer length set to 2000ms.

Gentoo Linux post-2004.0, built with 2.6.3 headers and New Posix Thread Library.
Kernel 2.6.3-mm4

Terratec Aureon 7.1 on ALSA from the kernel, OSS emulation on (1.0.2c?)
ALSA-lib/utils - 1.0.2
No esound/arts/anything.

java version "1.4.1"
Java™ 2 Runtime Environment, Standard Edition (build Blackdown-1.4.1-01)
Java HotSpot™ Client VM (build Blackdown-1.4.1-01, mixed mode)

<edit>
More info: No loop, No fast switching

No text printed to std{out,err} on crash.
Title: AAC @ 128kbps listening test discussion
Post by: robUx4 on 2004-02-29 18:53:35
Quote
Quote
I can try to reproduce the problem if you tell what exactly "start the application from a console" means - and what I have to do.

By this I mean opening a command prompt and using "java -jar abchr.jar" (in the directory where you unzipped the files) to start the application, instead of just clicking on abchr.jar in Explorer. If there's a fatal crash, Java will print debug information which can be useful to find the error.
Won't be necessary in this case, however. I'm pretty sure what the cause of these effects is. Thanks for your description, I'll try to fix it in the next version.

I also got the same exact problem. I'd just like to add that I use the Loop a lot... Other than that, very nice peace of work
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 19:46:00
About considering ranked references as 5.0, as the ranked score, or drop them...

Here is the rationale Garf sent me:

Quote
Ranking a reference means that you underestimated the codecs performance on that sample.

Consider the following: you ABX it 110/200. That's a significant result, meaning you hear a difference. But you can't really tell the encoded one from the original, can you? Certainly reasonable to give 5.0 then.

Nice paradox


Here is the one Schnofler sent me:

Quote
Intuitively, I would either count the rating of the reference as the encoded sample's rating (the argument being that choosing sliders is just one more ABX trial) or discard the rating altogether (this being the conservative tactic, because the first method might be regarded sleazy). But as the listener obviously didn't intend to rate the sample as transparent and furthermore proved that it is in fact not transparent to him, your solution isn't immediately clear to me.


As you can see, two very good rationales, and both conflict.

What I will probably do is: at first consider only the "clean" results. If there are enough of these, the official test results will come from them. If not, I'll throw in the ranked references. I guess that's the best way to make everyone happy - otherwise, there would be no test results.

Now, the big question is, if I use the ranked references, should I use the ranked score, or grant a 5.0 score to them?

Please discuss.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: AstralStorm on 2004-02-29 20:03:47
My method:
If there's an ABX result with (number of trials)+1 on the margin (<0.1 but >0.05), treat the result as 5.0.
Else throw out the ranking.

Roberto, when will the test end exactly? Do you have enough results?
Title: AAC @ 128kbps listening test discussion
Post by: elmar3rd on 2004-02-29 20:06:54
Quote
Now, the big question is, if I use the ranked references, should I use the ranked score, or grant a 5.0 score to them?

Assuming the ranked references are spread equally over the samples, this will of course affect the ratings, but also equally.
So maybe it's unimportant, how to rate.
What are the exact values of the ratings good for? The ratings of different listening-tests can't be directly compared (with a very bad anchor you tend to rate the other samples higher).
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 20:24:32
Quote
Now, the big question is, if I use the ranked references, should I use the ranked score, or grant a 5.0 score to them?

Please discuss.

anyone knowing marx' dialectic?
here is my approach using this method to find a synthesis between garfs and schnoflers theses

imo in the case of a ranked reference there a two possibilities why the user voted this way:
he thought to hear a difference which
1) simply wasnt there
2) was there and he liked the enode quality better than the source (for whatever reason)

ad 1)
this could be caused by a mistake (as i understand schnofler's thesis)
but
i doubt that anyone does the final ranking as 1 more abxing without double checking that his final vote is correct (at least i wouldnt act this way)

so to say it can be divided into serious testers and not serious testers:
for people who serioulsy attend this test such "failures" shouldnt happen normally (also considering point 2 i wouldnt discard the results from these)
results of people who didnt seriously attend the test and voted in a hurry could be discarded (for example if there are too many (over the average) ranked sources in the results aso...)

ad 2) well thats how garfs thesis can be understood
in that way the voting would look that way:
source: 5
encode: a score higher than 5
as the later isnt possible, a vote of 5 is ok for the encode, when the source was voted worse


to sum it up/the synthesis:
ranked sources from not serious testers (which look like the person voted anything, too many (over the average) ranked sources aso...) can be discarded
all others should be used and used with score 5.0
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-02-29 20:28:10
You forgot the possibility:

a) User could hear a difference, ABX it, but it was rather small and he missed the right slider
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 20:35:09
Quote
You forgot the possibility:

a) User could hear a difference, ABX it, but it was rather small and he missed the right slider

thats point 1) voting by mistake
shouldnt happen (even without abxing), in fact abx helps to avoid these mistakes,
i mean if someone handles to really abx the sample with a high propability i doubt that on the final vote he will suddenly make a mistake
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-02-29 20:37:28
No, you are completely wrong, see my earlier example.

You can ABX 110/200, which is significant, but your chance of pulling the correct slider is only 55%.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-29 20:37:40
I think a kind of matrix can be constructed, with some of the options and reactions:

Options
1a.  Don't allow any ranked references
1b.  Allow only 1 ranked reference
1c.  Allow multiple ranked references
2a.  Ranked reference must be accompanied by ABX to 95% confidence
2b.  Ranked reference does not need to be accompanied by ABX results
3a.  Score of ranked reference is not lower than another properly-ranked codec
3b.  Score of ranked reference is allowed to be lower than another properly-ranked codec


Reactions
A.  Entire file is thrown out
B.  Score of codec with ranked reference is changed to 5.0
C.  Score of codec with ranked reference is given the listener rating 

Obviously, the most conservative approach is 1a + A

Here's how I might order the choices, from most conservative to less so:

Code: [Select]
Options           Reaction
1a                A

1b, 2a, 3a        B
1b, 2a, 3a        C

1b, 2a, 3b        B
1b, 2a, 3b        C

1c, 2a, 3a        B
1c, 2a, 3a        C

1b, 2b, 3a        B
1b, 2b, 3a        C


ff123
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 20:37:56
Quote
Roberto, when will the test end exactly? Do you have enough results?

I'll stop accepting results tonight, at midnight brazilian time.

I probably have enough results, byt maybe not enough if I dump the ranked references.
Title: AAC @ 128kbps listening test discussion
Post by: AstralStorm on 2004-02-29 20:38:26
Check my proposal, it should eliminate those 'tiny differences'.
If you can ABX it well and you make a mistake, it shouldn't be counted.
If you DIDN'T ABX it (or barely ABXed it), it should be treated as 5.0.

-
Midnight brazillian? So it is already closed?

-
What does A option mean? WHOLE results file?

I'd consider it if there are >2 ranked references.
I'd throw out all results w/o passed ABX in the file then and of course all ranked references.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 20:40:43
the results of abx'es should in no way influence the decision whether to use the ranked sources results or not!!!
noone is forced to do abx, you cant rely on whether someone did abx or not

in fact someone can do the whole test without abxing and without being unserious or having anything bad in mind


as i proposed unserious testers should be sorted out via the way if there are far over the average ranked sources in the results

Quote
No, you are completely wrong, see my earlier example.

You can ABX 110/200, which is significant, but your chance of pulling the correct slider is only 55%.

well your example is not usable in this case as its unrealistic/only theoretical
noone will do 200 abx'es
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 20:50:54
Quote
Midnight brazillian? So it is already closed?

Nope, there are still more than 6 hours to go:
http://www.timeanddate.com/worldclock/ (http://www.timeanddate.com/worldclock/)
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 20:54:37
Quote
as i proposed unserious testers should be sorted out via the way if there are far over the average ranked sources in the results

That makes no sense. Just because a listener is serious doesn't mean he'll come close to the average results.

Quote
well your example is not usable in this case as its unrealistic/only theoretical
noone will do 200 abx'es


Garf did almost that once, in the MAD challenge :B
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-29 20:55:23
Quote
the results of abx'es should in no way influence the decision whether to use the ranked sources results or not!!!
noone is forced to do abx, you cant rely on whether someone did abx or not

in fact someone can do the whole test without abxing and without being unserious or having anything bad in mind


as i proposed unserious testers should be sorted out via the way if there are far over the average ranked sources in the results

In my matrix, this is an option, so it is a matter of deciding (debating) how to order the list from most conservative to least conservative.  Another way of stating your proposal would be to come up with a numerical score which says how bad ranking the reference is with respect to the other scores.  For example, let's say somebody scores:

A = 4.9 ranked reference
B = 3.0
C = 2.0

A figure of merit score might be the ratio of the ranked reference to the average of the other scores.  First, transform into difference scores:

A = -0.1 ranked reference
B = -2.0
C = -3.0

then, F.O.M = A / average(B, C)

and if this ratio is under some acceptable value, you would either accept the results file as is, or change the rating of codec A to 5.0

This is a mess, isn't it?  Much easier just to discard files with ranked references.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 20:58:02
Quote
Quote
as i proposed unserious testers should be sorted out via the way if there are far over the average ranked sources in the results

That makes no sense. Just because a listener is serious doesn't mean he'll come close to the average results.

sure it makes sense, if you see it from that point that you dont want to sort out the serious ones, but the unserious ones (which will surely be far over the average)
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 20:59:56
Quote
This is a mess, isn't it?

Jesus Christ, it is!
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 21:03:16
Quote
This is a mess, isn't it?  Much easier just to discard files with ranked references.

first of all i assume that we dont have that much results, we can discard many, only because there are some ranked references

second your calculations show you are a developer (this isnt meant in a bad way  )
my proposal:
i would say the average user has 1 ranked reference per sample
if someone has an average (over all samples) of 2,5 he is out, all others are in with ranked sources scored as 5.0 (of course the reality can be different, but rjamorim will soon find this out)
easy and clean solution with no mess

third everyone who does this hard test and than gets discarded because he did ranked sources will feel pissed off
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 21:11:47
Quote
if someone has an average (over all samples) of 2,5 he is out, all others are in with ranked sources scored as 5.0 (of course the reality can be different, but rjamorim will soon find this out)


First, it was never in the plans to drop all of a listener's results because he ranked part of them. Unless something very creepy is going on (check the results package I linked earlier), even if a guy got 11 ranked references, the clean result will stay.

Also, what's with the 2,5?
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-29 21:17:40
Quote
What does A option mean? WHOLE results file?

I'd consider it if there are >2 ranked references.
I'd throw out all results w/o passed ABX in the file then and of course all ranked references.

A means throw out the whole file, which in the past was done if any reference was
ranked.  If you don't throw out the whole file, then you have the option of changing
the scores of the ranked references to 5.0 or keeping the score (but assigning it to the codec instead of to the reference, of course).

It isn't possible (at least with my statistics software) to only throw out part of a file.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 21:18:48
Quote
First, it was never in the plans to drop all of a listener's results because he ranked part of them. Unless something very creepy is going on (check the results package I linked earlier), even if a guy got 11 ranked references, the clean result will stay.

Also, what's with the 2,5?

2.5 ranked references per sample (or a similar value) can be used as indication as unserious testing, meaning the whole tester is out
all other ranked references are considered as from serious testers and will not be dropped and given a 5.0 score as garf proposed

thats my proposal, but it doesnt seem to have much friends anyways so do as you like 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-02-29 21:21:42
Quote
2.5 ranked references per sample (or a similar value)

Another problem introduced by this is: where to draw the line?
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 21:24:17
Quote
Quote
2.5 ranked references per sample (or a similar value)

Another problem introduced by this is: where to draw the line?

calculate the average (i guessed its 1 ranked reference per sample) and add 1.5

also someone can say that a user has a 50% chance to vote the reference, which is equal to 2.5 votes per sample, over this 50% is too much bad luck for a serious tester
Title: AAC @ 128kbps listening test discussion
Post by: elmar3rd on 2004-02-29 21:25:32
Quote
Another problem introduced by this is: where to draw the line?

Maybe those, who rated the reference below 4.5 or 4.0?
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-02-29 21:37:07
Quote
Quote
Another problem introduced by this is: where to draw the line?

Maybe those, who rated the reference below 4.5 or 4.0?

bond and Roberto were referring to how many ranked references should be acceptable.  bond's proposal is to actually look at the data and find out how many references are ranked on average in those results files where it occurs.  Then add 1.5 to that number to determine where to draw the line of how many ranked references are acceptable.

Over that line and the entire file is thrown out.  Under that line, and the scores of the ranked reference codecs are changed to 5.0.  This is a reasonable and conservative proposal.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: AstralStorm on 2004-02-29 21:47:12
Not a bad thing to do.
I'm for this proposal, it should filter out people trying to make results a white noise.
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-29 22:10:45
I still think that the existence of a successful ABX test is a much better indication of seriousness. Of course, as bond noted, a listener can be serious without doing any ABX tests. But if he has done some, then we can be *certain* that he was indeed serious about it.
Just to clarify, my thoughts, which Roberto posted above, were only about ranked references with successful ABX tests. If there is no ABX test or a failed ABX, I think the file should be discarded, since for all we know the listener just played around with the sliders (see the results package Roberto posted earlier).
If there is a successful ABX result, I personally don't find it necesary to discard the results, especially if there are not enough results for such luxury. It's unfortunate that we can't just throw out the ranked reference and still consider the other rankings. On the other hand, it's obvious that it might cast a shadow on the professionalism of the test if we consider the rating of the reference for the encoded sample. In my personal opinion, I wouldn't mind using this method if the listener obviously made an effort, but I can see that it might not be in the best interest of the test as a serious reference later on. That considered, it might be best to count a ranked reference with a valid ABX result as 5.0 (contrary to my earlier thoughts).

Furthermore, I guess Roberto will as usual publish the results files, so if anyone is interested in what the results would have looked like when calculated with a laxer method, we could still do this after the test ("inofficially" so to say).
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-02-29 22:22:51
Quote
If there is no ABX test or a failed ABX, I think the file should be discarded, since for all we know the listener just played around with the sliders

i think our different opinions on that maybe simply depend on the personal way of doing the test? (as i said before i dont think that the relationship pointed out by schnofler is that clear at all)

anyways i think i made my point clear on how to detect unserious testers on a, imho, clearer way than via abx results
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-02-29 22:57:05
Quote
i think our different opinions on that maybe simply depend on the personal way of doing the test?

It is indeed the case that I never rank a sample without an ABX unless the difference is extremely obvious to me. However, I prefer to see it the other way round: my preference on how to do the test stems from my opinion on what should be considered serious, not vice versa. 

Anyway, please don't take this personal. As I said above I personally don't have any problems considering results as valid if it is somehow clear to me that the listener was serious (and yes, simply because you participate in this discussion, I trust you on your seriousness). The problem is how to make this reasonable for others. We have to choose a definite, easily justifiable way of doing this, so the results of this test can be used as a serious reference. A successful ABX test is, in my opinion, a very strong sign that the listener did indeed put a considerable effort into this. The proposal you support just doesn't seem as strong to me.

Just to clear things up, I think there is a bit of misunderstanding about this proposal. If I understood you correctly (please correct me if I'm wrong), you want to look at all the results files from a certain listener, count the ranked references, and calculate the average of ranked references per file. If it's above 2.5 throw out the whole set of files from this listener.

But from this
Quote
Over that line and the entire file is thrown out.

I understood that ff123 wants to decide this on a file-by-file-basis (again, please correct me if it's a misunderstanding).

In any case, if you move sliders randomly you still have a 50% chance that your results (single files or the whole set) will be accepted, which is far too high in my opinion. If it is done that way, I would draw the line much lower (e.g. 1 ranked reference on average).

I still like using ABX results better, because they actually give you evidence of the listener's efforts, while the other proposal just aims at making an educated guess.
Title: AAC @ 128kbps listening test discussion
Post by: Dologan on 2004-02-29 23:25:16
I stand by my proposal. Untamperable ABX scores is IMO the most reliable tool to know whether the person could hear the difference or not. Since we are in a position where every valid result is valuable, dumping ranked references which have been ABXed successfully seems a waste of data we usually cannot afford. Moreover, ABX test provide data not only from the p-value but also, to a certain degree, from the number of trials the person needed to get the significant result (esp. if we have access to the ABX trial sequence for that sample). If a person needed 40 trials to get a statistically valid result you can tell at least that the difference wasn't all that obvious on the part he ABXed (and much less with 200 trials). However, a respectable result of say, 10/11, 14/16 or 20/24 speaks for a consistently picked up difference. In such a case, counting ranked references as 5.0 also removes potentially relevant information about a codecs quality, since we are disregarding the evidence that the sample indeed is not transparent.
Ranking a sample >4.0 acknowledges that the sample doesn't really sound worse than the original to the listener, only "different"; and therefore, with good ABX results backing it, it is reasonable to assume that due to fatigue or a lapse of concentration, the reference could be mistakenly ranked, without making the result less valid.
The course of action I would take is:
1) See if we have enough kosher results to get statistically valid results.
2) If not, or if the non-kosher, marginally valid results are too voluminous to be discarded with a clear conscience; then take the rank value of the ranked references with successful, non-extreme ABXs above 4.0.
Title: AAC @ 128kbps listening test discussion
Post by: jido on 2004-02-29 23:52:07
Sorry I didn't help for this test 

For the discussion about ranked references, I would take them as 5.0, unless there is evidence of a mistake at the time of ranking (ABX results with small p-value) in which case I would use the specified rank.

I would also contact the authors to inform them of what you do with their ranked references.

Bogus data (someone playing with the sliders) can be recognised in that it doesn't fit with the other results and makes the error margin excessive.
Title: AAC @ 128kbps listening test discussion
Post by: jido on 2004-02-29 23:59:26
Just realised that ranked reference with a small ABX value does not necessarily indicates a mistake. As commented before, the tester could like more the encoded sample. Make them all 5.0 I think.
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-03-01 00:11:48
From my own experience:

it happens that I've rated the reference, but ABX the file without any difficulty. How? Often, the mistake occurs with the first pair of the test. I'm not careful, and I hear weird sound which is a property of the reference (some samples are weird for some users). Then, I'm rating (correctly) all other files, but I forget to reevaluate the first file.
This mistake is rare (occuring when I'm performing the test too quickly), but it happens to me some times...
In my opinion, if ABX tests are correct, the wrong notation must be invert, and not be cancelled.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 00:24:28
Fortunately, Phong's wonderful chunky (http://www.phong.org/chunky/) result parser can take into consideration ABX pvals to accept or discard ranked references (you may now bow to Phong), so it would be quite easy and fast for me to calculate results without ranked references and with ABXd ranked references, and then decide if the ranked references will make a difference in final scores or not.

Considering I/we decide to go with ABX results to decide about using a ranked references or not, what would be a good pval cutoff line? p < 0.05? p < 0.01?


BTW: It can also parse results taking into consideration number of ABX trials (Phong rocks). Do you think this should be used to select results too?
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-03-01 00:29:52
Just a comment (to ABC/HR software development).

I'm somtimes tempted to perfom multiple ABX tests on a difficult part of a sample. There are some points where artifact is obvious, but I'd like to test something more difficult. I never do it, simply because I fear to fail on this test. It wouldn't be serious to rank a file 2.0/5, with bad ABX scores.
Therefore, comment should be added ofor ABX sessions (and why not, the possibility of doing multiple ABX tests for one single sample). Simply to inform the reader of the final log about the real conditions of the ABX tests (and score)
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 00:53:05
So? Opinions on pvalues? Pretty please? :/
Title: AAC @ 128kbps listening test discussion
Post by: schnofler on 2004-03-01 01:11:39
Quote
Considering I/we decide to go with ABX results to decide about using a ranked references or not, what would be a good pval cutoff line? p < 0.05? p < 0.01?

By "using a ranked reference", do you mean using the score for the encoded sample or counting it as a 5.0?
As for pval cutoff, that's not an easy question. I usually try to get it down to 1% if the sample is really hard. But if I can hear a somewhat clear difference, I usually don't bother going on after achieving 7/8 (pval=3.5%) or something similar. So if you cut off at 1% you might discard some instances where the listener heard a pretty good difference but just, well, missed the slider. I don't know how others handle this. Maybe you could try it and see what p-val most people go for, and if too much would be discarded at 1%, use 5% as cutoff. Or you could count the listener's rating if pval is <1% and count it as 5.0 if pval <5%.

Quote
BTW: It can also parse results taking into consideration number of ABX trials (Phong rocks). Do you think this should be used to select results too?

No. A big number of trials is not necessarily a sign that the listener didn't hear a clear difference. He might have switched the playback range halfway through or needed a bit of practice (I remember one test where I got 1 correct during the first 8 trials, and 23 correct in the following 24 trials). The pval is the measurement of significance, after all, and it should be used as such.

Quote
I'm somtimes tempted to perfom multiple ABX tests on a difficult part of a sample. There are some points where artifact is obvious, but I'd like to test something more difficult. I never do it, simply because I fear to fail on this test. It wouldn't be serious to rank a file 2.0/5, with bad ABX scores.

Yeah, I know that problem. You could try a different part and if it doesn't work out, add a few more trials from a part which is easy for you to get the pval back down. But I admit this is not the most elegant of solutions.

Quote
Therefore, comment should be added ofor ABX sessions (and why not, the possibility of doing multiple ABX tests for one single sample). Simply to inform the reader of the final log about the real conditions of the ABX tests (and score)

How would you want such a comment feature to work? (I mean, how would it be different from the sample comments?). Multiple tests might be a solution, but it also might be hard to handle. Or the program could just write a more detailed ABX log into the results file, maybe something like WinABX. But then the ABX tests would probably make up the largest part of the results files. Anyway, ideas are welcome.

edit: Woohoo! 100th post.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-01 01:13:03
Quote
So? Opinions on pvalues? Pretty please? :/

One other option:

Recalculate the ABX p-value, including the trial where the listener ranks the reference.  So if the ABX reads 14/16 in the abx section and then he ranks the reference in the ABC/HR section, the total ABX would be 14/17.  That's what I did for the first 64 kbit/s test.  It's a lot of manual labor, though

But to answer your question, I would use p = 0.05
Title: AAC @ 128kbps listening test discussion
Post by: Dologan on 2004-03-01 01:21:14
Quote
Considering I/we decide to go with ABX results to decide about using a ranked references or not, what would be a good pval cutoff line? p < 0.05? p < 0.01?


BTW: It can also parse results taking into consideration number of ABX trials (Phong rocks). Do you think this should be used to select results too?

I'd say p < 0.05, since it's usually the scientifically accepted p-value of significance and I don't think we want to be too strict with the "ranked reference consideration threshold". I do, however, think we should take into consideration the number of ABX trials, since a large amount of trials does speak for a hard to distinguish difference which might as well be considered as 5.0 (for the sake of Garf's rationale). (*) How large should this amount be, is arguable. I would set the cutoff whenever the number of successes to get a p <0.05 for the given number of trials is less than 75%, this could be seen from some table. (As a rough estimate I'd say the treshold should lie above 30, which is, from my experience, the number of trials I need for nearly-transparent samples).

BTW, does Phong's chunky enable you to set a rank limit (such as my proposed >4.0) for consideration as well as how to consider the ranked references (as a 5.0 or the given value)?  If so, I would also enforce the minimum rank of 4.0 for taking the given score and set a 5.0 for below-threshold scores. How credible is a test that states the original is annoying with respect to the encoding?

(*) EDIT: After reading Schnoflers reply, I acknowledge he has a point regarding the "warming up" or range selection during ABXing. This kind of issues could be easily solved if the program tracks the ABX trials, instead of the final result only. A long run of successes after several failures suggests the warm-up kicking in or a change of range to an easier one. However, interspersed successes and failures over many trials DOES almost unequivocally speak for a hard to distinguish difference.
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-03-01 01:48:03
Quote
I'm conducting the test right now, but sometimes the applet crashes

Yes, it seems to freeze when it plays past the end point (which is shouldn't be doing)

I just saved my sessions after any significant change... unfortunately I did lose hard ABX results sometimes
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 01:49:25
Quote
BTW, does Phong's chunky enable you to set a rank limit (such as my proposed >4.0) for consideration as well as how to consider the ranked references (as a 5.0 or the given value)?

Unfortunately, I don't think it does.

Anyway, I am fairly sure I got enough "clean" results. And preliminary testing shows that there won't be much of a difference if I count ranked results or not. So I will publish "clean" results at the results page, but will also make the ranked individual results available for download so that people can use them to play around with and see if they come with different rankings somehow.

Just to clarify: If there are enough clean results (and I'm fairly sure there will be), the clean results will be used to create the "official" rankings that will be posted at rjamorim.com/test.
I will discard any files where the reference was ranked, even if it was ABXd.

Edit: by "clean", I mean result files where the test participant didn't rank the reference files.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 01:55:28
And enlightening comment about the fun of ABX, courtesy of music_man_mpc:

Quote
1L File: Sample04\gone_5.wav
1L Rating: 4.9
1L Comment: I GOT IT!!!!  I FINALLY GOT IT!!!!!  67/102 for ABX.  GODDAMN $#@! HARD!!




@bond: People do take more than 100 trials, it seems
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-03-01 02:00:12
Quote
the results of abx'es should in no way influence the decision whether to use the ranked sources results or not!!!
noone is forced to do abx, you cant rely on whether someone did abx or not

in fact someone can do the whole test without abxing and without being unserious or having anything bad in mind


as i proposed unserious testers should be sorted out via the way if there are far over the average ranked sources in the results

Quote
No, you are completely wrong, see my earlier example.

You can ABX 110/200, which is significant, but your chance of pulling the correct slider is only 55%.

well your example is not usable in this case as its unrealistic/only theoretical
noone will do 200 abx'es

ABX results are part of the test dude

I know someone who abxed every single sample to >0.001

And yes... some people do abx 200+ times... although the most I ever went was about 50 I think

Luckily I don't claim to be a golden ear
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 02:01:07
One hour to go!

After that, I will stop receiving results and will start calculating the plots.

If you didn't send me your results yet, do so now!
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-03-01 02:03:11
Quote
Quote
This is a mess, isn't it?

Jesus Christ, it is!

So, is it normal for a test to have this many ranked references?

Or is it a sign of how  aac@128kbps is fairly transparent for a large segment of the test population?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 02:07:37
Quote
So, is it normal for a test to have this many ranked references?

Or is it a sign of how  aac@128kbps is fairly transparent for a large segment of the test population?

Second answer

I guess several people are finding out their transparency thresold is around 128kbps, and not around 160-192 as it has been believed for years (although the latter figures probably still apply to MP3)
Title: AAC @ 128kbps listening test discussion
Post by: Dologan on 2004-03-01 02:07:57
Quote
Or is it a sign of how  aac@128kbps is fairly transparent for a large segment of the test population?

Yes. It's also a sign that people are not always careful, that placebo exists and that they dislike not being able to hear a difference and put a 5.0.
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-03-01 02:15:54
Quote
Considering I/we decide to go with ABX results to decide about using a ranked references or not, what would be a good pval cutoff line? p < 0.05? p < 0.01?

BTW: It can also parse results taking into consideration number of ABX trials (Phong rocks). Do you think this should be used to select results too?

Whatever it is when ABC/HR.jar goes green

I don't think non abx'd non-ranked-reference results should be discarded , after all abxing is NOT compulsory...
Title: AAC @ 128kbps listening test discussion
Post by: Stux on 2004-03-01 02:17:53
Quote
Just a comment (to ABC/HR software development).

I'm somtimes tempted to perfom multiple ABX tests on a difficult part of a sample. There are some points where artifact is obvious, but I'd like to test something more difficult. I never do it, simply because I fear to fail on this test. It wouldn't be serious to rank a file 2.0/5, with bad ABX scores.
Therefore, comment should be added ofor ABX sessions (and why not, the possibility of doing multiple ABX tests for one single sample). Simply to inform the reader of the final log about the real conditions of the ABX tests (and score)

Yes, for instance, you want to see if you can ABX one artifact, but know you can ABX another artifact...

Anywho, you could do this by opening up a new session and doing your 'hard' abx in that session...
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-03-01 02:36:35
Quote
Anywho, you could do this by opening up a new session and doing your 'hard' abx in that session...

Mmhhhh...
It's possible if the whole test include one file against the reference. But if you have 5 files (and therefore 10, because of blind reference for each encoded sample), you have first to find again the specific sample where you believe you heard something wrong. Not easy... and waste of time. And with encrypted log files, it's even harder to be sure that you're ABXing the good file
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 02:44:22
Can someone enlighten me on the origins of Velvet?
http://lame.sourceforge.net/download/samples/velvet.wav (http://lame.sourceforge.net/download/samples/velvet.wav)

All I know is that it was submitted by Roel (r3mix).

Does anybody know artist (Velvet Underground?), title and album of this song? Also, what would be the style (no way to figure out from just the introduction)

Thank-you.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 03:01:02
Listening test is now CLOSED

Expect results to be posted real soon...

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-03-01 12:22:48
rjamorim
sorry for bugging again, but what way on how to treating ranked refernces did you use now? i couldnt see that from your posts 

btw also you seemed to have missed out my sample 09 results!? maybe it was because they were all transparent rated?

Quote
ABX results are part of the test dude

no its not 
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 17:12:45
Quote
rjamorim
sorry for bugging again, but what way on how to treating ranked refernces did you use now? i couldnt see that from your posts 

I simply ignored ranked references. Since I got enough "clean" results, it was OK.

Quote
btw also you seemed to have missed out my sample 09 results!? maybe it was because they were all transparent rated?


Nope. I couldn't decrypt your sample 09 results. It's the only result file that gave me problems in the entire test. I sent it to schnofler so that he can investigate. Sorry about that.
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-01 17:18:05
Quote
Can someone enlighten me on the origins of Velvet?
http://lame.sourceforge.net/download/samples/velvet.wav (http://lame.sourceforge.net/download/samples/velvet.wav)

All I know is that it was submitted by Roel (r3mix).

Does anybody know artist (Velvet Underground?), title and album of this song? Also, what would be the style (no way to figure out from just the introduction)

ff123 already enlightened me about it. Thank-you very much.

Details are available at the listening test results page.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-03-01 17:18:17
Quote
Nope. I couldn't decrypt your sample 09 results. It's the only result file that gave me problems in the entire test. I sent it to schnofler so that he can investigate. Sorry about that.

damn, i shouldnt have tried to manipulate the resultfiles
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-05 06:49:41
[span style='font-size:14pt;line-height:100%']A VERY IMPORTANT STATEMENT[/span]

OK. It seems I f-ed up very badly this time.

First, let me specify what ISN'T wrong: The ranking values are absolutely correct, as well as the screening methodology and the statystical calculations.

What is wrong: The error bars.

I didn't check how the error bars were being drawn in the excel spreadsheet I got from ff123. I thought the plots were getting values from a certain cell, but actually the values were hard-coded in the plot building routines.

So, the error bars are to this day the same as the ones used in his 64kbps listening test. And it affects all my listening tests. Both the overall plots and the individual ones.

I can't express how sorry I am.

Tomorrow I'll start fixingall the test results pages. Until I announce the results have been fixed, please disregard them.

In case someone is in a hurry to check the corrected zoomed result plot for the AAC test:
http://pessoal.onda.com.br/rjamorim/screen2.png (http://pessoal.onda.com.br/rjamorim/screen2.png)
The only thing that changed is that iTunes is now clearly first place and Nero is second place.

Again, I'm terribly sorry. I can already feel my credibility going down the drain.

Kind regards;

Roberto Amorim.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 07:00:48
Quote
What is wrong: The error bars.

I didn't check how the error bars were being drawn in the excel spreadsheet I got from ff123. I thought the plots were getting values from a certain cell, but actually the values were hard-coded in the plot building routines.

The fault is also mine for not making it perfectly clear how I was drawing the error bars.  Plus I violated an Excel/software rule by not using a spreadsheet as a spreadsheet should be used, instead hard-coding in the error bar values.

Quote
Again, I'm terribly sorry. I can already feel my credibility going down the drain.


Your integrity is intact.  Credibility is a matter of trust.  If you own up to your mistakes, correct them, and prevent future ones, that goes a long way towards enhancing your credibility.

I suggest keeping both the old (incorrect) overall graphs and showing the new, corrected overall graphs side by side, to show the before and after.  I think the individual sample graphs can just be replaced.

ff123

Edit:  You should probably rename the old overall graph and then use the original name of the graph for the corrected one.  That way, websites which link to your overall graphs will be automatically updated.
Title: AAC @ 128kbps listening test discussion
Post by: rpop on 2004-03-05 07:12:55
Quote
Your integrity is intact.  Credibility is a matter of trust.  If you own up to your mistakes, correct them, and prevent future ones, that goes a long way towards enhancing your credibility.

Your integrity is, indeed, intact. I've seen a few other listening tests online, and discussion of their results always stops soon after the tests, with the page receeding in internet history. Updating these tests now goes a long way toward proving their reliability will be maintained in the future .
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-03-05 08:01:49
Quote
In case someone is in a hurry to check the corrected zoomed result plot for the AAC test:
http://pessoal.onda.com.br/rjamorim/screen2.png (http://pessoal.onda.com.br/rjamorim/screen2.png)
The only thing that changed is that iTunes is now clearly first place and Nero is second place.

Aaaaaah, this explains my previous complaint that the graph didn't seem to align with your written statement about the test significance

Now it does. iTunes indeed almost beats Nero by a significant margin.

As far as the moral winner is concerned, though:
Title: AAC @ 128kbps listening test discussion
Post by: Continuum on 2004-03-05 08:20:54
Quote
As far as the moral winner is concerned, though:


"Moral winner"?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-05 08:27:52
Quote
Now it does. iTunes indeed almost beats Nero by a significant margin.

Erm.. I use Darryl's method to evaluate ranking positions.

Check, for instance, thear1 in his 64kbps test results
http://ff123.net/64test/results.html (http://ff123.net/64test/results.html)

Oggs are ranked second, according to him, although they overlap a little with MP3pro.

To put it short, I (and ff123, it seems) only consider codecs tied when one's confidence margin overlaps with the other's actual ranking. Or, to make things simpler, when more than half of the entire margins overlap.
Title: AAC @ 128kbps listening test discussion
Post by: Gabriel on 2004-03-05 08:28:54
Quote
I can already feel my credibility going down the drain


Finding, admitting, correcting your own errors only increases credibility I think.
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-03-05 12:29:22
Your credibility, your honesty and your honor are now stronger. Thank you.
Title: AAC @ 128kbps listening test discussion
Post by: ScorLibran on 2004-03-05 15:12:50
You have nothing to worry about, Roberto...you're credibility is quite secure.  Anyone who conducts tests like this will occasionally have a mistake.  It's inevitable.  You took the best approach in resolving it.  Our trust in you is only higher now. 

Quote
Quote
Now it does. iTunes indeed almost beats Nero by a significant margin.

...To put it short, I (and ff123, it seems) only consider codecs tied when one's confidence margin overlaps with the other's actual ranking. Or, to make things simpler, when more than half of the entire margins overlap.

That's what I had always thought was the case, but it was just an assumption on my part (that I never communicated).  Glad to know it was correct.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 15:34:35
Quote from: ScorLibran,Mar 5 2004, 07:12 AM
...To put it short, I (and ff123, it seems) only consider codecs tied when one's confidence margin overlaps with the other's actual ranking. Or, to make things simpler, when more than half of the entire margins overlap.

That's what I had always thought was the case, but it was just an assumption on my part (that I never communicated).  Glad to know it was correct. [/quote]
To be absolutely correct, a codec wins with 95% confidence, for that group of listeners and set of samples, when the bars do not overlap.  Or to put it another way, 19 times out of 20, those results would not occur by chance.  Any overlap reduces that confidence.  If the bars just barely overlap, there is still quite a high likelihood that that result did not occur by chance.  A reasonable way to describe this situation would be to say that the results are suggestive (if not significant).  Actually, in an ideal world, the graphs would speak for themselves, and there would be no "interpretation" to cause controversy.

If this were a drug test or something else where there is a lot at stake for making the right decision, everything below 95% confidence (or whatever threshold is chosen) would not be considered to be significant.

Also, the test would be corrected for comparing multiple samples, which would make the error bars overlap more.  I personally don't think it's a real big deal if the type I errors in this sort of test (falsely identifying a codec as being better than another) are higher than they would be in a more conservative analysis.  But others, for example on slashdot, can (and do) complain about this sort of thing.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-03-05 15:47:50
I take it from the previous comment by rjamorim that 'bars' should be interpreted as 'error bars' and 'mean score marker' and not 2x 'error bars'?
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 16:05:22
Quote
Check, for instance, thear1 in his 64kbps test results
http://ff123.net/64test/results.html (http://ff123.net/64test/results.html)

Oggs are ranked second, according to him, although they overlap a little with MP3pro.

In that test I used an "eyeball" method to rank the codecs when trying to determine an appropriate overall ranking.  People (including me) didn't like the subjectivity involved in that method, so I changed to the method used now, which is to perform another ANOVA/Fisher LSD once the means for each music sample are determined.  The assumption this method makes is that each sample is equally important to the final overall results.  This may not actually be true if, for example, there are lots of people listening to some samples and only a few listening to others.  Also, the choice of samples greatly affects the overall results.

But at least it seems to produce reasonable results, and it's removed the subjectivity involved in the earlier method.

Quote
I take it from the previous comment by rjamorim that 'bars' should be interpreted as 'error bars' and 'mean score marker' and not 2x 'error bars'?


The length of each error bar from top to bottom (mean in the middle) is equal to the Fisher LSD.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-03-05 16:16:10
Quote
To be absolutely correct, a codec wins with 95% confidence, for that group of listeners and set of samples, when the bars do not overlap.  Or to put it another way, 19 times out of 20, those results would not occur by chance.  Any overlap reduces that confidence.  If the bars just barely overlap, there is still quite a high likelihood that that result did not occur by chance.  A reasonable way to describe this situation would be to say that the results are suggestive (if not significant).  Actually, in an ideal world, the graphs would speak for themselves, and there would be no "interpretation" to cause controversy.

If this were a drug test or something else where there is a lot at stake for making the right decision, everything below 95% confidence (or whatever threshold is chosen) would not be considered to be significant.

Also, the test would be corrected for comparing multiple samples, which would make the error bars overlap more.  I personally don't think it's a real big deal if the type I errors in this sort of test (falsely identifying a codec as being better than another) are higher than they would be in a more conservative analysis.  But others, for example on slashdot, can (and do) complain about this sort of thing.

ff123

Right, well, with 95% confidence for the tested 12 samples:
iTunes is better than Real,FAAC and Compaact
Nero is better than Real and Compaact

With lower confidence for the tested 12 samples:
Nero is better than FAAC (small overlap)

With even lower confidence for the tested 12 samples:
iTunes is better than Nero (a bit bigger overlap than with Nero-FAAC)

Correct?
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-03-05 16:36:26
Quote
The length of each error bar from top to bottom (mean in the middle) is equal to the Fisher LSD.

So there shouldn't be any overlap between error bars at all, if I get that correctly, since no overlap between error bar and mean is only half the error length. (And hence my original comment was right).
Title: AAC @ 128kbps listening test discussion
Post by: Zed on 2004-03-05 16:53:18
Quote
Quote
Now it does. iTunes indeed almost beats Nero by a significant margin.

Erm.. I use Darryl's method to evaluate ranking positions.

Check, for instance, thear1 in his 64kbps test results
http://ff123.net/64test/results.html (http://ff123.net/64test/results.html)

Oggs are ranked second, according to him, although they overlap a little with MP3pro.

To put it short, I (and ff123, it seems) only consider codecs tied when one's confidence margin overlaps with the other's actual ranking. Or, to make things simpler, when more than half of the entire margins overlap.

how about this one (http://www.infoanarchy.org/comments/2002/9/8/23472/23921/0/post)?

where is the truth?
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 17:05:34
Quote
Quote
The length of each error bar from top to bottom (mean in the middle) is equal to the Fisher LSD.

So there shouldn't be any overlap between error bars at all, if I get that correctly, since no overlap between error bar and mean is only half the error length. (And hence my original comment was right).

Yes.  If the error bars do not overlap, that is a difference to 95% confidence.  And yes, iTunes almost beats Nero with 95% confidence.
Title: AAC @ 128kbps listening test discussion
Post by: eagleray on 2004-03-05 17:12:57
Is there anything in the testig methodology to assure that iTunes does not sound "better" than the original CD through the addition of some audio "sugar"?

I hope the experts around here do not think this is too off the wall.  For that matter I don't know if there is a way to make any recording sound "better" than the original.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 17:16:00
Quote
this one (http://www.infoanarchy.org/comments/2002/9/8/23472/23921/0/post)?

where is the truth?

The biggest weakness of this test IMO is that there were only 3 samples tested, and they made it even worse by combining them into one medley.  Other problems:  IIRC, people were asked to rank the codecs from best to worst, not to compare and rate against a known reference.  I believe the reference was hidden as one of the samples to be ranked.

But the 3 sample medley is really the killer.  They would have been much better off distributing lots of different samples (with that amount of listeners they could have distributed 50 different samples with ease) to determine which codec is better overall.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-05 17:17:17
Hello.

Thank-you very much for your support

I have been correcting the plots (will upload them later) and so far, it seems very few will change:

-At the first AAC@128kbps test, it only becomes more clear that QuickTime is the winner.
-At the Extension test, it seems Vorbis and WMAPro are no longer tied to AAC and MPC, and now share second place. I'll leave it to others to discuss.
-The 64kbps test results stay the same: Lame wins, followed by HE AAC, then MP3pro, then Vorbis. LC AAC, Real and WMA are still tied at fifth place, and FhG MP3 is still way down the graph.
-The MP3 test stays the same as well.

Regards;

Roberto.
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 17:17:47
Quote
Is there anything in the testig methodology to assure that iTunes does not sound "better" than the original CD through the addition of some audio "sugar"?

I hope the experts around here do not think this is too off the wall.  For that matter I don't know if there is a way to make any recording sound "better" than the original.

Yes, the listener is asked to rate the sample against the reference.  The reference is 5.0 by default, so any difference, even if it "sounds better" than the reference must be rated less than 5.0

ff123
Title: AAC @ 128kbps listening test discussion
Post by: Zed on 2004-03-05 17:28:52
Quote
But the 3 sample medley is really the killer.  They would have been much better off distributing lots of different samples (with that amount of listeners they could have distributed 50 different samples with ease) to determine which codec is better overall.

but small number of the ears is also the killer i guess...
Title: AAC @ 128kbps listening test discussion
Post by: ff123 on 2004-03-05 17:57:56
Quote
Quote
But the 3 sample medley is really the killer.  They would have been much better off distributing lots of different samples (with that amount of listeners they could have distributed 50 different samples with ease) to determine which codec is better overall.

but small number of the ears is also the killer i guess...

They had about 3000 listeners for both the 64 kbit/s and 128 kbit/s tests.  If they had distributed 50 separate samples instead of the one medley, they could have gotten more than 50 listeners per sample.  That's more than enough to make a statistical inference.  In fact, one can do quite well with far fewer.

ff123
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-03-05 18:07:46
The test also seems at least 1.5 years old. Lots has happened in that time with AAC.
Title: AAC @ 128kbps listening test discussion
Post by: ScorLibran on 2004-03-05 18:21:47
Quote
The test also seems at least 1.5 years old. Lots has happened in that time with AAC.

I was going to say the same thing.

Plus, it seems the results of that test have been recycled from time to time over the past year-and-a-half.  I'm sure I've seen them multiple times now.

After at least one major update has been made to a codec involved in a test, the test results will be out of date (at least when discussing the current version of a format).  Only recent results can be trusted, and only until such a code change occurs again.

(This is in addition to the other shortcomings of the afforementioned test.)
Title: AAC @ 128kbps listening test discussion
Post by: guruboolez on 2004-03-05 18:33:26
Quote
Plus, it seems the results of that test have been recycled from time to time over the past year-and-a-half.  I'm sure I've seen them multiple times now.

I could confirm that.
Title: AAC @ 128kbps listening test discussion
Post by: dewey1973 on 2004-03-05 18:49:58
I think rjamorim might take offense to this...

http://www.macworld.co.uk/news/top_news_item.cfm?NewsID=8097 (http://www.macworld.co.uk/news/top_news_item.cfm?NewsID=8097)

Quote
Apple's iTunes has emerged victorious in a series of listening tests run by the [span style='font-size:21pt;line-height:100%']CD Freaks Web site[/span].


Title: AAC @ 128kbps listening test discussion
Post by: indybrett on 2004-03-05 18:58:34
Holy cow. Will Apple be using Roberto's test results in commercials now
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-05 19:10:46
Quote
I think rjamorim might take offense to this...

http://www.macworld.co.uk/news/top_news_item.cfm?NewsID=8097 (http://www.macworld.co.uk/news/top_news_item.cfm?NewsID=8097)

Quote
Apple's iTunes has emerged victorious in a series of listening tests run by the [span style='font-size:21pt;line-height:100%']CD Freaks Web site[/span].


 

[bad language removed by moderation]

This is too much. I already got pissed when Slashdot claimed my tests were conduced by Hydrogenaudio. But this is ridiculous!
Title: AAC @ 128kbps listening test discussion
Post by: smok3 on 2004-03-05 19:48:59
Quote
I think rjamorim might take offense to this...

probably a silly mistake, i took a liberty and send email to the news editor (Jonathan Evans, http://www.macworld.co.uk/contact/ (http://www.macworld.co.uk/contact/) ) with a request to fix the error.
Title: AAC @ 128kbps listening test discussion
Post by: bidz on 2004-03-05 19:50:24
Quote
Quote
I think rjamorim might take offense to this...

probably a silly mistake, i took a liberty and send email to the news editor (Jonathan Evans, http://www.macworld.co.uk/contact/ (http://www.macworld.co.uk/contact/) ) with a request to fix the error.

Me too
Title: AAC @ 128kbps listening test discussion
Post by: indybrett on 2004-03-05 19:51:36
Quote
Quote
Quote
I think rjamorim might take offense to this...

probably a silly mistake, i took a liberty and send email to the news editor (Jonathan Evans, http://www.macworld.co.uk/contact/ (http://www.macworld.co.uk/contact/) ) with a request to fix the error.

Me too 

Me too, and I got a reply. Guess his mailbox is getting busy
Title: AAC @ 128kbps listening test discussion
Post by: Garf on 2004-03-05 19:54:22
Their interpretation of the result is also factually wrong, and different from what roberto wrote.
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-03-05 19:54:39
Quote
This is too much. I already got pissed when Slashdot claimed my tests were conduced by Hydrogenaudio. But this is ridiculous!

Hehe, I think the problem is that they want to refer to some known web site. Rarewares is not so known really. And since you don't want to mention Hydrogenaudio's strong support anymore in the results page in providing test discussion and test participants (in fear of sites likes Slashdot referring to the test as HA test), this is what happens..

I'm sure CD-Freaks don't mind though.. 
Title: AAC @ 128kbps listening test discussion
Post by: indybrett on 2004-03-05 20:48:42
Look again...

http://www.macworld.co.uk/news/top_news_item.cfm?NewsID=8097 (http://www.macworld.co.uk/news/top_news_item.cfm?NewsID=8097)
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-05 23:56:59
Quote
And since you don't want to mention Hydrogenaudio's strong support anymore in the results page in providing test discussion and test participants (in fear of sites likes Slashdot referring to the test as HA test), this is what happens..

Blah. Bollocks. I give lots of credits to HydrogenAudio at the listening tests start page. And they insisted on keeping the CDfreaks link even after I pointed they were wrong (they also ruined their HTML, heh. morons). So, it's just that they seem to want to advertize Freaks, in some way or another.
Title: AAC @ 128kbps listening test discussion
Post by: JohnV on 2004-03-06 00:08:15
Quote
Quote
And since you don't want to mention Hydrogenaudio's strong support anymore in the results page in providing test discussion and test participants (in fear of sites likes Slashdot referring to the test as HA test), this is what happens..

Blah. Bollocks. I give lots of credits to HydrogenAudio at the listening tests start page. And they insisted on keeping the CDfreaks link even after I pointed they were wrong (they also ruined their HTML, heh. morons). So, it's just that they seem to want to advertize Freaks, in some way or another.

Well, at the same time did you ask them to change the "CD-Freaks is reporting" to "Hydrogenaudio is reporting"?
This is not the first time freaks take credit for something they have no business what so ever..
Title: AAC @ 128kbps listening test discussion
Post by: tigre on 2004-03-06 10:30:54
Zed's ranting moved here (http://www.hydrogenaudio.org/forums/index.php?showtopic=19376).
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-07 20:49:02
Quote
[bad language removed by moderation]

Duh...
http://cornelldailysun.com/articles/11061/ (http://cornelldailysun.com/articles/11061/)

Anyway: Already fixed the plots for the first AAC test and the 128kbps extension test. Working on the 64kbps test plots now.

Edit: 64kbps results are up too. Will work on MP3 128 and AACv2 later. Stay tuned...
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-07 20:50:37
Quote
Well, at the same time did you ask them to change the "CD-Freaks is reporting" to "Hydrogenaudio is reporting"?

Nope, because I didn't know they would insist on mentioning CDfreaks. I guessed they would replace it with a link to rjamorim.com. It seems to me they deliberately want to give some sort of credit to CDfreaks.
Title: AAC @ 128kbps listening test discussion
Post by: music_man_mpc on 2004-03-07 23:34:13
Quote
It seems to me they deliberately want to give some sort of credit to CDfreaks.

Thats absolutely absurd.  Does CDfreaks have some affiliation with Apple?
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-03-07 23:47:56
Quote
Thats absolutely absurd.  Does CDfreaks have some affiliation with Apple?

Not that I know of. But why insist of mentioning them at their article? CDfreaks weren't the only ones announcing it, it was also announced by Afterdawn, Slashdot etc.
Title: AAC @ 128kbps listening test discussion
Post by: music_man_mpc on 2004-03-08 00:25:15
Quote
Not that I know of. But why insist of mentioning them at their article? CDfreaks weren't the only ones announcing it, it was also announced by Afterdawn, Slashdot etc.

Thats what I was saying, I meant it was absurd of them, not you .  Sorry for not being totally clear.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-04-21 08:22:08
i just noted that the results of the last aac test still point to the old, not corrected plots

it gets displayed this old plot by default:
(http://rarewares.hydrogenaudio.org/rja/plot12z.png)
(rarewares.hydrogenaudio.org/rja/plot12z.png)
which is the one i created right after the test and surely hasnt the corrected error bars

still you seem to have already uploaded the correct plots already here:
(http://www.rjamorim.com/test/64test/plot12z.png)
(www.rjamorim.com/test/64test/plot12z.png)
but the result page doesnt link to it
Title: AAC @ 128kbps listening test discussion
Post by: rjamorim on 2004-04-21 08:43:14
OK, after lots of interpretation, I understood what you are saying.

Keep in mind this is the AAC test thread. And you mention "the results of the last aac test", and right under you link graphics of the 64kbps test. There was a knot in my brain ("what is he talking about after all?") 

Fixed it now.  Thanks for reporting.
Title: AAC @ 128kbps listening test discussion
Post by: bond on 2004-04-21 08:47:06
me stupid 
Title: AAC @ 128kbps listening test discussion
Post by: DeepDose on 2004-04-26 18:00:17
I encode my audio cds with NERO mp4 @ 387kps....then listen with my 5.1...and thrash out to some insane sounding euphoric sound!