Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: (AAC) advantages & disadvantages (Read 53212 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

(AAC) advantages & disadvantages

Reply #50 –
I'm just using Foobar's standard procedure. I understand that you need a certain procedure to be able to compare various results and be able to put them into perspective. But please tell me the probability of hitting 0.8% in <= 22 rounds in just one trial (not multiple!) based on nothing but pure guessing. I expect that I could live with that number. As said, limiting the number of rounds is one measure to put results into perspective. Analysing the tests record is another (error patterns, time taken, pauses made). Regarding the trustworthiness of ABX results in a public forum, it would be far easier to fake a 8/8 run, than play endlessly a wild guess game until you like the results.

Edit/PS:

People doing these tests at home are different than people taking part in a study. I know for myself that cheating on my own ABX tests would be a senseless waste of time. So when I'm doing this in my free time and know for a fact that a sample is so laughably easy, that I could extrapolate a 8/8 run to a 30/30 (I have never gone above 30), then I take this short circuit without worrying much about it. With harder samples as the 320 CBR I would go all the way to 30 at max instead. But as I am writing this, I realize it would be better if I hadn't stopped after 22 but had really gone all the way to 30. The results were too shaky to extrapolate them. So I think in the future I will use a virtual 30/30 policy. When there is at least one mismatch, I'll go all the way to 30, no less, no more. With truly easy content, I will continue to allow myself the extrapolation of 8/8 results, though.

Thanks for the hint! At first, I was in a kind of opposed state to your criticism, because I knew I didn't cherry pick. But it really has some sense to it now that I have reflected about it.

(AAC) advantages & disadvantages

Reply #51 –
Quote from: rpp3po on
I'm just using Foobar's standard procedure.


Yes but you're using it incorrectly (not to single you out, C.R.Helmrich did the same thing as you).

Generally people choose a larger number of tests (say 15 trials) and do that many.  Then if after 8 trials if they are 8/8 terminate early, noting that they stopped the trial early.  What you cannot do is pick a small number of trials and extend the number.  In your lame 320 CBR trial, after 8 tests you failed to obtain statistical significance, and thus the correct thing to do would be declare you could not ABX the difference in that trial.  You did not do this, and thus failed to properly conduct your test. 

Of course if you thought you could do better you could always do a second trial, deciding in advance to do 20 rounds (or however many you prefer).  But you have to mention that you failed the first trial before restarting the second . . .

Quote from: rpp3po on
Analysing the tests record is another (error patterns, time taken, pauses made). Regarding the trustworthiness of ABX results in a public forum, it would be far easier to fake a 8/8 run, than play endlessly a wild guess game until you like the results.


No one is saying that people here are faking the results, the problem here is presenting results that are believed correct but in fact are compromised due to poor procedure.  If you want meaningful results, you must do the tests correctly.  A broken test is no better then a false test, whatever the author's intention. 

Quote from: rpp3po on
Thanks for the hint! At first, I was in a kind of opposed state to your criticism, because I knew I didn't cherry pick.


But you did cherry pick good results, otherwise every trial would have been 8 tests as you originally intended!  The fact that you didn't mean to do it doesn't change the fact that you did do it.

(AAC) advantages & disadvantages

Reply #52 –
Hmm, yes, I just clicked Utils -> ABX two files -> OK -> listen -> press some other buttons  That automatically shows whether you guessed right after each trial. Agreed that when failing a trial, this totally influences your concentration and decision of how a) how long you rest between trials, b) how many trials you take. Maybe the plug-in developers should turn this off by default?

Having said that, I ran yesterday's nero CBR 192 kbps ABX test again, this time with 8 trials, like my other two tests. I got 7/8 (I'll spare you the code). Took a major pause of a few minutes after my failed trial # 4. By the way, does taking such pauses in between trials violate the ABX procedure?

Mike:

Quote
But you've already explained that those results don't support or even test your hypothesis, so why did you recommend I read them exactly?

Don't know what you mean by that. Anyway, going back to my hypothesis, which was:

Quote
1. Most likely, these items will not be transparent for MP3 at any bitrate up to and including 320 kbps.
2. And most likely, they will be transparent for AAC at 320 kbps or even less.
3. That's still the case with today's codecs.
Part 1 I proved to myself. To test part two of my hypothesis, I just conducted a foobar ABX as recommended by you and Soap. Unfortunately, hiding the results also hides the number of trials, and I failed miserably in trying to remember what trial I'm in  Wanted to do only 8. So, I'm unable to ABX nero CBR 256 kbps reliably. Since I already had that problem once with nero CBR 192 kbps, I conclude that a) nero 256 kbps is very close to transparency to my ears, b) nero 320 kbps will be transparent to my ears, so I proved Part 2 to myself.

Part 3 is not fully proven, since I only auditioned two modern (and apparently well-tuned) codecs.

Code: [Select]
foo_abx 1.3.4 report
foobar2000 v0.9.6.7
2009/05/31 10:32:06

File A: C:\Users\Christian\Desktop\40_original.wav
File B: C:\Users\Christian\Desktop\40_nero256kb.mp4

10:32:06 : Test started.
10:32:41 : 01/01  50.0%
10:32:59 : 01/02  75.0%
10:33:24 : 01/03  87.5%
10:33:43 : 02/04  68.8%
10:34:09 : 03/05  50.0%
10:37:24 : 04/06  34.4%
10:37:40 : 05/07  22.7%
10:38:06 : 06/08  14.5%
10:38:33 : 06/09  25.4%
10:38:59 : 07/10  17.2%
10:39:00 : Test finished.

 ---------- 
Total: 7/10 (17.2%)

Chris
If I don't reply to your reply, it means I agree with you.

(AAC) advantages & disadvantages

Reply #53 –
Quote from: Mike Giacomelli on
But you did cherry pick good results, otherwise every trial would have been 8 tests as you originally intended!


Maybe I didn't make myself clear enough. I did not originally intend to do 8 only. 8 (or 7 depending on the weather) is my early exit for trivial comparisons. 30 is my usual ceiling. Is anybody able to compute the probability of hitting 0.8% anywhere in a max 30 round run of random results?

Or what is the formula used for calculating Foobar's probability value. I'll do it myself then.

Quote from: Mike Giacomelli on
A broken test is no better then a false test, whatever the author's intention.


Nice generalization, how does it apply here?

This is the result of my 320 CBR run, this is a random result, 22 "X is A" clicks.

A child would see the difference. So no, a test that you consider broken, can be better (contain significant data) than a false test. Anything else is useless black & white extremism.

PS Can't embed Imageshack images anymore to post plots. Does anybody know an alternative?

(AAC) advantages & disadvantages

Reply #54 –
I have just noticed, that the axis descriptions in above images are inverted.

Anyway, I have already agreed that a test's accuracy can be improved by always doing a full set of rounds and will consider that in the future.

(AAC) advantages & disadvantages

Reply #55 –
Just finished half an hour of trying to infallibly identify an artifact in the Quicktime 128 CBR AAC. I failed. For me it is about as hard or maybe even harder than the LAME 320 CBR MP3. I trained about 10 minutes to get used to it (range 0:00.4-0:01.6). Again one sample seems to have its highs clanking somewhat and then tried 30 rounds in a row:

Code: [Select]
foo_abx 1.3.3 report
foobar2000 v0.9.6.2
2009/05/31 17:00:49

File A: C:\Dokumente und Einstellungen\Administrator\Desktop\ABX\tec_sqam_40a_bwf_tcm6-12548.wav
File B: C:\Dokumente und Einstellungen\Administrator\Desktop\QCBR128-tec_sqam_40a_bwf_tcm6-12548.wav

17:00:49 : Test started.
17:01:24 : 01/01  50.0%
17:01:52 : 02/02  25.0%
17:02:26 : 03/03  12.5%
17:02:44 : 04/04  6.3%
17:03:23 : 05/05  3.1%
17:03:48 : 06/06  1.6%
17:04:20 : 06/07  6.3%
17:05:00 : 07/08  3.5%
17:05:18 : 08/09  2.0%
17:05:50 : 08/10  5.5%
17:06:22 : 09/11  3.3%
17:06:35 : 09/12  7.3%
17:07:06 : 10/13  4.6%
17:07:52 : 10/14  9.0%
17:08:12 : 11/15  5.9%
17:08:43 : 12/16  3.8%
17:09:00 : 12/17  7.2%
17:10:13 : 13/18  4.8%
17:10:35 : 13/19  8.4%
17:11:18 : 14/20  5.8%
17:11:47 : 15/21  3.9%
17:13:05 : 16/22  2.6%
17:13:21 : 17/23  1.7%
17:14:09 : 17/24  3.2%
17:14:51 : 17/25  5.4%
17:15:52 : 17/26  8.4%
17:16:43 : 18/27  6.1%
17:17:14 : 19/28  4.4%
17:17:51 : 20/29  3.1%
17:19:00 : 20/30  4.9%
17:19:03 : Test finished.

 ---------- 
Total: 20/30 (4.9%)

The CBR 192 and VBR encodes were both close to 50% guessing after 8 tries. Not worth taking further.

So for this sample Quicktime AAC, the first (comparably bad) release was 2002, beats LAME MP3, released 1998. I just won't believe that the difference between MP3 and AAC is just a matter of encoder tuning. There has been enough time to tune and LAME still fails a prominent natural instrument sample. Locating the cause in ithe MP3 format's limitations is just more plausible. Do you know any sample where MP3 whoops AAC's ass in comparable fashion for up to date encoders?

(AAC) advantages & disadvantages

Reply #56 –
Quote from: rpp3po on
Do you know any sample where MP3 whoops AAC's ass in comparable fashion for up to date encoders?

I don't. If someone knows one, I would really like to know too.

Chris
If I don't reply to your reply, it means I agree with you.

(AAC) advantages & disadvantages

Reply #57 –
Quote from: C.R.Helmrich on
Quote from: rpp3po on
Do you know any sample where MP3 whoops AAC's ass in comparable fashion for up to date encoders?

I don't. If someone knows one, I would really like to know too.

Probably because there is absolutely no feature of the AAC format which is better handled within the MP3 format. If someone find a sample which is better handled by mp3 encoders than by aac encoders, it would likely be a matter of implementation, and not a matter of format.

note: after a bit of thinking, I'm not sure if AAC supports mixed blocks. So perhaps if you create a test sample specially designed to demonstrate mixed blocks efficiency, you might have such a sample where mp3 is better than aac. Of course, you'll first have to find an mp3 encoder able to produce mixed blocks (Uzura3).

Btw, I am wondering why is there still some discussion to know if aac is a better format than mp3 or not. AAC-LC is simply what MP3 should have been if it was properly designed.

(AAC) advantages & disadvantages

Reply #58 –
Quote from: Gabriel on
note: after a bit of thinking, I'm not sure if AAC supports mixed blocks.

The standard certainly does. Look for the "common information" flag in MPEG-2 AAC in the ICS header.
Quote
Btw, I am wondering why is there still some discussion to know if aac is a better format than mp3 or not. AAC-LC is simply what MP3 should have been if it was properly designed.


Well, the use of a long MDCT was blocked by the MUSICAM group in MPEG-1. The accomodation of good stereo coding was, likewise, deemed "too risky".  Finally, don't get me started on the MP3 bitstream, I hated it then and I hate it now. I'm not real keen on the MPEG 4 bitstream, either. So it goes.
-----
J. D. (jj) Johnston

(AAC) advantages & disadvantages

Reply #59 –
Quote from: Gabriel on
Btw, I am wondering why is there still some discussion to know if aac is a better format than mp3 or not. AAC-LC is simply what MP3 should have been if it was properly designed.

Agreed. It seems that despite Apple's and nero's free AAC encoders, articles like the one on Wikipedia, and comments like your's and JJ's, this information has not reached the "average Jane and Joe" (hope I translated that correctly from German). When I tell people I work on "something called AAC, that's the successor to MP3", they say, "Cool! When will it be released?"

MP3 really has reached cult status, arguably in part (and that's my personal opinion) because of the excellent work of the LAME developers.

Chris
If I don't reply to your reply, it means I agree with you.

(AAC) advantages & disadvantages

Reply #60 –
Quote from: Woodinville on
Quote from: Gabriel on
note: after a bit of thinking, I'm not sure if AAC supports mixed blocks.

The standard certainly does. Look for the "common information" flag in MPEG-2 AAC in the ICS header.

I'm not sure if you talk about the flag for sharing common side info between 2 channels or the short window grouping, the former is nothing like mixed blocks in MP3, the latter is similar but not the same.
Short block grouping works in the time domain by grouping several short blocks and have only 1 set of side information for the whole group, mixed blocks in MP3 work in the frequency domain by handling the lower frequencies as long blocks and the higher ones as short (this is only possible thanks to the subband filterbank).

(AAC) advantages & disadvantages

Reply #61 –
Quote from: menno on
Short block grouping works in the time domain by grouping several short blocks and have only 1 set of side information for the whole group, mixed blocks in MP3 work in the frequency domain by handling the lower frequencies as long blocks and the higher ones as short (this is only possible thanks to the subband filterbank).


Indeed, I'd forgotten about that mode in MP3. Has anyone ever figured out how to use it productively?
-----
J. D. (jj) Johnston

(AAC) advantages & disadvantages

Reply #62 –
Quote from: Woodinville on
Indeed, I'd forgotten about that mode in MP3. Has anyone ever figured out how to use it productively?

Not that I know. If I remember correctly what some MP3 developers from Fraunhofer told me, some decoders don't even support this.

Chris
If I don't reply to your reply, it means I agree with you.

(AAC) advantages & disadvantages

Reply #63 –
Quote from: Woodinville on
Indeed, I'd forgotten about that mode in MP3. Has anyone ever figured out how to use it productively?

I think that no one really tried, as there is an interesting feature with mixed blocks: you are only allowed to do start-mixed-stop transitions, but mixed-short is not allowed. Thus if you want to use a mixed blocks in a frame you'd better be sure that you won't need to use short blocks within the following ones (ie you need at least 2 frames of look-ahead in order to be able to use this)

(AAC) advantages & disadvantages

Reply #64 –
Interesting. The more I think of this: Wouldn't MDCTs of different lengths in different QMF bands lead to additional aliasing artifacts between those differing MDCTs?

Chris
If I don't reply to your reply, it means I agree with you.

(AAC) advantages & disadvantages

Reply #65 –
Quote from: C.R.Helmrich on
Interesting. The more I think of this: Wouldn't MDCTs of different lengths in different QMF bands lead to additional aliasing artifacts between those differing MDCTs?

Chris



Well, it would make the anti-aliasing twiddles already written somewhat (read rather) complex. I'm not even sure if they are defined for this case.
-----
J. D. (jj) Johnston

(AAC) advantages & disadvantages

Reply #66 –
Quote from: Woodinville on
Quote from: C.R.Helmrich on
Interesting. The more I think of this: Wouldn't MDCTs of different lengths in different QMF bands lead to additional aliasing artifacts between those differing MDCTs?

Chris



Well, it would make the anti-aliasing twiddles already written somewhat (read rather) complex. I'm not even sure if they are defined for this case.


The original reference decoder doesn't do any anti-aliasing on mixed blocks, the standard text is unclear about it. Currently the reference decoder has a flag to enable it at wish. Anti-aliasing, when enabled, is then only done between the first and second subband (2 subbands are used for long MDCT's).

(AAC) advantages & disadvantages

Reply #67 –
Quote from: Gabriel on
Quote from: C.R.Helmrich on
Quote from: rpp3po on
Do you know any sample where MP3 whoops AAC's ass in comparable fashion for up to date encoders?

I don't. If someone knows one, I would really like to know too.

Probably because there is absolutely no feature of the AAC format which is better handled within the MP3 format. If someone find a sample which is better handled by mp3 encoders than by aac encoders, it would likely be a matter of implementation, and not a matter of format.

note: after a bit of thinking, I'm not sure if AAC supports mixed blocks. So perhaps if you create a test sample specially designed to demonstrate mixed blocks efficiency, you might have such a sample where mp3 is better than aac. Of course, you'll first have to find an mp3 encoder able to produce mixed blocks (Uzura3).

Btw, I am wondering why is there still some discussion to know if aac is a better format than mp3 or not. AAC-LC is simply what MP3 should have been if it was properly designed.


Thank you. End of story. I don't care if these guys who argue for argument's sake keep arguing, but as far as I am concerned I have gotten my answer.

(AAC) advantages & disadvantages

Reply #68 –
Quote from: menno on
Quote from: Woodinville on
Quote from: C.R.Helmrich on
Interesting. The more I think of this: Wouldn't MDCTs of different lengths in different QMF bands lead to additional aliasing artifacts between those differing MDCTs?

Chris



Well, it would make the anti-aliasing twiddles already written somewhat (read rather) complex. I'm not even sure if they are defined for this case.


The original reference decoder doesn't do any anti-aliasing on mixed blocks, the standard text is unclear about it. Currently the reference decoder has a flag to enable it at wish. Anti-aliasing, when enabled, is then only done between the first and second subband (2 subbands are used for long MDCT's).


For what its worth, I think most decoder do apply anti-aliasing.  At least I know libmad does unless you change the preprocessor defines.  I've never listened to see if it makes any audible difference.