Probability of passing a sequencial ABX test
Reply #27 – 2003-11-10 19:14:12
This is not. Actually these results make perfect sense. By guessing, you might very well guess two thirds of the trials correctly if you only do a few. But it's extremely improbable that you can maintain this two-thirds-streak for like 100 trials, if you really are only guessing. Conversely, if someone really manages to get two thirds right for 100 trials, you can be pretty sure he heard a difference. Good point. But it clearly means that we had to take care with pval. For exemple, when KikeG said that he would't trust (too much) pval > 0.05, this mean that if people want to convice him, it's better to send him a 30/45 than a 10/15. Or, differently, if you have difficulties to maintain a good concentration and achieve good ABX score on 16 trials, better than performing another test, you should resume the first one, and reaching the 45...50 trials . It supposes of course that the tester is able to maintain the two third right on 50 trials. I'm sure that I could do it with some difficult samples : when 16/16 is strictly impossible, 30/45 isn't too difficult (not for ABXing Flac & PCM of course ;-)). I often "failed" on ABX tests : I did three, four or five different sessions of 16 trials, and all were 11/16 or 12/16. If I had decided to merge the small tests in one big 60 trials test, conclusion would change, from "failed" to "succeed". I'm agree with your first comment ("there's no need to be all that dogmatic about the issue of "when can a test be considered meaningful?"). ABX score are nothing without precise comments about conditions of the test. For exemple, I often had 10/12 tests on anchor-like encodings, but 12/12 for high quality lossy encodings. The first is so easy that I need 30 seconds for 12 trials (and doing stupid mistake - sometimes with keyboard shortcuts), and the second is so hard that I need 15 minutes to perform it, taking "breaks" in order to keep some fresh ears.