Is double-blind testing potentially biased?
Reply #17 – 2004-11-04 18:54:47
Thus, ABX cannot ever be used to prove that "A & B sound the same", or that "there is no difference between A & B", even if you restrict the statements to the subject being tested. So you can never prove that some codec is transparent by ABX testing. Odd, but true. [a href="index.php?act=findpost&pid=251898"][{POST_SNAPBACK}][/a] Lemme try this hypothesis on for size: If there really is no difference between A & B, then you're still going to get results where 12/16s or what not are produced. They'll be rare, but they will exist. These results, in the context of ABC/HR testing, will follow a predictable distribution down from the peak (representing transparency). Couldn't this be used to construct a mean rating, and couldn't an interval be defined around that to produce a noninferiority test? Additionally, wouldn't the standard deviation around that mean change as well (with a large enough sample size) if the encoding was not transparent? EDIT: now that I think about it more, all that it would seem is necessary to use is the proportion of correct responses, which is what I assume gaboo was talking about in the first place. I see how this could be used to derive a mean even for a bare-bones ABX test. [a href="index.php?act=findpost&pid=251912"][{POST_SNAPBACK}][/a] Axon, I'm an engineer, thus no expert on perceptual testing. The way I see it, the main problem in setting up a non-inferiority perceptual test is exactly the fact that the effort required to get a "1" bit (i.e. different) is greater than the effort to get a "0" bit. You can get a bunch of deaf people to declare than any two encoders are equal. Maybe ff123 (who read at least one relevant book) has an idea how do this... I do know a Ph.D. candidate in sociology, I'll ask for outside help if all else fails.