History and Accreditation of ABX Testing/Invention?
Reply #46 – 2014-12-08 14:59:06
I have to point out that the "abx" test of Munson was a sequential test, not a time-proximate test in which the subject had control over stimulus selection at all points in time. Hi JJ. Thanks for chiming in. What you describe was very clear in the original abstract from ASA. I think giving control to the user a variation of such a test. It is not an all new invention in my opinion.This test has been brought up from time to time, and various people have taken upon themselves to scold the entire AES about how this sequential test is such a bad test (as it is, the lack of time proximity is massively desensitizing), ignoring the fact that this is NOT the modern test called "ABX". The ABX test from the Michigan bunch was time-proximate. This is a substantial improvement, and is what is commonly referred to in the present day as the ABX test. There may still be an issue of silences inserted, or clicks. It's a difficult problem to solve with 1960's technology, but it's at least mostly time-proximate. If there is no source control for segment selection and looping, I don't think adding user control gets us to where we want to be. Yes, it is better than timed switching. But one can't compare the past to the future as was done with the ABX comparator itself. Modern ABX tests using windowed digital switching are even better, of course. They certainly are a huge improvement over the real-time versions with just the output switching. I think you agree that the less we are "time proximate" the more the outcome tilts toward negative outcome. User gets frustrated that he can't recall the the other track and will vote randomly or "always A." Much of an ABX test, however, comes in the material selection, listener training, listener comfort, and presence of listener feedback. A failing unfortunately for just about every test cited on audio forums as proof of this and that. The outcomes may still be right but the protocol certainly not. I would also add to your list poor user interface. I find foobar ABX plug-in woefully inadequate for finding small differences. Its segment selection interface is horrid on full track for example. Lack of recall for the segments once you find them makes it worse. All of this again tilts the odds toward the "house" by generating negative outcomes. The flaws are not material if we are talking about gross distortions that are audible to masses. But the moment we start to say that it applies to any and all people, I think we are being improper and unfair. So in that sense, the proper ABX test for finding small differences is yet to be invented. In MPEG as you know we have our list of codec killers. Where is that list for jitter over HDMI? Where is that list for differences between amplifiers? The people who obsess over double blind ABX tests are seemingly very comfortable with negative outcomes so there is no desire to find the scenarios that are most revealing. The people who believe there are differences don't care about ABX testing, nor have the necessary knowledge. Net, net, when these tests are applied in forums and among hobbyists, I believe it is a biased game aimed to produce one outcome more than the other. Culturally we need to make it OK and "safe" in corporate speak to criticize the tests and their outcomes. Only then can we create the much more sensitive and defensible results we desire when differences get small.