Yep he gave up & stopped listening (but he never owned up to this on AVS) - the test was over at that point - probably trial 4, certainly trial 5. Yet he presented this set of results without mention that he stopped listening.

I've reproduced the ABX log along with the trial timings beside each trial

Looks to me he tried 21 seconds and thought to have found a difference but didn't. He tried 60 seconds on the next try. After that he lost interest or gave up because he was not sure.
How on earth does this discredit abx testing?

The one test that was claimed to take 1 second is probably an artifact of the 1 second resolution of the time stamps and the fact that it actually took more like 2 seconds.  Just because I may have blown off one trial doesn't mean that I necessarily blew off the whole test.

Quote from: jkeny on 2015-02-03 01:59:22

Yep he gave up & stopped listening (but he never owned up to this on AVS) - the test was over at that point - probably trial 4, certainly trial 5. Yet he presented this set of results without mention that he stopped listening. Yes, honesty is needed in these tests but that's a fairly shaky foundation on which to judge the results (as we've seen on this thread)

Whose trying to discredit ABX testing? I'm saying we could do with a measure of the false negatives in the results - in Arny's trial 5 to 15, a cannon shot could have been in one sample & a humming bird in another  he wouldn't have discriminated the difference due to not listening.

False negative? I see a perfect negative abx test of Arny. I guess he is used to 16 trials so he finished it.

Quote from: jkeny on 2015-02-03 01:59:22

Yep he gave up & stopped listening (but he never owned up to this on AVS) - the test was over at that point - probably trial 4, certainly trial 5. Yet he presented this set of results without mention that he stopped listening.

Wait, so if I don't think I can hear a difference after a few trials, I'm supposed to tick off some check box at the bottom of the test, which I can't seem to find at this moment, and my 8 correct out of 16 trials isn't enough for the test conductors to figure this out for themselves? Did I get this right, or again am I missing something?

Hmm, so let's recap the recent activity here:

1) Man who sells $400-600 Euro DAC assemblies claims that "false negatives" (which can be parsed a number of ways; type is conditionally dependent) are a serious problem in audio testing; man does not want to accept ABX style testing, but appears to offer no reasonable alternative (sighted trials being an inherently unreasonable alternative)

2) Individuals who have engaged in extensive ABX testing of hardware and/or encoders dispute claims of man in 1)

3) Man from 1) asserts a belief that "Type II" errors (again, must be specifically parsed) are a significant defect associated with ABX-style testing

4) Man from 1) presents no significant evidence whatsoever to validate assertion of belief from 3)

5) Forum users who disagree with assertions made by Man from 1) are accused by Man from 1) of being dishonest, and told that the onus is upon them to disprove his apparently baseless contentions.

So you suggest that the correct way to do an ABX test is like Arny did - stop listening after 4 trials

Type I and Type II error are standard statistical terms and shouldn't need special parsing.

I'm so tired of seeing old argument brandished as if they were brand new and cutting.  Old heads may recall that Les Leventhal was hammering on Type II erros way back in the mid 80s, and AFAIK Stereophile still trots his objections out every so often as part of their longstanding campaign to impugn audio DBT  (and ABX specifically) by any means necessary

They reprinted this 1985 exchange about Type II error in ABX in 2000:
http://www.stereophile.com/features/141/index.html

Over time , Leventhal published several articles about listening test stats in JAES or AES conventions, some more controversial than others

How Conventional Statistical Analyses Can Prevent Finding Audible Differences in Listening Tests
Author: Leventhal, Les
Affiliation: University of Manitoba, Winnipeg, Manitoba, Canada
AES Convention:79 (October 1985) Paper Number:2275
Publication Date:October 1, 1985

Type 1 and Type 2 Errors in the Statistical Analysis of Listening Tests
AES Volume 34 Issue 6 pp. 437-453; June 1986
(followed by two corrections published the same year in JAES,  and comments by David Clark, Tom  Nousaine, and Daniel Shanefield)


Statistically Significant Poor Performance in Listening Tests
JAES Volume 42 Issue 7/8 pp. 585-587;
Publication Date:July 1, 1994


Analyzing Listening Tests with the Directional Two-Tailed Test
Authors: Leventhal, Les; Huynh, Cam-Loi
JAES Volume 44 Issue 10 pp. 850-863; October 1996
Publication Date:October 1, 1996

Quote from: jkeny on 2015-02-03 12:17:43

So you suggest that the correct way to do an ABX test is like Arny did - stop listening after 4 trials

Why not? He couldn't tell them apart. Why wasting lifetime better spent elsewhere?

Where are exactly your positive abx logs of the same samples?

Quote from: krabapple on 2015-02-03 05:41:26

Type I and Type II error are standard statistical terms and shouldn't need special parsing.

I'm so tired of seeing old argument brandished as if they were brand new and cutting.  Old heads may recall that Les Leventhal was hammering on Type II erros way back in the mid 80s, and AFAIK Stereophile still trots his objections out every so often as part of their longstanding campaign to impugn audio DBT  (and ABX specifically) by any means necessary

They reprinted this 1985 exchange about Type II error in ABX in 2000:
http://www.stereophile.com/features/141/index.html

Over time , Leventhal published several articles about listening test stats in JAES or AES conventions, some more controversial than others

How Conventional Statistical Analyses Can Prevent Finding Audible Differences in Listening Tests
Author: Leventhal, Les
Affiliation: University of Manitoba, Winnipeg, Manitoba, Canada
AES Convention:79 (October 1985) Paper Number:2275
Publication Date:October 1, 1985

Type 1 and Type 2 Errors in the Statistical Analysis of Listening Tests
AES Volume 34 Issue 6 pp. 437-453; June 1986
(followed by two corrections published the same year in JAES,  and comments by David Clark, Tom  Nousaine, and Daniel Shanefield)


Statistically Significant Poor Performance in Listening Tests
JAES Volume 42 Issue 7/8 pp. 585-587;
Publication Date:July 1, 1994


Analyzing Listening Tests with the Directional Two-Tailed Test
Authors: Leventhal, Les; Huynh, Cam-Loi
JAES Volume 44 Issue 10 pp. 850-863; October 1996
Publication Date:October 1, 1996

Thank you for those links - much appreciated.

I'm not just identifying the problem - I'm also offering the solution!

It seems like it's both not accepted that there is a problem & therefore the solution is not even discussed

There are positive ABX results on that forum if you care to look - so that shows Arny's reults to be false negatives (although the fact that he stopped listening is proof enough)

Quote from: jkeny on 2015-02-03 12:48:03

There are positive ABX results on that forum if you care to look - so that shows Arny's reults to be false negatives (although the fact that he stopped listening is proof enough)

We are going in circles. Arny couldn't hear a difference so it is not a false negative. You suggest that if one positive was done all negatives must be false?

I would think that a person who has made such far reaching claims related to the topic would already know of these articles and have studied them. I know that I have.

Quote

It seems like it's both not accepted that there is a problem & therefore the solution is not even discussed

Speaks to a less-than-adequate familiarity with the topic, particularly for a person who seems to seek to correct practitioners with decades of hands-on experience.

(1) Type 1 and Type 2 errors are  constant dangers in any experiment.

(2) Even though sighted evaluations  are usually utterly corrupted and overwhelmed by type 1 errors, the pervasive type 1 errors may hide type 2 errors.

(3) Controlling type 1 errors can be reasonably expected to make type 2 errors more apparent and thus facilitate their management. I think this has happened quite a bit with ABX, for example.

(4) Nobody with a brain tries to rank their importance because they are both potentially fatal errors.  Harping on type 2 errors while glossing over type 1 errors or vice versa is unwise.

(5) Mr. Keny's hobby seems to be going ballistic over type 2 errors and glossing over type 1 errors.

(6) Figuring what these and other errors are in the context of a particular experiment may be very difficult. Its not as trivial as Mr. Keny makes it seem with his wild unfounded accusations.

Quote from: jkeny on 2015-02-03 12:48:03

There are positive ABX results on that forum if you care to look - so that shows Arny's reults to be false negatives (although the fact that he stopped listening is proof enough)

We are going in circles. Arny couldn't hear a difference so it is not a false negative. You suggest that if one positive was done all negatives must be false?

Of course it is - anybody who deliberately stops listening, as you maintain, should be eliminated from the test & their results discarded. Read up on how to do these tests or would you prefer to make up your own guidelines as you go along - seems to be the MO here

man wants ABX testing to include internal controls & has suggested examples of these controls & how they would be used.

Man is suggesting a way to improve ABX testing

Quote from: jkeny on 2015-02-03 14:18:46

Of course it is - anybody who deliberately stops listening, as you maintain, should be eliminated from the test & their results discarded. Read up on how to do these tests or would you prefer to make up your own guidelines as you go along - seems to be the MO here

Own rules? Where is the rule to listen at least for a specific time when i realize i hear no difference? Is there a minimum fail time to do a valid fail?
I give up.