Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How do you listen to an ABX test? (Read 352317 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

How do you listen to an ABX test?

Reply #1300 –
If audiophiles are able to pass tests for known small audible differences, and hear differences between A and B sighted but then fail when it comes to X under the same conditions, then I'm not sure what else could be done.

It's not only dishonest but again hypocritical to hold blind testing to impossible standards, but give sighted listening a free pass.


Btw, Gecko asked you some good question above.
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1301 –
Quote from: jkeny on
Sorry but I don't see pre-screening & pre-training as sufficient.


Would anything short of brainwashing satisfy you?

The most important audio cables are the ones in the brain
 

How do you listen to an ABX test?

Reply #1302 –
Quote from: xnor on
If audiophiles are able to pass tests for known small audible differences, and hear differences between A and B sighted but then fail when it comes to X under the same conditions, then I'm not sure what else could be done.

It's not only dishonest but again hypocritical to hold blind testing to impossible standards, but give sighted listening a free pass.


Btw, Gecko asked you some good question above.

Who said I give sighted listening a free pass? I am very sceptical of others sighted listening results for the same reasons - I don't know the quality of the results. I will listen to those I trust (those that I have found to hear what I hear) & if I find it interesting, I will then personally test it myself. I prefer the type of report where examples of specific music & what is heard are given - I don't always hear this & so, for me, that result does not hold.

So sighted listening reports are not trusted - they are personally verified.

I don't see this as holding ABX testing to a higher standard - I'm talking about ABX because it has no internal controls that allow me to have some measure of the quality of the results (same as sighted results). I see the typically run ABX tests as flawed just as much as I see the typical meet-up sighted tests as flawed. They are interesting social events (the sighted ones often more interesting )

Oh, I missed Gecko's post

How do you listen to an ABX test?

Reply #1303 –
Quote from: Gecko on
Quote from: jkeny on
[...]
Quote
A major consideration is the inclusion of appropriate control conditions. Typically, control conditions include the
presentation of unimpaired audio materials, introduced in ways that are unpredictable to the subjects. It is the differences
between judgement of these control stimuli and the potentially impaired ones that allows one to conclude that the grades
are actual assessments of the impairments.

[...]
Just to note - the BS.1116 is strongly recommending the use of such controls in formal testing to validate the test. It would seem to me that there is an even greater need for such controls in the less formal environment of home tests

In an ABX test, the "unimpaired" sample is presented as X in an unpredictable way during an a priori unknown number of trials. How about that?
I'm sorry but I don't follow

Quote
On a different note: I fail to see how the controls could be designed without leaving a backdoor for the usual placebophile criticism.
Let's say I wanted to test the transparency of a 256k MP3 for a specific sample. I have read and heard many audiophiles claim that MP3 can never be transparent (at any bitrate). I myself make no such claim and will supply the null-hypothesis: the MP3 is audibly indistinguishable from the original. The alternative hypothesis is of course, that the MP3 is indeed audibly distinguishable from the original. Experts, please correct me If I'm stating this wrong.

jkeny, please suggest a suitable control which will convince audiophiles that the null-hypothesis has credibly not been rejected. In a sort of role reversal, I will try to discredit your suggestion using oft-heard audiophile excuses.

I see you posted this before I made this post
"And it may not be possible to implement these same controls in an ABX test - without disrupting it in a negative way - I accept that now.
I initially thought it would be possible but you have convinced me that I'm most likely wrong."

So I guess this answers your question - internal controls of the type called "catch trials" don't seem to be feasible in ABX testing or, at least, I can't see a way of implementing them which I initially thought possible

How do you listen to an ABX test?

Reply #1304 –
Quote from: jkeny on
Who said I give sighted listening a free pass? I am very sceptical of others sighted listening results for the same reasons - I don't know the quality of the results. I will listen to those I trust (those that I have found to hear what I hear) & if I find it interesting, I will then personally test it myself. I prefer the type of report where examples of specific music & what is heard are given - I don't always hear this & so, for me, that result does not hold.

So sighted listening reports are not trusted - they are personally verified.


You can't make this shit up. Seriously.

Sometimes I wish Arny was Justin and 2bdecided was Mr Dunning. 
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1305 –
Quote from: jkeny on
Who said I give sighted listening a free pass?


Jphn, you do it right here:

Quote
I am very skeptical of others sighted listening results for the same reasons - I don't know the quality of the results. I will listen to those I trust (those that I have found to hear what I hear) & if I find it interesting, I will then personally test it myself. I prefer the type of report where examples of specific music & what is heard are given - I don't always hear this & so, for me, that result does not hold.


It has been conclusively proven that casual sighted listening evaluations are 100% susceptible to false positives and false negatives, and yet you persist in say all of the above?

Please tell me how to quantify the false positive and false negatives in a sighted listening evaluation.  I want peer reviewed scholarly articles, and math on the same grade as we have for DBTs.

How do you listen to an ABX test?

Reply #1306 –
Quote from: jkeny on
I may have used bad examples or got some things wrong in this thread but I'm still left with this large question mark about ABX results - I just have no way of knowing how sensitive that particular ABX test was & therefore I have no way of judging the validity/meaning of it's results.

It is a quest for results that reject the null hypothesis and pursue further inquiry once they have been identified in order to glean something useful.

Those who are earnest may decide to continue searching through multiple haystacks in order to fine that elusive needle....

...and they may just find it.


Others may consider such endeavors futile...

...and some of these endeavors may very well be futile.

How do you listen to an ABX test?

Reply #1307 –
That must be the nicest way of describing confirmation bias. At the same time it completely ignores the problem of prejudice, biases (everything we've talked about) during listening and why we're even doing blind tests.

... wow.


"I hear it when I see it."

How do you listen to an ABX test?

Reply #1308 –
Quote from: Arnold B. Krueger on
Quote from: jkeny on
Who said I give sighted listening a free pass?


Jphn, you do it right here:

Quote
I am very skeptical of others sighted listening results for the same reasons - I don't know the quality of the results. I will listen to those I trust (those that I have found to hear what I hear) & if I find it interesting, I will then personally test it myself. I prefer the type of report where examples of specific music & what is heard are given - I don't always hear this & so, for me, that result does not hold.


It has been conclusively proven that casual sighted listening evaluations are 100% susceptible to false positives and false negatives, and yet you persist in say all of the above?

Please tell me how to quantify the false positive and false negatives in a sighted listening evaluation.  I want peer reviewed scholarly articles, and math on the same grade as we have for DBTs.

I don't consider that a free pass. A free pass would be accepting sighted tests as always correct.

"casual sighted listening evaluations are 100% susceptible to false positives" doesn't mean that all sighted tests therefore give the wrong result. As I said if I'm interested, I check it myself. So I have verified to my own satisfaction that sighted results can be correct

How do you listen to an ABX test?

Reply #1309 –
Quote from: xnor on
That must be the nicest way of describing confirmation bias. At the same time it completely ignores the problem of prejudice, biases (everything we've talked about) and why we're even doing blind tests.

... wow.

But again, not all biases are controlled by doing a blind test & hence I find blind tests without some sort of internal controls an open question as to the validity of their results

How do you listen to an ABX test?

Reply #1310 –
jkeny, do you still not even understand why we do blind tests?
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1311 –
Quote from: jkeny on
I don't consider that a free pass. A free pass would be accepting sighted tests as always correct.

"casual sighted listening evaluations are 100% susceptible to false positives" doesn't mean that all sighted tests therefore give the wrong result.


Where are your scholarly papers to support that claim - on the same level as the ones you demanded for DBTs?

If you accept sighted tests with no theoretical support and in the face of universal criticism in scholarly circles, then you are indeed giving sighted evaluations a free pass.

How do you listen to an ABX test?

Reply #1312 –
Quote from: xnor on
jkeny, do you still not even understand why we do blind tests?

should I have said sensitivity of their results (to me it equates to the same outcome)

How do you listen to an ABX test?

Reply #1313 –
Quote from: Arnold B. Krueger on
Quote from: jkeny on
I don't consider that a free pass. A free pass would be accepting sighted tests as always correct.

"casual sighted listening evaluations are 100% susceptible to false positives" doesn't mean that all sighted tests therefore give the wrong result.


Where are your scholarly papers to support that claim - on the same level as the ones you demanded for DBTs?

If you accept sighted tests with no theoretical support and in the face of universal criticism in scholarly circles, then you are indeed giving sighted evaluations a free pass.

As I said, I personally check any I am interested in - so I don't accept other's sighted reports.
The thing is, Arny, most decisions in audio are made with sighted listening so I think you might have a problem arguing about their reliability. Most audio differences don't require blind testing.

How do you listen to an ABX test?

Reply #1314 –
You are completely tapping in the dark regarding audible differences....

But you "select" biased claims about audible differences that are not supported by a shred of evidence to agree with what basically is just your own biased opinion?

LOL.
aj was right, you couldn't make this shit up.
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1315 –
Quote from: xnor on
jkeny, do you still not even understand why we do blind tests?

It may seem that way, though I often think this assessment is being overly-charitable, since we keep repeating the why over and over and over and over.

That said, I still appreciate the concession that this community doesn't live and die by all things ABX.  Hopefully it sticks.

How do you listen to an ABX test?

Reply #1316 –
Quote from: xnor on
But you "select" biased claims about audible differences that are not supported by a shred of evidence to agree with what basically is just your own biased opinion?

...but they've always done it that way in audio, so it must be right!

How do you listen to an ABX test?

Reply #1317 –
Quote from: jkeny on
Quote from: Arnold B. Krueger on
Quote from: jkeny on
I don't consider that a free pass. A free pass would be accepting sighted tests as always correct.

"casual sighted listening evaluations are 100% susceptible to false positives" doesn't mean that all sighted tests therefore give the wrong result.


Where are your scholarly papers to support that claim - on the same level as the ones you demanded for DBTs?

If you accept sighted tests with no theoretical support and in the face of universal criticism in scholarly circles, then you are indeed giving sighted evaluations a free pass.

As I said, I personally check any I am interested in - so I don't accept other's sighted reports.


OK John so you are the only person you trust. Your trust is misplaced because the evaluations you trust are still horribly flawed even when it is you that do them.

Given all of the lies and errors that we have caught you in along the way John, why should we trust your listening tests, especially when we are aware of their inherent and serious flaws.

Quote
The thing is, Arny, most decisions in audio are made with sighted listening so I think you might have a problem arguing about their reliability. Most audio differences don't require blind testing.


I guess you never raised kids John, because that's one of the logically flawed arguments that kids like to use and no parent with brains is fooled by it.  When kids say it, they are saying that something must be right because it is what everybody does. That's called lowest denominator ethics and its no ethics at all. It is also horrible logic.

If some kid tells their parents that "All the other parents let their kids do it" the parent who didn't just fall off the turnip truck asks for the names of the parents who allegedly agreed with their kids, calls them up, and finds out that no way did all the parents agree with their kids.

The above answers are not even remotely answers to the questions that I asked. They prove John  that you can't admit that you have made any errors at all even though you are in the midst of making some very old and bad errors.

Don't belittle us or lie to us because we know better than you, John.  It is called Science and it could help you if you let it.

How do you listen to an ABX test?

Reply #1318 –
Quote from: jkeny on
As I said if I'm interested, I check it myself. So I have verified to my own satisfaction that sighted results can be correct


You'll never be able to discern between your perceptions and physical reality, nor understand the difference between sound and preference, as jj tried to explain many times to you on AVS. As I've said all along, your issue isn't with blind (now aka ABX) tests at all, despite all the chicanery.
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1319 –
Quote from: Arnold B. Krueger on
that's one of the logically flawed arguments that kids like to use

That's funny, because I often see it raised by middle-aged men who entrusted with the responsibility of running corporate operations.

How do you listen to an ABX test?

Reply #1320 –
Quote from: greynol on
Quote from: Arnold B. Krueger on
that's one of the logically flawed arguments that kids like to use

That's funny, because I often see it raised by middle-aged men who entrusted with the responsibility of running corporate operations.


They obviously lacked good parenting! ;-)

IME these are three biggest myths that children need to be disabused of:

(1) Its OK to do something because you perceive "Everybody else is doing it".

(2) The world will treat you fairly.

(3) All you need to do is treat the world like they treat you. (Yes, the so-called Golden Rule is a myth)

Reality is that Everybody else isn't doing it, and even if they were using other's actions as an ethical guide is what Lemmings do.

Reality is that you will be treated unfairly and need to suck it up and do better.

Reality is that you need to treat people well regardless of how shabbily they treat you.

How do you listen to an ABX test?

Reply #1321 –
OK, we seem to have reached a point where the situation is that, according to jkenny...

jkenny does not trust blind tests.

jkenny does not trust sighted tests except his own.

If that's how jkenny wants to go on in his audio life, that's fine: but why is he even bothering to discuss it, let alone argue it feverishly like a man with bottomless lungs.

One post would have done: "I don't trust your tests, I don't trust their tests, I only trust my own tests." Fine. Fifty pages and a lot of other people's time saved
The most important audio cables are the ones in the brain

How do you listen to an ABX test?

Reply #1322 –
Quote from: Arnold B. Krueger on
Quote
I know when I'm wasting my time.


I seriously doubt that!

You've replied to me here, thinking you were quoting jk. Again!

But in a general sense, I'm sure you're right anyway.

Cheers,
David.

How do you listen to an ABX test?

Reply #1323 –
@Thad: yup.

Btw, as a seller this behavior perfectly matches that of snake oil salesmen.

"I reject the science because it shows null results. But my magic bracelets work because I feel they work and so I know they work. Also look at others having similar experiences, except those who don't - I reject those anecdotes. But I know it works!"
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1324 –
Quote from: greynol on
Quote from: Arnold B. Krueger on
that's one of the logically flawed arguments that kids like to use

That's funny, because I often see it raised by middle-aged men who entrusted with the responsibility of running corporate operations.

... selling miracles, revelations and not to mention: eternal life - which you have offered absolutely nothing to disprove?