Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 248967 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

What's the problem with double-blind testing?

Reply #250 –
Quote from: greynol on
The point is that there is no reason to put unnecessary limits on what the test is capable of measuring, giving our recent "skeptic" something else to which he can cling.  That we're debating unfounded speculation about what may occur on a subconscious or unconscious level is bad enough.


Agreed.

It doesn't matter what the actual percept is (i.e. the actual thing, sensory, mental, whatever...) the person percieves that helps them distinguish A from B from X.

All that matters is that they do, or that they don't.

-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #251 –
Quote from: greynol on
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.
Like ajinfla I'm confused. Can you explain how an ABX test with acoustical stimuli can reveal "the other stuff" ? Aren't you mixing up ABX and DBT ?
I think ajinfla nailed it down accurately: "My interest is acoustics. I'll leave all the "other stuff" of interest to the psychologists." IMO this is pretty much the attitude on HA, which is perfectly understandable when your field is codecs. OTOH there's the unsatisfying sensation of "I shouldn't hear a difference, but I do" (non-blindly) and ignoring or ridiculing that phenomenon is also short-sighted. Desperately trying to find acoustical explanations (like subjectivists often do) is too though.

What's the problem with double-blind testing?

Reply #252 –
Quote from: Kees de Visser on
Quote from: greynol on
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.
Like ajinfla I'm confused. Can you explain how an ABX test with acoustical stimuli can reveal "the other stuff" ?


The idea that ABX is solely for demonstrating differences is debunked by the fact that ABX or testing methods like it are commonly used in industry for things like judging food and beverages.

The idea that ABX is solely for demonstrating the traditional idea of hearing which involves only the ears is debunked by the fact that abx naturally involves the sensations of the listener's entire body.

Some people get confused  when we say that our tests are blind, and speculate about blindfolds or poking out eyeballs. These are very badly confused people, but I have encountered them.

ABX tests are actually fully sighted except that we conceal one tiny aspect of reality for any peson with normal sensations, which is the true identity of what's playing right now. Just that one little thing!

We can even tell you what's playing as long as we tell you enough other things (like the identies of other components that might be playing right now)  that you can't figure out exactly what's playing right now.

The only thing that the subject person can't know is exactly what's playing right now. One tiny little piece of concealment in a divese, real world that is otherwise completely open to the subject person's senses. ABX tests are hardly blind at all! ;-)

Other than that, ABX tests are fully sighted, smell-enabled, tactiley enabled, taste enabled, etc., etc., etc. And, the listening sessions can go on as long as one desires, as many times as one desires.

Needless to say, there are many highly confused people who say ignorant things like ABX can't be used for long term listening. They speak out of ignorance, and I fear in many cases just a little bit of fear of the truth. ;-)

What's the problem with double-blind testing?

Reply #253 –
Quote from: greynol on
Quote from: Mark DeB on
And there is no necessity to assume that the subject can reliably report the effect using a term in her language, or that the experimenter would know what to ask for.

It's already been pointed out that this particular concern of yours is irrelevant.  No where in a double-blind ABX test or any of its variants is there a requirement that a person report the effect is his own language or anyone else's.


Said concern is in response to

Quote from: rpp3po on
No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.


 

What's the problem with double-blind testing?

Reply #254 –
Quote from: rpp3po on
Quote from: Mark DeB on
So, lest the point not be clear, I am pointing to the hypothesis that the effects of something analogous to such maskers, which are inaudible short term, may become greater with longer stimulus duration.


Isn't that getting pathetic? Besides not supporting your point in any way (since differences could be detected), how is a <=300 millisecond experiment "analogous" to long term stimulus exposure?

But I said I'd be out and really should be. Just found it somewhat amusing that you thought that you had found something...


If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.  This difference might, to be sure, be one that the subject could readily notice if she could somehow compare those parts of the experiences side by side, but which she cannot do because they are embedded in longer experiences.  And those parts could not simply be reproduced in isolation because the effect is context dependent.

If you switched back and forth between A and B, where A is the signal with said effect, you wouldn't notice a difference since both A and B would be affected by the prior exposures to A.

I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.

What's the problem with double-blind testing?

Reply #255 –
Quote from: Arnold B. Krueger on
The idea that ABX is solely for demonstrating differences is debunked by the fact that ABX or testing methods like it are commonly used in industry for things like judging food and beverages.


Then my apologies for being confused. I thought Hydrogen Audio would be primarily interested in the perception of audibility, with regards to music, including codecs or whatever DUT.
So even if the subjects(s) can't tell a lick of audible difference in sound tracks under test, if they get hungry (food) or thirsty (beverages), well this is of interest too.

cheers,

AJ
Loudspeaker manufacturer

What's the problem with double-blind testing?

Reply #256 –
Quote from: Mark DeB on
If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.  This difference might, to be sure, be one that the subject could readily notice if she could somehow compare those parts

I don't see that anyone here has given any convincing argument, or scientific reason

Agreed
Loudspeaker manufacturer

What's the problem with double-blind testing?

Reply #257 –
Quote from: Arnold B. Krueger on
The idea that ABX is solely for demonstrating differences is debunked by the fact that ABX or testing methods like it are commonly used in industry for things like judging food and beverages.
With "testing methods like it" you mean other forms of double blind testing ? In that case we're probably in complete agreement. It was just my understanding that ABX is useful for detecting perceptual differences whereas other DBT versions that include scaling and multidimensionality are more useful to quantify the variables that are responsible for various judgements.

What's the problem with double-blind testing?

Reply #258 –
Quote from: Mark DeB on
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.

See Russell's Teapot.
This is a forum of scientific inquiry.  Faith is an incompatible language.
Creature of habit.

What's the problem with double-blind testing?

Reply #259 –
Quote from: andy o on
Quote from: Mark DeB on
Quote from: andy o link=msg=0 date=
Quote from: Mark DeB link=msg=0 date=
If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

On the basis that you haven't made a worthwhile case to be considered. You're running on assumptions that are unwarranted and contradictory (one can "unconsciously perceive" something as different than something else, yet it is not detectable?). And, you're trying to explain


Please see below.  And I have not made any assumption that anyone "unconsciously perceives" something as different from something else; that would be to blur the distinction between (1) and (2) in my post #125.  Again, the thought is not the crude and unempirical one that some difference exists that cannot be detected in any way whatever; it is that the means that would reveal those differences would have to be more subtle and indirect than via the sorts of listening tests under discussion.  The psychology literature is replete with descriptions of situations in which such means are necessary, as in the discussion of blindsight in the first article I cited, or in J. Miller's discussion of "discrimination without awareness" (American Journal of Psychology, 1939).

Quote
Quote
Quote
a phenomenon that is easily and better explained by other known mechanisms (placebo and all that).


Sorry, but statements about me are not a scientific basis for anything relevant.  And the sorts of phenomena described in the article clearly do raise the plausibility level of "might" well beyond the "flat earth" or "dragon in garage" level at which you caricatured them as being; against the background of the knowledge that such phenomena exist, it is not at all pointless to ask whether the sorts of phenomena I have asked about exist.


Huh? Where am I making an ad hominem here? You're the one raising the question, and you're the one not making a good case. Your case is bad. And you didn't address that criticism at all (from me or anyone else). You're doing classic pseudoscientific misdirection moves.


I was referring to "On the basis that you haven't made a worthwhile case to be considered."

Look, let's look at the structure of the argument.  Just to clarify, I am not trying to "explain" anything, or make a "case" that some phenomenon exists.  I am saying that certain supposed evidence that a certain (not extremely well defined) kind of phenomenon doesn't exist does not, in fact, constitute good evidence for said conclusion.  At any rate, I have never seen any good argument that it does.

Quote
Yet no question remains answered.


Yes.  Exactly.

What's the problem with double-blind testing?

Reply #260 –
Quote from: Soap on
Quote from: Mark DeB on
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.

See Russell's Teapot.
This is a forum of scientific inquiry.  Faith is an incompatible language.


Basic principle: believe something if you have a reason for it.

The default position, in the absence of a reason, is doubt. 

No one has actually stated a relevant reason.

Faith would come into it if I were hawking expensive cables, but I'm not.  I'm just saying X doesn't follow from Y.  Moreover, no one, as far as I know, has stated much of a reason to think that Y-and-not-X has an antecedently low probability based on the rest of what we know.  Essentially, what I have suggested is a counterexample to conventional listening tests, a situation that, if it obtained, would be hard to detect using such tests.  The impression I get is that people here simply haven't thought about such a situation and reject it simply because it is unfamiliar, not because they know it can't obtain.

But scientific inquiry includes asking whether, if a certain situation obtained, existing methods would show that it does.  That sort of internal criticism is, according to Popper and others, a key element of science.  So if you say, I'm not going to worry about this until you prove that that situation does obtain, then unless you have theoretical reasons to think that it has an antecedently extremely low probability, you are just sticking your head in the sand.

As I say, doubt, or suspension of belief, is the right attitude to have in this case.  If you disagree, then please give a reason for something other than suspension of belief.  Otherwise, you are believing something without having a reason, which is what you refer to as faith.

What's the problem with double-blind testing?

Reply #261 –
Quote from: Mark DeB on
The impression I get is that people here simply haven't thought about such a situation and reject it simply because it is unfamiliar, not because they know it can't obtain.

No, you mean unfounded, not unfamiliar.

Quote from: Mark DeB on
As I say, doubt, or suspension of belief, is the right attitude to have in this case.

It sure is!

Quote from: Mark DeB on
If you disagree, then please give a reason for something other than suspension of belief.

That's pretty ironic.  As you are the one who's making the claims that ABX doesn't work because of the possible existence of XY and Z, it's really up to you to demonstrate their existence.  You have yet to do so.

What's the problem with double-blind testing?

Reply #262 –
Quote from: greynol on
Quote from: Mark DeB on
The impression I get is that people here simply haven't thought about such a situation and reject it simply because it is unfamiliar, not because they know it can't obtain.

No, you mean unfounded, not unfamiliar.


Please see below.

Quote
Quote from: Mark DeB on
As I say, doubt, or suspension of belief, is the right attitude to have in this case.

It sure is!


Well, then, you have conceded the point.

Quote
Quote from: Mark DeB on
If you disagree, then please give a reason for something other than suspension of belief.

That's pretty ironic.  As you are the one who's making the claims that ABX doesn't work because of the possible existence of XY and Z, it's really up to you to demonstrate their existence.  You have yet to do so.


No, because the point is that the nonexistence of Y-and-not-X is not a consequence of what we know (i.e., psychoacoustics, acoustics, etc.).  If you disagree, then you must think its nonexistence is a consequence, in which case please say why.

As to the claim that ABX "doesn't work," that is your formulation, not mine.

What's the problem with double-blind testing?

Reply #263 –
This is absurd.  You've come here with imaginary reasons why ABX might not work and expect us to prove how these imaginary reasons can't be true?

The onus is not on us, Mark.  It is on you to demonstrate that these reasons aren't imaginary.  You've given us nothing; you have nothing.

What's the problem with double-blind testing?

Reply #264 –
Quote from: Mark DeB on
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.


Well, if you insist on being scientific, in science the one who proposes something bears the burden of providing evidence.  Since it's you who is proposing such an effect it is up to YOU to provide evidence that it exists and, until you do it is, I suggest, perfectly reasonable to ignore it.

You say there "might be" such an effect?  Fine, show it and show that it makes a detectable difference.  Otherwise, "a difference that makes no difference is no difference".  Or to put it more technically, your suggestion is devoid of actual content.


Ed Seedhouse
VA7SDH

What's the problem with double-blind testing?

Reply #265 –
Quote from: Mark DeB on
As to the claim that ABX "doesn't work," that is your formulation, not mine.


So, to elaborate, it depends on what you mean by does or doesn't "work."  If somebody were to claim that certain cables cause a certain gross and readily audible difference, in frequency response say, then I agree that ABX testing could well debunk that claim.  If it seemed to a person that there was such a difference, the ABX test could support the hypothesis that this was an expectation effect.

What's the problem with double-blind testing?

Reply #266 –
OMG READING COMPREHENSION!!!

What's the problem with double-blind testing?

Reply #268 –
Quote from: Ed Seedhouse on
Quote from: Mark DeB on
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.


Well, if you insist on being scientific, in science the one who proposes something bears the burden of providing evidence.  Since it's you who is proposing such an effect it is up to YOU to provide evidence that it exists and, until you do it is, I suggest, perfectly reasonable to ignore it.

You say there "might be" such an effect?  Fine, show it and show that it makes a detectable difference.  Otherwise, "a difference that makes no difference is no difference".  Or to put it more technically, your suggestion is devoid of actual content.


On "actual content," please see post #215, substituting "actual content" for "meaning."

Actually, I have taken care to distinguish the notion of a difference detectable to the subject from that of a difference in perceptions detectable to the experimenter, way back to the distinction between (1) and (2) in post #125.

As to where the burden of evidence lies, please remember that I am saying that X does not follow from Y.  For all we know, there are some events that are Y-and-not-X.  Now do you agree or disagree with that?  If you disagree, then you must think that we know that there aren't any such events.  If so, how do we know that?  If, on the other hand, you agree with me, then I'm happy with that.

You may, in addition, feel that it is entirely reasonable, nevertheless, to go on to ignore the possibility of such events, even though you don't actually know that there can't be any such events.  Fine.  It's not my concern to disabuse you of a belief in the reasonableness of ignoring that possibility given lack of knowledge one way or the other.  However, it is not at all clear why it would be reasonable to ignore it, for purposes of the present discussion, if you don't know one way or the other.

What's the problem with double-blind testing?

Reply #269 –
Quote from: greynol on
This is absurd.  You've come here with imaginary reasons why ABX might not work and expect us to prove how these imaginary reasons can't be true?

The onus is not on us, Mark.  It is on you to demonstrate that these reasons aren't imaginary.  You've given us nothing; you have nothing.


As I said, the question of whether ABX "works" depends on the purposes to which it is put.  Consider:

Quote from: Pio2001 on
....  But in fact I am more interested in debunking false claims about sound quality. We are surrounded by unfounded information about 15,000 € CD players, or "unlistenable" 320 kbps MP3. And all these claims assume that the difference is conciously audible. Showing that this is not the case can be acheived with ABX tests.
Maybe there are also unconcious differences, that work in these cases, but it is already such a hard work to show that there are no "kickass differences" between two interconnect cables, and that silver interconnect cables do not emphasize treble, that I'm not planning to go beyond ABX (or AXY, or any other decision pattern) for the time being.


"Debunking," whatever exactly is meant, is something stronger than mere doubt or suspension of belief.  It is meant to show or establish something, to show that certain claims aren't true or justified.  And so, yes, there is an onus on someone who would want to do this.  The onus is on this person to give, if requested, a reason to think that his conclusion X is warranted on the basis of the supposed evidence Y: that Y really is evidence for X.  Now if it is possible that Y-and-not-X obtains, then that raises a question as to whether Y really does warrant the conclusion X.  Of course, it is not that the matter is closed at this point, because our background theory might say that such a possibility, while conceivable, is very unlikely; but in this case there is some burden on the debunker to explain how the theory predicts this.

So I am not saying a priori that the onus is on you to prove that Y-and-not-X can't be true, because (for one thing) I don't know if you are a "debunker."  But if you want to infer X from Y, or you want to say that X follows from Y, or you want to "debunk" the claim that not-X on the basis of Y, then it is reasonable to ask you to give a reason to think that the inference is warranted.  And consideration of possible counterexamples is not at all beside the point.

Interestingly, Pio2001 seems to consider the possibility of limitations to ABX also.  I was impressed with the quality of some of the earlier discussion and how well informed some of the participants were in cognition and psychology.

What's the problem with double-blind testing?

Reply #270 –
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.

What's the problem with double-blind testing?

Reply #271 –
Quote from: Mark DeB on
If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.


Who said ABX testing requires brief exposure?  Out of curiosity, are you talking "long exposure" like the duration of an album, or like decades? 

What's the problem with double-blind testing?

Reply #272 –
Quote from: rpp3po on
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't this all in reality just a quite inflated straw man discussion for a true, but much lamer story, started in [a href='index.php?showtopic=77910']this[/a] thread:

You had the feeling that upsampling your CD audio to 96 kHz (before it gets upsampled again to 110kHz in your DAC1), A, sounded better than letting the high quality resampler of the DAC1 do the same in a single step, B. You probably did some sighted testing while expecting yourself to sustain maximum objectivity. That all didn't quite work out, since your brain had already locked into believing that A sounded better than B in sighted comparison (a super common phenomenon), but that could basically not be held up after ABX comparison.

Now instead of accepting that your auditory system was just as easily fooled as anyone else's is by sighted comparison, you start this straw man crusade to establish a theoretical pseudo basis for the feeling you had instead of having to feeling fooled. 

What's the problem with double-blind testing?

Reply #273 –
Quote from: rpp3po on
Quote from: rpp3po on
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't this all in reality just a quite inflated straw man discussion for a true, but much lamer story, started in [a href='index.php?showtopic=77910']this[/a] thread:

You had the feeling that upsampling your CD audio to 96 kHz (before it gets upsampled again to 110kHz in your DAC1), A, sounded better than letting the high quality resampler of the DAC1 do the same in a single step, B. You probably did some sighted testing while expecting yourself to sustain maximum objectivity. That all didn't quite work out, since your brain had already locked into believing that A sounded better than B in sighted comparison (a super common phenomenon), but that could basically not be held up after ABX comparison.

Now instead of accepting that your auditory system was just as easily fooled as anyone else's is by sighted comparison, you start this straw man crusade to establish a theoretical pseudo basis for the feeling you had instead of having to feeling fooled. 


That is what it looked like to me too, but I said nothing. Anyway, the discussion as it's being conducted at the moment is obviously endless: "you have experimental results that prove X for experimental durations up to S_1 seconds; this does not prove that X holds for durations over S_1 seconds. Thus, X may not hold for durations over S_1 seconds". Well yes, this is so painfully obvious that it is barely worth discussing. I don't think anybody has disagreed with this (ie, that it does not follow; this is not the same as "it is not true") anywhere in this thread.

Beyond that there are the following issues, in the random order they occurred to me:
a) what X are we actually talking about? This has shifted around quite a bit during the discussion.
b) is it or is it not reasonable to extrapolate our results for X for durations less than S_1 to those over S_1? Obviously, it is a matter of opinion. In mine, for instance, in the absence of evidence or plausible mechanisms to the contrary, I'd just provisionally extrapolate the results. However, this is apparently unscientific (I'd have suggested some reading of actual scientific literature, as opposed to philosophers discussing science, to see whether this is true or not, but I'd just look smug so won't).
c) There were a few random references to papers, which did not have much to do with the discussion (and it does not seem that the person mentioning them had read anything but the abstract).
d) The tactic of taking a sentence out of a hundred or so and focusing only on that was employed a few times (eg focusing on "common sense" and attacking a particular meaning of that, which differed from the intended, as explained in detail in the post being replied to).
e) others, but I've already wasted enough space.

So in conclusion, and in view of points (a), © and (d), I submit that we are being wound up

What's the problem with double-blind testing?

Reply #274 –
Quote from: Mark DeB on
Quote from: andy o on
How do you think they tested if there were any effects of unconscious perception? We're not saying that there can't be unconscious perception, only that its effects if any can be tested, and clearly the authors of that paper think the same.


Yes, you are right that the authors of the article think that unconscious perception can be detected in some ways (otherwise, how would they be able to write the article?), but the point, or one point, is that those ways of detecting it need not be limited to, and might have to be more subtle and indirect, than the sorts of listening tests that are supposed to be adequate for relevant purposes here. 

By the way, unconscious priming is a good example of what I mean by indirect detection.  Priming is the phenomenon in which exposure to one stimulus (the "prime") affects the speed or accuracy of response to a subsequent stimulus (the "target").  Semantic priming is well known, where the priming effect depends on whether the target is related semantically, or by association, to the prime, for example, "dog" may be recognized with greater speed or accuracy if it is preceded by "cat" than by "table" (Healy, ed., p. 453).  It turns out that such priming can occur when the subject is not consciously aware of the prime (Marcel, 1983).  This is known as unconscious or subliminal priming.

So in a situation such as this, the experimenter may be able to detect the effects of the unconscious exposure, all right, but it is not through the subject's ability to discriminate in the normal way; rather, the route to detecting this is much more indirect, and the experimenter has to be pretty sophisticated in knowing what to test for.

Here are some references:

Alice F. Healy et al., ed., Handbook of Psychology, Volume 4: Experimental Psychology (Wiley, 2004)

http://books.google.com/books?id=VNPV0s6I6...cel&f=false

(see pp. 11 and 453)

Marcel, A. J. (1983). Conscious and unconscious perception: Experiments on visual masking and word recognition. Cognitive Psychology, 15, 197-237.

Greenwald, Klinger, and Liu, "Unconscious processing of dichoptically masked words."
http://faculty.washington.edu/agg/pdf/GK&L.1989.OCR.pdf