Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 248646 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What's the problem with double-blind testing?

Reply #175 –
Quote from: MichaelW on
If the hypothesis is that different sound signals which are audibly, and in all other ways, indistinguishable by the subject, might produce different effects on the human organism, that would be a claim in science.


I hope that this in fact is Mark DeB's hypothesis. I see it this way every since he first posted on this thread.

But this should also be ABX-able, as greynol (and others) have already stated. In my somewhat funny suggestion, I was thinkig of such a prolonged ABX test. Those, who believe that Mark DeB's hypothesis is on to something, should take such a test and get done with it already. 

If one would want to do the test without ABX, here's another one of my funny suggestions. 

In a mad scientist kind of way, this hypothesis could be tested with 10 pairs of twins, which would have been separated since birth until they stopped growing. One group of twins would be exposed to lossy samles of the same sounds as the other group would be exposed to by original samples. The mad scientist could mesure all kinds of body functions and processes and whatever he would think of. If he would want, he could try them psychologically - they would debate about experiences with certain sounds and see if they feel different about them. Between the groups there should be absolutely no other differences, besides the sounds. This test is virtualy impossible to do and I am only describing it, so people who support this hypothesis would see how absurd it is to be concerned about such subtle differences. They may be so subtle that something else can have a greater impact on us and that they don't matter at all. In fact, I believe that that is the case here - if there even is a different effect on our organism by lossy sounds.
lame -V 0

What's the problem with double-blind testing?

Reply #176 –
Quote from: Mark DeB on
Sorry, that's a bit cryptic: could you please explain?


What specifically do you not understand?

Quote from: Mark DeB on
Suppose we are talking about comparison of two similar, but not necessarily identical, one-minute, complicated, atonal passages of music.  You may not be able to say reliably whether they were identical or not.  But surely, if they were not identical, you perceived different pitches and rhythms at corresponding times in the samples: a difference in what is perceived without a perceived difference.


Here we come to the center of your problem. You accuse others here of inferring to much, while making the biggest unfunded jump yourself, i. e. you are inferring from the outside knowledge of factual differences to the existence of different subjective perceptions. Two super-complex, atonal passages with very little difference might be perceived as the same 'bringbrangpling' by an untrained subject with neither different perceptions nor causal differences. In contrast for someone familiar with atonal music there might be different perceptions giving one track at least another (partial) gestalt than the other, but he will be able to gain knowledge about it.

The only niche that remains for you would be:

1. Actually different tracks
2. Neither perceived difference nor different perceptions
3. Still causal differences (e.g. emotionally) between A and B in relation to S

The existence of the latter can be verified by long term, double-blind, correlated A/B testing. It's quite a task, it's very improbable that there is actually anything to find, but feel free to go that route. Findings could be revolutionary. But the could-be-would-be-space of highly improbable problems doesn't bring us forward. The space of that is just too vast*. You are welcome to present any new significant data and I'm sure this community would welcome it. But the causally-effective-but-imperceivable-in-blind-comparison-class of problems without accompanying proof is better taken care of in other forums.

* Lets start with I might be an operator assigned to your brain in tank No. 284398398A.

What's the problem with double-blind testing?

Reply #177 –
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?
(2) How would you test this hypothesis?

If psychoacoustics rules it out or says that it is unlikely, why?  What is the explanation, or "good theoretical reasons"?  And if, though our best currently available theory does not rule it out, it describes no known mechanism by which such an effect might occur, what are we entitled to conclude from that?  Is the current state of our theory so advanced that it would describe such a mechanism if one existed?

For testing, quick-switch ABX would be useless because the phenomenon occurs only with longer stimuli.  And longer-stimuli ABX tests may not be reliable, as is illustrated by the example of the atonal piece I suggested to rpp3po in a previous post.  In the present case, how a note sounds midway through the stimulus in one presentation may differ from how it sounds in the other, but the subject would lack any way of directly comparing them, and hence of detecting or reliably reporting that they differ.

What's the problem with double-blind testing?

Reply #178 –
Quote from: rpp3po on
Quote from: Mark DeB on
Sorry, that's a bit cryptic: could you please explain?


What specifically do you not understand?

Quote from: Mark DeB on
Suppose we are talking about comparison of two similar, but not necessarily identical, one-minute, complicated, atonal passages of music.  You may not be able to say reliably whether they were identical or not.  But surely, if they were not identical, you perceived different pitches and rhythms at corresponding times in the samples: a difference in what is perceived without a perceived difference.


Here we come to the center of your problem. You accuse others here of inferring to much, while making the biggest unfunded jump yourself, i. e. you are inferring from the outside knowledge of factual differences to the existence of different subjective perceptions. Two super-complex, atonal passages with very little difference might be perceived as the same 'bringbrangpling' by an untrained subject with neither different perceptions nor causal differences. In contrast for someone familiar with atonal music there might be different perceptions giving one track at least another (partial) gestalt than the other, but he will be able to gain knowledge about it.

The only niche that remains for you would be:

1. Actually different tracks
2. Neither perceived difference nor different perceptions
3. Still causal differences (e.g. emotionally) between A and B in relation to S

The existence of the latter can be verified by long term, double-blind, correlated A/B testing. It's quite a task, it's very improbable that there is actually anything to find, but feel free to go that route. Findings could be revolutionary. But the could-be-would-be-space of highly improbable problems doesn't bring us forward. The space of that is just too vast*. You are welcome to present any new significant data and I'm sure this community would welcome it. But the causally-effective-but-imperceivable-in-blind-comparison-class of problems without accompanying proof is better taken care of in other forums.

* Lets start with I might be an operator assigned to your brain in tank No. 284398398A.


No, if we are talking about a piece by Webern, say, and the sort of listener who enjoys listening to music, attends concerts, etc., though she may not know much music theory, the best available theories in music cognition tell us that she is going to have conscious mental representations of specific pitches as they go by.  She will not (only) have a perception of "bringbrangpling."  But if these are one-minute presentations, she is not going to remember the specific pitch in one presentation long enough to compare it with its counterpart in the other.
 

What's the problem with double-blind testing?

Reply #179 –
Quote from: Mark DeB on
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.

Quote from: Mark DeB on
(2) How would you test this hypothesis?


Long term A/B testing. Recording when fatiguing effect has been noticed. After sufficient number of trials, correlate records with actual succession of As and Bs.

For the 100th time...

What's the problem with double-blind testing?

Reply #180 –
But Mark DeB said that this fatiguing effect cannot be reliably identified...

What's the problem with double-blind testing?

Reply #181 –
Quote from: MichaelW on
Quote from: Mark DeB on
RE: "For any reasonable meaning of "experience," different experiences are detectable."

That is just a dogmatic claim, armchair neuroscience.


No it's not. It's not science at all. It is about the meaning of words. "Experience," in this context, means, in normal usage, something which affects the consciousness of a subject, even if it can't be precisely identified. I am reliably informed that an appalling number of neutrinos whizz through my body every day, but this is not part of my "experience," though it is something which, I believe, happens to me.

If the hypothesis is that different sound signals which are audibly, and in all other ways, indistinguishable by the subject, might produce different effects on the human organism, that would be a claim in science. To call these effects part of the human being's experience would be an idiosyncratic used of the word.


I wouldn't have thought that "different experiences are detectable" is true in virtue of meaning, on the normal meaning of "experience," but be that as it may: do you mean detectable by the subject who has the experiences?  Always?  Consider the example of the atonal piece (described in another post) in which a small change is introduced.  In one presentation the subject hears one pitch; in another he hears a different pitch.  The experiences are different, but the subject may not be able to detect that they are.

What's the problem with double-blind testing?

Reply #182 –
Quote from: rpp3po on
Quote from: Mark DeB on
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.

Quote from: Mark DeB on
(2) How would you test this hypothesis?


Long term A/B testing. Recording when fatiguing effect has been noticed. After sufficient number of trials, correlate records with actual succession of As and Bs.

For the 100th time...


But what "noticing" the effect would mean here is that the subject compares corresponding parts of the stimulus and sees that the way one sounds is different from the way the other sounds, which is hard to do reliably if they are separated in time.

What's the problem with double-blind testing?

Reply #183 –
Quote from: Mark DeB on
No, if we are talking about a piece by Webern, say, and the sort of listener who enjoys listening to music, attends concerts, etc., though she may not know much music theory, the best available theories in music cognition tell us that she is going to have conscious mental representations of specific pitches as they go by.
She will not (only) have a perception of "bringbrangpling."  But if these are one-minute presentations, she is not going to remember the specific pitch in one presentation long enough to compare it with its counterpart in the other.


It may seem obvious at the first look, but it isn't. Please show how you infer from different pitch at time x to different mental representation at time x. As soon as modify the test-setup to give that answer it will also be able to give the "perception of a difference" answer. This is, because your long-term effect is only a pseudo long-term effect. Your analogy just pads an easily discernible difference (two different pitches at time x) with a long timespan during which it is forgotten again. The crux is the easily discernible difference at time x. This is a flawed analogy to the hypothesis you were originally trying to make: an at no time by direct comparison detectable difference that still exists with causal effect.

PS

Quote from: Mark DeB on
But what "noticing" the effect would mean here is that the subject compares corresponding parts of the stimulus and sees that the way one sounds is different from the way the other sounds, which is hard to do reliably if they are separated in time.


No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.

PPS Your excessive block citing style is not appreciated.

What's the problem with double-blind testing?

Reply #184 –
Quote from: rpp3po on
Quote from: Mark DeB on
But what "noticing" the effect would mean here is that the subject compares corresponding parts of the stimulus and sees that the way one sounds is different from the way the other sounds, which is hard to do reliably if they are separated in time.


No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.



Why should we assume that the subject can reliably report when the effect (not necessarily a feeling of fatigue) is occurring?  If she cannot, then a test such as you describe won't tell us much.

What's the problem with double-blind testing?

Reply #185 –
Quote from: rpp3po on
Quote from: Mark DeB on
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.

Quote from: Mark DeB on
(2) How would you test this hypothesis?


Long term A/B testing. Recording when fatiguing effect has been noticed. After sufficient number of trials, correlate records with actual succession of As and Bs.

For the 100th time...



Quote from: rpp3po on
No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.


Moreover, how are you going to instruct the subject as to what effect, what property of her experience, she is supposed to report?  You are assuming that there is an expression in a public language, shared by subject and experimenter, that designates this property of experience and that the experimenter knows what expression to use.

What's the problem with double-blind testing?

Reply #186 –
We are turning in circles: No detectable trace, no relevant causal effect. You are like Don Quijote, endlessly defending the possible existence of a priori undetectable causal effects. Look at it from a computability perspective: The set of possibly detectable causal effects has already greater cardinality than infinity. You can't work with that. Science is essentially reducing this set to the most reasonable subsets. So your whole approach is fundamentally unscientific, because you try to extent the global set to even a priori undetectable (by your own definition) effects for no practical (or even theoretical) benefit. While you use modern terminology to describe your position, it is the same class as the clerical defense of the geocentric model. Neglecting the huge benefits of scientific discoveries by accepting the sheer fact of their possible incompleteness as an argument.

Now you even retreat to that (e. g.) upsampling may raise a feeling that there is no word for. OK, I think we finally can come to an end then. The position you are defending is so exclusive that

1. you yourself can't detect its existence, because its effects are unobservable
2. there isn't even a word for it

What's the worth of such a position?? What's the sense of defending it?

I make the following alternative claim:

Upsampling should be avoided because it causes 'gooness'. Gooness is harmful. Sadly I cannot say more because 1. and 2. (in the sense that we don't share the word in a common sense) apply.

What's the problem with double-blind testing?

Reply #187 –
Quote from: rpp3po on
We are turning in circles: No detectable trace, no relevant causal effect. You are like Don Quijote, endlessly defending the possible existence of a priori undetectable causal effects. Look at it from a computability perspective: The set of possibly detectable causal effects has already greater cardinality than infinity. You can't work with that. Science is essentially reducing this set to the most reasonable subsets. So your whole approach is fundamentally unscientific, because you try to extent the global set to even a priori undetectable (by your own definition) effects for no practical (or even theoretical) benefit. While you use modern terminology to describe your position, it is the same class as the clerical defense of the geocentric model. Neglecting the huge benefits of scientific discoveries by accepting the sheer fact of their possible incompleteness as an argument.

Now you even retreat to that (e. g.) upsampling may raise a feeling that there is no word for. OK, I think we finally can come to an end then. The position you are defending is so exclusive that

1. you yourself can't detect its existence, because its effects are unobservable
2. there isn't even a word for it

What's the worth of such a position?? What's the sense of defending it?

I make the following alternative claim:

Upsampling should be avoided because it causes 'gooness'. Gooness is harmful. Sadly I cannot say more because 1. and 2. (in the sense that we don't share the word in a common sense) apply.


Who says it is undetectable?  Maybe there are some means by which it is detectable.  The question was how to detect or test for it, the kind of test you suggested won't work, and now you say anything undetectable by those means is not worth talking about.

It is not a "retreat" to say that there are feelings there are no words for, since I never claimed that there weren't such feelings to begin with.  Obviously, there are lots of such feelings.  Why shouldn't the listener care about them?

And even if the subject does have a word for the feeling, the experimenter has to know what feeling to ask about, in your suggested test.  It may not suffice to ask, "Are you having a feeling of fatigue now?" since the subject may not associate the effect with that expression.  How is the experimenter going to know what expression to use?

What's the problem with double-blind testing?

Reply #188 –
This is really quite pathetic.

If the person who is trying to determine whether X sounds different from Y cannot identify a "feeling" or whatever you want to call it that allows him to distinguish a difference then there is no difference to that person at that point in time.  If the person needs help from someone else who can, that's basically an admission that he cannot distinguish the two prior to getting the help, assuming the help even makes a difference.

Again, Mark, you have absolutely nothing.

What's the problem with double-blind testing?

Reply #189 –
This is getting really tiring. Please let us know which effect you have observed and documented, that might be interesting to others here, and how it is reproducible. Anything else won't find much interest. Feelings should be cared about, yes, as long as they are occurrent. Completely under the radar, out of the scope of any possible documentation, they don't matter here. There is not the slightest indication or finding, that your claim is in any way relevant. If you want to change that, show some data. The ontological status of the "could" you have presented belongs to the same class as the question wether aliens live among us without us knowing. I'm not interested to follow that any further.

Bye

What's the problem with double-blind testing?

Reply #190 –
(this thread is getting painful to lurk)

I agree that if you claim something exists then the onus is on you to prove it exists. Until that's done there is really nothing more to talk about here.

What's the problem with double-blind testing?

Reply #191 –
Quote from: greynol on
This is really quite pathetic.

If the person who is trying to determine whether X sounds different from Y cannot identify a "feeling" or whatever you want to call it that allows him to distinguish a difference then there is no difference to that person at that point in time.  If the person needs help from someone else who can, that's basically an admission that he cannot distinguish the two prior to getting the help, assuming the help even makes a difference.

Again, Mark, you have absolutely nothing.


You are assuming that the experimenter is the same person as the subject.  Go back and look at the distinction between (1) and (2) in my post #125:

(1) You perceive a as X and you perceive b as Y, and X is different from Y.
(2) You perceive that a is different from b.

You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.

Would you mind being civil, though?  Thank you.

What's the problem with double-blind testing?

Reply #192 –
Quote from: Mark DeB on
You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.
Which other ways would there be (unless the subject is told)?

What's the problem with double-blind testing?

Reply #193 –
Quote from: Mark DeB on
Would you mind being civil, though?

Calling you pathetic would have been uncivil.  Calling your failed attempt at successfully defending your point of view pathetic (and it is pathetic) is not.

Would you mind answering Nick.C's question?  Thank you.

What's the problem with double-blind testing?

Reply #194 –
Quote from: rpp3po on
This is getting really tiring. Please let us know which effect you have observed and documented, that might be interesting to others here, and how it is reproducible. Anything else won't find much interest. Feelings should be cared about, yes, as long as they are occurrent. Completely under the radar, out of the scope of any possible documentation, they don't matter here. There is not the slightest indication or finding, that your claim is in any way relevant. If you want to change that, show some data. The ontological status of the "could" you have presented belongs to the same class as the question wether aliens live among us without us knowing. I'm not interested to follow that any further.

Bye


Fine.  But then it remains an open question whether the signals differ in properties relevant to perception, since you haven't suggested any way to resolve it.  Your request for documentation of an observed effect misses the point.  I never claimed to have observed any such effect (and, if you have been following things, you know I don't assume that the effects in question can be reliably reported).  The question is whether there are any effects of the kind described in the hypothesis, and whether tests such as you describe would be relevant to finding this out.  Evidently, they would not be.

Concerning the "ontological status" of this versus aliens, not so.  This is a question in psychology and cognition.

It may be that a hypothesis that posits things that are undetectable in principle would not be worth considering, but I am not suggesting such a hypothesis.  There might well be ways of detecting whatever properties and effects might make the hypothesis true, although those ways would probably vary with the properties/effects.  So there is no reason why some fixed method of determination should be hard wired into the hypothesis.  However, it is not hard to imagine an alternative to ABX listening tests of the kind you suggest: brain scans.  We might well have excellent theoretical reasons to think that, when a part of the brain lights up in a certain way in response to one stimulus but not another, that this corresponds to a difference in the experiences.

The fact that the hypothesis cannot be falsified using your methodology does not mean that it is unfalsifiable tout court, or cognitively meaningless.

Look, here is another way of putting the point: Why should I bother with lossy formats for listening when I can use FLAC?  I don't want to lose any relevant information.  What is the criterion for relevance?  Many people seem to assume that conscious discrimination (in certain conditions) is an adequate criterion, but I see no real argument for this.  It's just a dogma.

Thank you for the dialogue.

What's the problem with double-blind testing?

Reply #195 –
Quote from: Qest on
(this thread is getting painful to lurk)

I agree that if you claim something exists then the onus is on you to prove it exists. Until that's done there is really nothing more to talk about here.


Since I am not claiming that something exists, but asking how it can be proved that it doesn't, your claim about a burden of proof is based on a false assumption.

What's the problem with double-blind testing?

Reply #196 –
Quote from: greynol on
Quote from: Mark DeB on
Would you mind being civil, though?

Calling you pathetic would have been uncivil.  Calling your failed attempt at successfully defending your point of view pathetic (and it is pathetic) is not.

Would you mind answering Nick.C's question?  Thank you.


They are both uncivil.  A civil discussion of some topic means sticking to the subject and giving reasons and argument, not calling names, even applied to someone's view.  And that's what you did, and continued to do in the post to which this is a response.

What's the problem with double-blind testing?

Reply #197 –
Quote from: Nick.C on
Quote from: Mark DeB on
You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.
Which other ways would there be (unless the subject is told)?


Please see post #195 for a suggestion.

What's the problem with double-blind testing?

Reply #198 –
Oh, one other thing.  The suggestion that there can be effects of perceptual information that do not translate into conscious discrimination is not exactly an idea from Mars, but is well known in the literature on cognition.  Blindsight is a well-known example.

What's the problem with double-blind testing?

Reply #199 –
Quote from: Mark DeB on
We might well have excellent theoretical reasons to think that, when a part of the brain lights up in a certain way in response to one stimulus but not another, that this corresponds to a difference in the experiences.


On a side note, I suspected that you are coming from that school. Nowadays it is a quite popular position among current neurologist (maybe sometimes without a certain amount of philosophical background) and popular science. Even many proponents of the contemporary philosophy of mind follow that route. Trying to solve own philosophical inadequacies with tech seems much sexier than challenging the claims of an in-fashion discipline as neurology. I profoundly believe that the quoted claim can be refuted on phenomenological grounds alone. But that would be another discussion, at least 10x as long as the latter and not belonging here.