Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 248969 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

What's the problem with double-blind testing?

Reply #225 –
Thanks for the correction on short term/long term/quick audition terminology krab; I'm probably the most guilty party.

What's the problem with double-blind testing?

Reply #226 –
Quote from: krabapple on
This is not equivalent to your friend's problem though. It IS possible to quickly switch between 24/96 and 16/44, using whatever test signal length your friend desires.
Actually it's not his problem. Using 96 kHz in the studio is a no brainer these days, since there are hardly any downsides. I was just curious if his claim about listening fatigue differences could be verified. Since fatigue needs time to develop, I assume that fast ABX switching is not possible. Besides that he is willing to spend some time on testing, but not days or weeks, especially since there is nothing for him to gain apart from perhaps some insight.
Amongst the thousands of unverified subjective decisions made during each music production, this one is hardly going to make a difference
Quote
The real question is, is a 'perception' of difference a reliable indicator of objective difference?
Often I'm more interested in the perception and how it's being influenced than in the objective analysis of auditory stimuli alone (note how this is different from testing a lossy codec, where the codec is the DUT).

What's the problem with double-blind testing?

Reply #227 –
I would imagine the best way to do that test would to do what Meyer/Moran did. Assume that your friend has a digital audio link from his DAW to speakers. Add a device/DVD player/etc of known high quality that can either pass audio through unmodified or resample to 44khz.

What's the problem with double-blind testing?

Reply #228 –
Quote from: Mark DeB on
Quote from: andy o link=msg=0 date=
Quote from: Mark DeB link=msg=0 date=
If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

On the basis that you haven't made a worthwhile case to be considered. You're running on assumptions that are unwarranted and contradictory (one can "unconsciously perceive" something as different than something else, yet it is not detectable?). And, you're trying to explain a phenomenon that is easily and better explained by other known mechanisms (placebo and all that).


Sorry, but statements about me are not a scientific basis for anything relevant.  And the sorts of phenomena described in the article clearly do raise the plausibility level of "might" well beyond the "flat earth" or "dragon in garage" level at which you caricatured them as being; against the background of the knowledge that such phenomena exist, it is not at all pointless to ask whether the sorts of phenomena I have asked about exist.

Huh? Where am I making an ad hominem here? You're the one raising the question, and you're the one not making a good case. Your case is bad. And you didn't address that criticism at all (from me or anyone else). You're doing classic pseudoscientific misdirection moves.

Quote
Yes, you are right that the authors of the article think that unconscious perception can be detected in some ways (otherwise, how would they be able to write the article?), but the point, or one point, is that those ways of detecting it need not be limited to, and might have to be more subtle and indirect, than the sorts of listening tests that are supposed to be adequate for relevant purposes here.  Also, the authors talk about the effect of unconscious perception on emotional states, which might very well be detected not by the subject herself but using behavioral or physiological criteria.  That may give an idea of why I think that article is relevant, and I hope you find it useful.  Your questions in this post were good ones, and, in principle, I would be interested to discuss these issues further, but you have been sufficiently abusive to leave me disinclined to do this; I hope you will understand.  As I noted in my previous post, my question was answered.  Thank you for the discussion.

Ah, yes. The civility gambit again. Yet no question remains answered.

What's the problem with double-blind testing?

Reply #230 –
Quote from: Canar on
All the philosophical hand-waving in the world won't make it sensible to ignore a request of other people to define what it is you're talking about when you talk about a thing.


Exactly.

What's the problem with double-blind testing?

Reply #231 –
Quote from: greynol on
...not to mention another ironic comment about science again.



I await his answer to the simple, easily supported observation that ABX testing, signal detection testing, and other methods, all of them double-blind, have been shown to detect stimulii down to the level that mathematics and physics suggest is possible.

Absolute thresholds in the ear canal resonance range, for instance, stand out here, given the established noise level due to the molecular nature of the atmosphere.

It's also true that the basic SNR detection ability of each individual hair cell in the ear is known to be in the range of 30dB, which is extended over about 90-120dB (depending on how you choose toe define "extended) by loudness control mechanisms due to the outer hair cells.  This kind of understanding clearly demonstrates why level changes below a given amount are inaudible, those levels being due to coding, level control, EQ, what-have-you.

Such level calculations agree very well with the actual observed performance of subjects in blind tests. There really is nothing appearing to be missing.

Now, as to the failure to define terms, etc, that is a classical gambit of the dissembler, in that it allows use of the straw man gambit in one of several ways. First, it allows a redefinition of the subject via "but that's not what I meant" dissemblement. Second, it allows  the use of "you said x, therefore you mean y, because z" wherein z is then argued to be a "natural" definition of the original issue. There are other ways in which this gambit can be used, but nobody pretending to be scientific, skeptical, or the like can claim to be such if they are not willing to actually define the assertions that they are making. That, in a nutshell, is what refusing to define a term comes down to.

Such behavior is rude, obnoxious, and does not lead to either enlightenment or concensus, unless you're to take the Derridian approach that "enlightement means there is nothing".
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #232 –
Priceless.
 

What's the problem with double-blind testing?

Reply #233 –
Don't take my word for it...

"The experiment reported here tests a counterintuitive prediction of this linear-summation hypothesis, namely that a sound that itself is inaudible should, under certain circumstances, affect the audibility of subsequent sounds. The results show that, when two forward maskers are combined, the second of the two maskers can continue to produce substantial masking, even when it is completely masked by the first masker. Thus, inaudible sounds can affect the perception of subsequent sounds."

http://www.jneurosci.org/cgi/content/full/26/34/8767

Masking by Inaudible Sounds and the Linearity of Temporal Summation

Christopher J. Plack, Andrew J. Oxenham, and Vit Drga

The Journal of Neuroscience, August 23, 2006, 26(34):8767-8773; doi:10.1523/JNEUROSCI.1134-06.2006

What's the problem with double-blind testing?

Reply #234 –
If it produces effects on the audibility of subsequent sounds, the phenomenon is tractable to blind testing and so this article does not further your claims or explain your hesitation to define what "hear" means.

That article has no bearing on blind-testing whatsoever.

What's the problem with double-blind testing?

Reply #235 –
How does this relate to the debated issue for your cause in any way? While it is claimed in this paper that the presence of a masker could not be perceived, a difference could. You have instead been arguing for case where a difference cannot be perceived while still having causal effect.

What's the problem with double-blind testing?

Reply #236 –
Quote from: Canar on
If it produces effects on the audibility of subsequent sounds, the phenomenon is tractable to blind testing and so this article does not further your claims or explain your hesitation to define what "hear" means.

That article has no bearing on blind-testing whatsoever.


In the body of the article (as opposed to the abstract) it says: "On each trial, listeners were presented with three observation intervals separated by 300 ms. Two intervals contained the masker(s) only, and one interval (chosen at random) contained the masker(s) plus the signal."

Also, "Listeners were seated in a double- walled sound-attenuating booth and made their responses via a com- puter keyboard. “Lights” on the computer monitor indicated the time of occurrence of the observation intervals and provided feedback as to whether the response was correct or incorrect."

They were also trained on the tasks until performance was stable.

The word "inaudible" as used in this paper is being taken out of context in the posts to this thread.

What's the problem with double-blind testing?

Reply #237 –
Quote from: Mark DeB on
Don't take my word for it...

"The experiment reported here tests a counterintuitive prediction of this linear-summation hypothesis, namely that a sound that itself is inaudible should, under certain circumstances, affect the audibility of subsequent sounds. The results show that, when two forward maskers are combined, the second of the two maskers can continue to produce substantial masking, even when it is completely masked by the first masker. Thus, inaudible sounds can affect the perception of subsequent sounds."

http://www.jneurosci.org/cgi/content/full/26/34/8767

Masking by Inaudible Sounds and the Linearity of Temporal Summation

Christopher J. Plack, Andrew J. Oxenham, and Vit Drga

The Journal of Neuroscience, August 23, 2006, 26(34):8767-8773; doi:10.1523/JNEUROSCI.1134-06.2006


So, lest the point not be clear, I am pointing to the hypothesis that the effects of something analogous to such maskers, which are inaudible short term, may become greater with longer stimulus duration. Now surely that is not an outlandish hypothesis, that a psychophysical effect should become greater with longer stimulus duration?  But that would be a problem for ABX, because if the compared intervals are short then the effect would not occur, whereas if the stimuli are longer then the comparison would be less reliable.

(And there is no necessity to assume that the subject can reliably report the effect using a term in her language, or that the experimenter would know what to ask for.)

What's the problem with double-blind testing?

Reply #238 –
Quote from: Mark DeB on
So, lest the point not be clear, I am pointing to the hypothesis that the effects of something analogous to such maskers, which are inaudible short term, may become greater with longer stimulus duration.


Isn't that getting pathetic? Besides not supporting your point in any way (since differences could be detected), how is a <=300 millisecond experiment "analogous" to long term stimulus exposure?

But I said I'd be out and really should be. Just found it somewhat amusing that you thought that you had found something...

What's the problem with double-blind testing?

Reply #239 –
Quote from: Mark DeB on
Don't take my word for it...

"The experiment reported here tests a counterintuitive prediction of this linear-summation hypothesis, namely that a sound that itself is inaudible should, under certain circumstances, affect the audibility of subsequent sounds. The results show that, when two forward maskers are combined, the second of the two maskers can continue to produce substantial masking, even when it is completely masked by the first masker. Thus, inaudible sounds can affect the perception of subsequent sounds."

http://www.jneurosci.org/cgi/content/full/26/34/8767

Masking by Inaudible Sounds and the Linearity of Temporal Summation

Christopher J. Plack, Andrew J. Oxenham, and Vit Drga

The Journal of Neuroscience, August 23, 2006, 26(34):8767-8773; doi:10.1523/JNEUROSCI.1134-06.2006


What does this have to do with running a listening test? Such time delays are under one second, at the very most, and typically well under 200 milliseconds.  What is more, in a blind test, the subject will do this (assuming it happens) to all signals, and thereby not lose anything.

Of course, the ear just isn't a linear system, so the "linearity" comment is interesting in this light.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #240 –
Quote from: Mark DeB on
And there is no necessity to assume that the subject can reliably report the effect using a term in her language, or that the experimenter would know what to ask for.

It's already been pointed out that this particular concern of yours is irrelevant.  No where in a double-blind ABX test or any of its variants is there a requirement that a person report the effect is his own language or anyone else's.

What's the problem with double-blind testing?

Reply #241 –
Mark DeB, consider this:

The effects, you're talking about, that compressed sound COULD have, fall in the same category as the effects that COULD be caused by using different playback hardware.

For instance, let's say that sounds, which lack something undetectable, cause brain cancer. Now, let's have 2 pairs of subjects. First two are listening through the same hardware, difference being one listens to the original-uncompressed sound, the other one listens to the same sound, but compressed. The later gets brain cancer. The other two are listening to the same original-uncompressed sound, difference being one listens with best hardware there is, the other one with a hardware that provides same sounding sound, yet there is an undetectable difference. The later gets brain cancer.

You see... even if you PROVE that audio compression causes brain cancer (or whatever effect on human organism), where do you stop then? Next thing would be audio playback hardware. And if you then prove that bad hardware also causes brain cancer (or whatever effect on human organism), what's next. You go on and persue the idea that jazz causes brain cancer, whereas rock doesn't?!

Sorry, your theory concerns with effects so subtle, that it is just not worth your trouble.
lame -V 0

What's the problem with double-blind testing?

Reply #242 –
You should reconsider your use of the term proof, psycho.

What's the problem with double-blind testing?

Reply #243 –
rpp3po, please explain. I used proof together with if... (...if you prove...), so I think I used it OK, but I'm ready to learn.
lame -V 0

What's the problem with double-blind testing?

Reply #244 –
Quote from: Mark DeB on
For testing, quick-switch ABX would be useless

Your repetition of "quick" listening passage length is baseless. You can listen as long as you want. It is the switching time between passages that is quick. Lets quickly discard with that.
Quote from: Mark DeB on
I see no reason why I should conclude from the sort of ABX test I took yesterday that the relevant signals do not, in normal listening, cause different experiences in me, or cause me to perceive things in different ways, where the difference is one that matters.

By "normal" listening, you mean sighted and uncontrolled, as you would "normally" listen. Then you don't have an understanding of "perception". Specifically, what is being tested here is hearing perception. This consists of the sound field (the sound waves that impinge upon your ears part) and then the "other" factors of the brains "hearing" process, such as visual stimuli, emotions, biases, beliefs, etc, etc.
ABX is a method of reducing you perception choices to that of the sound field only. If there is a perceptible difference in the sound field, your ABX choices will reveal that. If not,your ABX choices will reveal that also. That does not mean that during the test your brain isn't trying to do what it normally does with regards to emotions and biases, etc. You can still very much imagine hearing differences not in the sound field, that you would during sighted, uncontrolled listening, but your choices will reflect random, coin-flip scoring. In other words, guessing.
The test is not of "your perceptions". You could still "perceive" getting cold listening to A and warm listening to B. The test is of hearing perception. You either hear a difference or you don't. A formal rule of logic is that you cannot prove a negative. A null does not mean there is no difference. It means that you could not "hear" an audible difference under the test conditions.
If you hold a belief that you can "hear" a difference under uncontrolled "listening" (given that you now know that the hearing process involves factors other than the sound field) then take the ABX (which forces you to "hear" just the sound field) where you can no longer "hear" this difference...and still believe that it is the test that has failed you, rather than your belief, then you are a subjectivist. The definition of which, is a person who will evade conclusions, by refusing to believe in them.
Take as much time as you need with the ABX (though switch quickly). The conclusion will be the same, whether you accept it or not.
We'll save fallibility of acoustic memory for another day  .

cheers,

AJ
Loudspeaker manufacturer

What's the problem with double-blind testing?

Reply #245 –
Quote from: ajinfla on
The test is not of "your perceptions". You could still "perceive" getting cold listening to A and warm listening to B. The test is of hearing perception. You either hear a difference or you don't.

This does not necessarily have to be the case.  When I said earlier that differences may manifest themselves in other ways I wasn't being facetious.
http://en.wikipedia.org/wiki/Synesthesia

Still, if listening to sample A causes a different conscious sensation (whatever that sensation may be and whether or not it can be properly described with language) than sample B, then this can still be demonstrated through an ABX test.

What's the problem with double-blind testing?

Reply #246 –
Quote from: greynol on
This does not necessarily have to be the case.  When I said earlier that differences may manifest themselves in other ways I wasn't being facetious.
http://en.wikipedia.org/wiki/Synesthesia

I thought that was exactly what I implied. So let me restate for clarity. I could care less whether you get warm/cold, whatever, listening to A or B.
Can you "hear" a difference in the sound fields. My interest is acoustics. I'll leave all the "other stuff" of interest to the psychologists.
Loudspeaker manufacturer

What's the problem with double-blind testing?

Reply #247 –
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.

What's the problem with double-blind testing?

Reply #248 –
Quote from: greynol on
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.

Then categorize it as a "all perceptions" test, rather than an "audibility" test, which is what I understand it's purpose to be.
Don't recall an audibility test of codecs asking if whether bitrate change made you feel hotter or colder, even if this was the effect.
Loudspeaker manufacturer

What's the problem with double-blind testing?

Reply #249 –
The point is that there is no reason to put unnecessary limits on what the test is capable of measuring, giving our recent "skeptic" something else to which he can cling.  That we're debating unfounded speculation about what may occur on a subconscious or unconscious level is bad enough.