Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 248895 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What's the problem with double-blind testing?

Reply #300 –
Quote from: Woodinville on
What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc.


Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?

Are the tests you refer to ABX?  If ABX tests aren't going to be good at revealing the existence of said effects, isn't this begging the question?

What's the problem with double-blind testing?

Reply #301 –
Quote from: Soap on
The thread is already moving in this general direction, and I beg it continue:

I propose that the only way to end this circle jerk is to limit the conversation to one voice at a time.  It is when there are four or five (or ten) responses to address that we see the most picking and choosing, bobbing and weaving.  If you feel, as I do, that this has gone on for days longer than necessary, I beg of you to refrain from addressing a point until that time the last point has been satisfactorily answered.

this is me agreeing

What's the problem with double-blind testing?

Reply #302 –
Quote from: Mark DeB on
Quote from: Woodinville on
What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc.


Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?

Are the tests you refer to ABX?  If ABX tests aren't going to be good at revealing the existence of said effects, isn't this begging the question?


If your "context-dependent effects" interfere with double blind tests they would also interfere with a normal (not blind) comparison as well. So why are you picking on ABX?!

What's the problem with double-blind testing?

Reply #303 –
Quote from: Mark DeB on
Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?


Since you haven't actually suggested anything particular and even resist to do so, although you act like you had, let me fill in the formal blank:

I bought a new toilet paper brand last month, I didn't perceive any difference in everyday usage until today, when you brought it to my attention again while I was thinking about an example for your nonsense. I can't ABX the rolls, they look exactly identical to the old brand. The only difference is the outer packaging and that they come from different factories. Well, since 2 weeks of thorough long-term usage I cannot report any conscious difference. There is always a lot of peaceful emotion involved when I take a shit, but the time spans are to long to get a clear picture of any difference. Exactly as you want it.  Still the papers are different of course, but since I haven't perceived it, I suggest you install a MRI in my toilet to evaluate the highly probable chance that I had indeed "different experiences" but was just unable to become aware of it. Your "proof" might involve claiming that in 78% of rounds, when I used paper A, region xyz of my brain resulted in green color on the screen, which else was orange. Is that what you want?

If it is, I reject it and will call it uncorrelated noise at position xyz with no causal effect onto my mental state. How do you want to react, what's your truth condition? You might want to increase the number of rounds to 1000 and then claim rock hard evidence after running an automated analysis on the MRI data. Well if I really could not perceive a difference after so many trials, I could not care less. Maybe you just confused it with a dead salmon (great paper!)? Do you have any truth condition other than things as brain scans, that you basically cannot even explain?

PS Soap, I don't believe that your proposal is going to work. As long as "the one" would answer with n>1 sentences Mark would be able to continue ignoring any n-1. So why not just take this with humor? The thread is a juggernaut already, anyway, and circling as you say.

What's the problem with double-blind testing?

Reply #305 –
Quote from: rpp3po on
Maybe you just confused it with a dead salmon (great paper!)? Do you have any truth condition other than things as brain scans, that you basically cannot even explain?


Wow, this makes my posters look lame by comparison! Brilliant!

Quote
PS Soap, I don't believe that your proposal is going to work. As long as "the one" would answer with n>1 sentences Mark would be able to continue ignoring any n-1. So why not just take this with humor? The thread is a juggernaut already, anyway, and circling as you say.


Right. I don't understand why anybody bothers to argue any more. This probably is some sort of show put up for his friends somewhere. Or he could be serious about it, I've met intelligent, articulate people who really did behave like this--a philosopher and two lawyers, as it happens. The two lawyers are good friends, and one is as smart as anybody I know. Didn't help her much: she claims that a ball, dropped in a (sealed) airplane, falls backwards. I tried all sorts of arguments: "which frame of reference does it fall back in?", "what force causes it to fall back?", "how does it know the thing is moving?", "you yourself could not tell whether the plane is moving or not if it's not accelerating, how could the ball?", "I am a physicist and I am telling you, it doesn't!!", to no avail. She still seems convinced that a ball dropped in a plane will move backwards (slightly, which is why we can't see it...). And no she's not winding me up.

So either we're dealing with a similar case, or our "skeptic" is in fact winding everybody up, and doing it suspiciously professionally (notice calm composure, mostly ignoring ridicule, apparently reasonable arguments, impressive looking references etc--all this looks very much designed to appeal to an outside audience and make it look like he does in fact have a point). I vote for the second.

What's the problem with double-blind testing?

Reply #306 –
I have a friend who used to go to a medium (or card-reader or whatever it was) and it's been pretty much years now that I've slowly made her skeptical to it, but once in a while she still comes telling me about her sister's bad luck or some sort of bad energy somewhere. The way humans have evolved, we have to make a special effort not to fall for this stuff. I almost bought an ultrasonic roach repellent today!

What's the problem with double-blind testing?

Reply #307 –
Quote from: andy o on
I almost bought an ultrasonic roach repellent today!

You should have. If it didn't work on roaches, you could always sell it for $4k on Audiogon as an "audiophile" super tweeter. They can hear up into the Ghz...and will buy anything.

cheers,

AJ
Loudspeaker manufacturer

What's the problem with double-blind testing?

Reply #308 –
Quote from: Mark DeB on
Quote from: Woodinville on
What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc.


Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?

Are the tests you refer to ABX?  If ABX tests aren't going to be good at revealing the existence of said effects, isn't this begging the question?



WHAT effects that won't happen in any test, ABX or not. Please be specific.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #310 –
Quote from: psycho on
I have an idea as how to test what djcombes and Mark DeB are trying to get tested... There should be a reality show that lasted for a month or so and the contestants would have to listen to a lot of music and they would never know the source and they would always have to grade their experience level, aswell as guess if it came from original source or if it was a compressed sample.

All in all, I'm sure the results would show that there would be no difference in experience grades between original sources and compressed ones. I am sure that somedays the contestants would grade compressed samples better and then other days they would grade the originals better...

Anyway... such a prolonged test would be interresting and would definetly bring more insight and more evidence for this debate.
You've just described a long term ABX test. It's been done already and is not what was being suggested by duff. Duff was suggesting that there might be a difference in perception below the threshold of conscious perception. It's a bit angels on the head of a pin (if it can't be perceived without some convoluted test methodology does it matter?), but he might be correct. Since it involves something unconscious, the hypothesis is difficult to investigate. It's necessary to come up with a test method in which the response is not mediated by conscious thought (such as would be required in rating an experience). Previous experiments (not related to music or lossy compression) have demonstrated that under some circumstances a stimulus which is below the threshold of perception (in the ABX sense), can nonetheless have an effect on reaction time (for instance).

I can't work out how you set up a decent test to investigate. You need to be able to translate a listening experience into some kind of motor response without the need for conscious thought. Tricky! I'm guessing that duff hasn't come up with a sensible approach, or he has and found he's wrong - i.e. no evidence for sub-conscious perception of lossy compression . Either way he's not returned to comment.

What's the problem with double-blind testing?

Reply #311 –
Quote from: djcombes on
You need to be able to translate a listening experience into some kind of motor response without the need for conscious thought. Tricky!

What about clicking those buttons in an ABX test app without conscious thoughts?

What's the problem with double-blind testing?

Reply #312 –
Quote from: muaddib on
What about clicking those buttons in an ABX test app without conscious thoughts?
I think the act of comparison involves conscious thinking about it. You'd need something more along the lines of measuring foot tapping response to a catchy tune, and seeing if it was correlated with the level of compression. That's obviously a hopeless example, but I can't think of a good way to design an experiment to check for this.

Which does kind of relegate it to the level of how many angels can dance on the head of the pin.

ETA - I've just re-engaged my sarcasm detector. It was obviously switched off.

What's the problem with double-blind testing?

Reply #313 –
Quote from: djcombes on
Duff was suggesting that there might be a difference in perception below the threshold of conscious perception.
WRT audio, again, based on what evidence exactly?

From his last relevant post on this forum, back in August of 2007:
Quote
That experiment is coming...
Guess what, we're still here waiting.

What's the problem with double-blind testing?

Reply #314 –
Quote from: greynol on
Quote from: djcombes on
Duff was suggesting that there might be a difference in perception below the threshold of conscious perception.
WRT audio, again, based on what evidence exactly?

From his last relevant post on this forum, back in August of 2007:
Quote
That experiment is coming...
Guess what, we're still here waiting.
WRT to audio specifically I seem to remember that there was some reference to an imperceptible change in masking noise level having a measurable effect on perception, even though the two noise levels couldn't be distinguished by ABX.

I agree - we are still waiting. In the absence of any information from duff I think it would be reasonable to infer that any such effect is very small and hard to detect, or that there is no such effect. Or that duff was just full of sh*t, and never had any intention of carrying out experiments into the matter.

;-)

What's the problem with double-blind testing?

Reply #316 –
Quote from: djcombes on
Or that duff was just full of sh*t, and never had any intention of carrying out experiments into the matter.
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies

What's the problem with double-blind testing?

Reply #319 –
Quote from: Kees de Visser on
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies

Assistant professor - what does that mean? Is that the lowest rank of the academic ladder in the US? What you'd call a lecturer in the UK? Does this mean that he is busy teaching and doesn't have time for much research these days?

What's the problem with double-blind testing?

Reply #320 –
Quote from: djcombes on
Quote from: Kees de Visser on
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies

Assistant professor - what does that mean? Is that the lowest rank of the academic ladder in the US? What you'd call a lecturer in the UK? Does this mean that he is busy teaching and doesn't have time for much research these days?


Yes, similar to "lecturer" in the UK. In the US the progression is Assistant > Associate > "Full" professor. All depends on disciplines and universities, but at my "shop" the assistants do very little teaching and a lot of research.  We have to offer only one semester a year of teaching to attract the top candidates. But again, this varies greatly by discipline, even within the same university.

edit: and in US, the Asst/Assoc/Full ranks are typically associated with tenure track. We also have lecturer type ranks that are typically non-tenure earning.
 

What's the problem with double-blind testing?

Reply #321 –
It's probably worth noting that the tenure track can be 25+ years long in some universities. I knew a guy who joined a dept as asst/associate professor (I forget which) in the mid 80s, and only got his tenure after I graduated in '02.

It's a really messed up system and I wouldn't belittle any professor who wasn't tenured.

What's the problem with double-blind testing?

Reply #322 –
I certainly wouldn't belittle an associate professor either, though I do question whether this person should be taken any more seriously than anyone else regarding the subject material simply because he's an Associate Professor in the Department of Communication Studies.

What's the problem with double-blind testing?

Reply #323 –
Quote from: Axon on
It's a really messed up system and I wouldn't belittle any professor who wasn't tenured.


agree, and I certainly wasn't dissing anyone without tenure....just pointing out the title differences and what they typically mean in the U.S.  Heck, 99.99999% of all employees in the world do NOT have tenure.

edit: and yes, professors of any stripe may or may not know more about a subject than other people (even in their purported specialty field).

What's the problem with double-blind testing?

Reply #324 –
Quote from: Kees de Visser on
Quote from: djcombes on
Or that duff was just full of sh*t, and never had any intention of carrying out experiments into the matter.
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies


“Duffman is thrusting in the direction of the problem!” *




(*Simpsons quote)