Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 245361 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What's the problem with double-blind testing?

Reply #100 –
Quote
Your objection is invalid.  [...]

Curiously enough, it has been shown in very graphic ways (i.e. by doing it) that one of the most effective ways of training on simple signals is to just decrease the probe signal [{POST_SNAPBACK}][/a]


You're right, my example is the opposite of what I wanted to illustrate.
My point was that in some way, training oneself = bringing the process from the "reaction" realm into the "decision" realm. But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction. If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

As I use to tell in french forums, I've never seen a hifi cable saleseman arguing "Buy this cable, I guarantee you'll never realize that it improves your high fidelity system" !

Quote
I don't know now, if this has been tried already for good  (or even medium/low quality) lossy vs. CD.[a href="index.php?act=findpost&pid=341272"][{POST_SNAPBACK}][/a]


It's currently done with power cables. Each listener has got 7 days to give one answer in a A/XYZ test (is X, Y or Z equal to A ?).
[a href="http://www.hifiwigwam.com/forum1/1614.html]http://www.hifiwigwam.com/forum1/1614.html[/url]

What's the problem with double-blind testing?

Reply #101 –
Quote
If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

This is one of the main points I have been trying to address. Just because some perception is unconscious does NOT mean that it cannot affect one's enjoyment. It most certainly can.

Long term ABX approaches might be better at getting at it.

What's the problem with double-blind testing?

Reply #102 –
Quote
Quote
If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

This is one of the main points I have been trying to address. Just because some perception is unconscious does NOT mean that it cannot affect one's enjoyment. It most certainly can.

Long term ABX approaches might be better at getting at it.
[a href="index.php?act=findpost&pid=341374"][{POST_SNAPBACK}][/a]



I have to agree with this to some extent. If people can think they are "guessing" and be getting 18/20 on a binomial choice, (not one, or two, but enough to be way, way outside the joint or entire distribution by lots and lots of sigma no matter how you cut it), one can't argue that they are unaffected.

On the other hands, they ARE getting the right answer in the test, too, so you can't use that to argue that the test does not work.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #103 –
Quote
Quote
If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

This is one of the main points I have been trying to address. Just because some perception is unconscious does NOT mean that it cannot affect one's enjoyment. It most certainly can. [a href="index.php?act=findpost&pid=341374"][{POST_SNAPBACK}][/a]


The difference is between an unconcious perception, and a perception with no concious effects.
The first can affect enjoyment, thus can be ABXed, if enough effort is put into the test.
The second can't affect enjoyment and can't be ABXed.

What's the problem with double-blind testing?

Reply #104 –
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction. If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.

Quote
The difference is between an unconcious perception, and a perception with no concious effects.


These are the same. It is not understood to what extent unconscious percepts affect experience. They might in quite subtle ways.

What's the problem with double-blind testing?

Reply #105 –
Quote
My point is simple, if there is no detectable remains of an original signal at the auditory nerve, then nothing the cortex does can recover what isn't there in the first place.


If the auditory cortex is involved in inferential processing, then the more degraded the signal, the harder it is to infer acoustic properties of it. An example is the missing fundamental phenomena. That is a clear example of the cortex recovering (i.e., through inference) information that is not present in the stimulus.

What's the problem with double-blind testing?

Reply #106 –
Quote
If the auditory cortex is involved in inferential processing, then the more degraded the signal, the harder it is to infer acoustic properties of it. An example is the missing fundamental phenomena. That is a clear example of the cortex recovering (i.e., through inference) information that is not present in the stimulus.
[a href="index.php?act=findpost&pid=342343"][{POST_SNAPBACK}][/a]


Irrelevant. You've reversed stimulus and probe.

The NOISE is below threshold, not the SIGNAL.

There is nothing "missing" to work on.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #107 –
Reversed stimulus and probe? This phenomenon just refers to the percept associated with a stimulus that requires unconscious inference. This has nothing to with noise, or probe stimuli. I am addressing your claim that the auditory cortex cannot recover missing information from a stimulus. Perhaps you only mean that in a very limited sense, but as you write it, it is relatively sweeping, and incorrect.

The reason I point this out is because you seem to disregard the importance of cortical processing (and instead emphasize peripheral processes), which is curious since we are talking about decision processes used in ABX tests which are by definition not just perceptual.

What's the problem with double-blind testing?

Reply #108 –
Quote
Reversed stimulus and probe? This phenomenon just refers to the percept associated with a stimulus that requires unconscious inference. This has nothing to with noise, or probe stimuli. I am addressing your claim that the auditory cortex cannot recover missing information from a stimulus. Perhaps you only mean that in a very limited sense, but as you write it, it is relatively sweeping, and incorrect.
[a href="index.php?act=findpost&pid=342363"][{POST_SNAPBACK}][/a]


In the future, you should learn the terminology before you criticize.

In my point about masking, the information on the MASKED signal, which is not the music signal, but is rather the noise (coding otherwise). This noise is not expected and is not present in the stimulus (the music) but is the interfering (probe) signal.

The information about these masked signals is not distinguishable from the information available at the far end of the auditory nerve.  My statement is qualified, is as sweeping as appropriate, and given all results from the psychometric community, precisely correct.

You insist on "inference" in situations which no inference is being made. You can start there with your presumptions. The "inference" is only going to occur if you attempt to hear noise, and then you have exactly the same problem with or without noise.

That, in and of itself, shows fully the internal contradiction inherent in your presumptions.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #109 –
In the future you should learn the appropriate context in which you make your arguments.

You clearly have little understanding of the role of unconscious inference in perception, and you instead interpret everything through your understanding of the audibility of masked noise in the creation of audio codecs (without realizing that's not what I'm talking about). Granted, this is one element of the debate here, but only one among many. The more general point that you fail to get is that decision processes involved in ABX judgments can be oblivious to information processing issues not well measured by conscious judgments. I have provided abundant evidence showing this principle in a variety of ways/domains.

I might add that you have provided no evidence supporting any of your positions.

Quote
The "inference" is only going to occur if you attempt to hear noise, and then you have exactly the same problem with or without noise.


No. Unconscious inference is a basic feature of low level auditory and visual processing, and is independent of conscious behavior. As a signal is encoded, characteristics are inferred, and these inferences vary as a function of the stimulus characteristics and the expectations of the system. For example, light information on the retina is quite impoverished compared to the eventual percept that people experience. We construct much of what we perceive. It has little-to-nothing to do with what people "attempt" to do. You are misunderstanding this use of the term "inference."

The missing fundamental phenomenon has nothing to do with masked noise, nor expectation, other than the "expectation" pitch perception systems have when encountering a complex sound (which is not conscious). People perceive a fundamental frequency in degraded spectral information that does not contain a fundamental. You claimed that the auditory system cannot recover information that is not there, and this is without a doubt false. Instead of acknowledging that, you just start talking about masked signals. It's non sequitur.

You also don't address the issue that there might be metabolic costs associated with processing noisy signals, other than just saying that can't be. Where's your falsifying evidence? I've shown that there are specialized neural systems that deal with degraded stimuli in the visual system (which require extra energy to implement), and all you can say is that because the noise is above threshold, there is an important distinction to be had with regard to metabolic costs of neural processing. I'm not convinced of that at all, and I don't think there is any clear research supporting it.

We do seem to agree that different implementations of the ABX paradigm could better capture long-term effects of processing on listeners' experiences. But given how ABX tests are currently used to test listeners' discriminations, tasks involving motor responses might capture differences that cognitive decision (i.e., verbal report) tasks don't capture.

Other than that, I don't think we are accomplishing anything by continuing. I think you have made good points about the role of peripheral hearing processes in the creation of lossy codecs, but you don't have a familiarity with psychophysics or cognitive science research that examines the full chain of processing from sound to percept, and this is causing us to argue in circles.

What's the problem with double-blind testing?

Reply #110 –
Quote
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.


Yes, but I was looking for an audio example.

Quote
Quote
The difference is between an unconcious perception, and a perception with no concious effects.


These are the same. It is not understood to what extent unconscious percepts affect experience. They might in quite subtle ways.
[a href="index.php?act=findpost&pid=342342"][{POST_SNAPBACK}][/a]


I don't mean the same thing.
A sound without concious effects would be a sound that we didn't pay attention to, and that don't affect our behaviour. It was not concious when it was perceived, and then had no further effect at all.
But an uncoucious sound can sometimes have concious effects. I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre). If we don't have the "absolute ear", we are not concious that the pitch is higher (about half a tone higher), but we find the vocal expession of the actors less "serious". The whole movie can seem less dramatic. We can be concious of this effect, while not being concious of its cause.

In an ABX test, if we try, several monthes apart, to recognize the pitch of the movie, we usually fail (exept some rare musicians who practice every day, and can notice the scale difference in the music score). But if we concentrate on the voice timbre, we can succeed. This is an example of stimulus with a concious side (the voice timbre), and an unconcious side (the pitch).

This way, if a sound difference can lead, by unconcious ways, to a perceived difference in quality for a given listener, there is always a possibility to ABX it by decision, granted that the decision is made on the right perception.
In last resort, the test must be run in the exact conditions that lead the listener to realize the difference in quality.
To tell "this seems to sound better to me" is a decision process in itself. If it only comes after several monthes of listening experience, then it will always be possible, though impractical, to ABX it in several monthes per trial.

What's the problem with double-blind testing?

Reply #111 –
Quote
Quote
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.


Yes, but I was looking for an audio example.


I did provide two studies showing that people can be unable to discriminate audio stimuli, but brain reactions were different. David has also linked to an article showing how phase correction occurred in synchronized motor movements due to unconsciously perceived elements in an auditory stimulus. Anyway, we are going to do the critical specific experiment I think you want.

Quote
A sound without concious effects would be a sound that we didn't pay attention to, and that don't affect our behaviour. It was not concious when it was perceived, and then had no further effect at all.
But an uncoucious sound can sometimes have concious effects.


And an unconscious sound can have unconscious effects. Careful about privileging conscious effects though, that is a bit of an illusion. Most of what our brains are doing is unconscious. But your example is good, and shows that long term effects need long term testing strategies (to extend as much as possible the sensitivity of decison-based tests).

Quote
This way, if a sound difference can lead, by unconcious ways, to a perceived difference in quality for a given listener, there is always a possibility to ABX it by decision, granted that the decision is made on the right perception


I agree that having people attend to different aspects of stimuli can affect ABX results in important ways. But that doesn't mean any two stimuli, no matter how slight the difference, can be tested with a decision based test. Many experiments show how motor tasks reveal perceptual differences where verbal tasks fail. ABX-minded people should be open to this possibility - it's not the end of the world, and despite what some folks here would have you believe, it is entirely possible in the context of lossy codecs!

Did I mention that I was NOT an audiophile? I don't care what the ultimate truth is - I have no investment emotionally, materially, or otherwise. I record my own music 16/44.1 for godsakes. But some people do have motivations to believe what they believe beyond plain curiosity. They should be prepared to revise their view I think. I am.

What's the problem with double-blind testing?

Reply #112 –
Quote
In the future you should learn the appropriate context in which you make your arguments.

The context of these remarks is the idea, the unproven, extraordinary assertion that perceptual codecs make people somehow have to listen harder.
Quote
You clearly have little understanding of the role of unconscious inference in perception,

You have no basis whatsoever to make that accusation.  As shown below, you are attempting to escape from the context of this discussion, and introduce issues irrelevant to this context.  You have shown no evidence for the idea that percpetual codecs make listeners work harder, and when compelling evidence has been handed you, you have attempted to change the subject, while making egregious professional accusations.

Quote
and you instead interpret everything through your understanding of the audibility of masked noise in the creation of audio codecs (without realizing that's not what I'm talking about).

The context of this discussion is the contention that somehow perceptual codecs cause listeners to  work harder.  You are attempting to escape from context./
Quote
Granted, this is one element of the debate here, but only one among many.

It is the context of this discussion. You may wish to introduce other issues, but I am continuing to examine your original, unproven, and extraordinary claim.  Before we can move on to more of your theories and contentions, you must prove your extraordinary claim, and you have yet to produce either a relevant argument or any evidence whatsoever of your claim.


Quote
The more general point that you fail to get is that decision processes involved in ABX judgments can be oblivious to information processing issues not well measured by conscious judgments. I have provided abundant evidence showing this principle in a variety of ways/domains.


Oddly enough, now you're talking about ABX tests, even though the context is DBT's.

Furthermore, it is I who has referred informally to data that shows, incontrovertably, that DBT judgements can easily and trivially show evidence of more than concious judgement.  Therefore, your accusation is preposterous, contentious, and completely false.

Furthermore, I have provided evidence in a germane setting, that of an audio DBT, wherein subjects thought they heard nothing, but did, in fact, get very accurate identifications.

You, on the other hand, have introduced discussions of tests where interferers are very far above threshold, rather than at or below threshold, which, at least for most audio testing, wherein it is attempted to discover any difference.  Your results may be accurate, but they are simply not germane, and do not apply to testing wherein the interfering signal is at or below threshold.

Quote
I might add that you have provided no evidence supporting any of your positions.

You are supporting the extraordinary position, not I. Do not attempt to shift the burden.

It is you who owe us evidence, lots of evidence, testable, verifiable, falsifiable evidence that can be repeated by others.  Evidence, that is, for the contention IN THE CONTEXT OF THIS THREAD, not for some other attempt to evade the issue as here stated.
Quote
No. Unconscious inference is a basic feature of low level auditory and visual processing, and is independent of conscious behavior.

First, we are talking about audio.  So, let's stick to audio.

Second, you appear to be claiming that the organ of corti does "inference". That is the "low level auditory processing" that actually exists.  What evidence for "inference" can you show for the organ of Corti?

Let's start there. Because my point about masked signals applies at that level.

If you're talking about once you're past the periphery, well, certainly many things happen. One of the things made clear by much subjective testing is that training can alter the concious (and, since subjects can clearly obtain some information that they can not verbalize and regard as "guessing" even though it's not, unconcious as well) response of everything once we leave the auditory nerve.

That's not in doubt.  It does not, however, impact the validity of DBT's in any fashion.

Your assertion, however, exactly requires that information be discernable at the end of the auditory nerve.  When a signal is below absolute threshold, or masked below tone-masking-noise thresholds, there is no evidence that any discernable information exists regarding such a signal, and there is nothing for this unconcious process to use.

THAT is my point.
Quote
[a href="index.php?act=findpost&pid=342482"][{POST_SNAPBACK}][/a]


As I haven't said the missing fundamental had anything at all to do with masking, your reply is simply disingenious and deceptive. The harmonics that are processed into the missing fundamental are very far above threshold, generally, thank you, so why do you introduce the issue at all? 

Your fallacy of the excluded middle, in suggesting my position of "no metabolic cost" is based entirely on your attempt to evade the context, which is the claim that there is a metabolic cost for listening to perceptual codecs.  There is undoubedly metabolic cost for hearing speech in noise, etc, that news is as old as the hills, and was determined before convenient things like PET allowed us to measure it directly.

That, however, is irrelevant in its entirety to the issue of the "effort' required to hear the music, or some alleged missing information, from a perceptual coder.

You failed to notice that I did point out that if one EXPECTS noise, and thereby tries to find auditory features related to the noise, that that EXPECTATION may indeed cause more work.

finally:
Quote
but you don't have a familiarity with psychophysics or cognitive science research that examines the full chain of processing from sound to percept, and this is causing us to argue in circles.

You have absolutely no grounds to make this egregious professional insult.

You have, time and again, avoided context, argued at straw men, engaged in name-calling rather than offer substantive responses, and you still have failed to produce a single shred of evidence, ordinary, let alone extraordinary, for the claim made in this thread, that perceptual codecs (I will cheerfully stipulate at reasonable rates, let's not get into these low-rate noisemakers, please) cause a metabolic load because some process is trying to extract the "missing information".

What have you done? Shifted context. Tried to shift the burden of proof away from your extraordinary claim.  Issued extremely crude, offensive, and absolutely disgusting professional accusations. Fallacies of the excluded middle, attempts at a magician's force question, yes. Evidence? No.

But no evidence. None, at least related to the subject at hand.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #113 –
Quote
Quote
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.


Yes, but I was looking for an audio example.

Quote
Quote
The difference is between an unconcious perception, and a perception with no concious effects.


These are the same. It is not understood to what extent unconscious percepts affect experience. They might in quite subtle ways.
[a href="index.php?act=findpost&pid=342342"][{POST_SNAPBACK}][/a]


I don't mean the same thing.
A sound without concious effects would be a sound that we didn't pay attention to, and that don't affect our behaviour. It was not concious when it was perceived, and then had no further effect at all.
But an uncoucious sound can sometimes have concious effects. I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre). If we don't have the "absolute ear", we are not concious that the pitch is higher (about half a tone higher), but we find the vocal expession of the actors less "serious". The whole movie can seem less dramatic. We can be concious of this effect, while not being concious of its cause.



There is a much more notorious example:  the tendency for a slightly *louder* of two otherwise identical sound presentations, to be perceived/reported as having higher *quality*  (rather than being perceived/reported as louder).

It is a case, like the one you cite, where an *effect* is perceived ('conscious') but its cause is misinterpreted ('unconscious').

What's the problem with double-blind testing?

Reply #114 –
Quote
You have shown no evidence for the idea that percpetual codecs make listeners work harder


If you cannot see the relevance of the many lines of research that speak to this issue, that's too bad. I have explained the idea, shown suggestive evidence that the idea could be right, and have described how it can be tested. I am going to test it. Can't really do much more than that. I'm not attempting to evade any issues - I am merely trying to get you to see the connection between different sorts of research that relates to the problem at hand.

You think I've resorted to name calling? That's weird, because I've done no such thing. I have noted that I think you have a weak grasp of the cognitive science research we are discussing. That's not name calling - that's just me calling it as I  see it. Sorry if that offended you - I just think you are misunderstanding me, and you say stuff that is just wrong.

For example, processing that takes place after the Organ of Corti is properly considered low-level. The nature of that encoding process and the subsequent effects downstream are poorly understood scientifically by anybody. Your professed certainty about auditory encoding demonstrates you aren't familiar with the literature. For instance, you acted like it was preposterous that cortical neurons can act like a linear filter. I showed you an article demonstrating how neural activity in the auditory cortex behaves as a linear filter, and you came up with some bizarre accusation that isn't the kind of filter you mean. It is well established that A1 neurons act like a linear filter, and this is meant, by neuroscientists, in the mathematical sense of the term. That was just one article amongst many demonstrating such a thing. I included that not to divert the discussion, but to refute what was a blatantly incorrect assertion on your part.


Sound localization happens in the auditory cortex, and the mechanisms implementing that function (in the Superior Olives) are considered low-level.

I don't ever change the subject ultimately - I correct you when you make mistakes regarding the various topics under discussion, which, by the way, all relate to my hypothesis (something by definition "not proven"). Somehow you are concerned with one proposal that I made about lossy codecs being harder to process, but really this is secondary to my main point about the limits of ABX testing. You are over-concerned about that detail.

Also, while this thread is about double-blind testing, all that really refers to is the situation where neither the experimenter nor subject are aware of the experimental condition on any give trial. The primary method under discussion is ABX testing, which is a specific method, not a principle. So that's why I am talking about ABX testing - not odd at all I would say.

Quote
It is the context of this discussion. You may wish to introduce other issues, but I am continuing to examine your original, unproven, and extraordinary claim.


Everything I have brought up is relevant, but all you keep claiming is that when noise is below threshold, it cannot affect the encoding process. I disagree. So you can stop saying that now.

When I get a result that shows people are unable to discriminate between two stimuli with different levels of inaudible noise in an ABX paradigm, but they respond differently on a reaction time task, then what will you say? All of my conjecture is geared towards explaining why this result is very possible. Again, if you disagree, fine, but you can stop with the Organ of Corti business, even though I know you love it.

What's the problem with double-blind testing?

Reply #115 –
Quote
I did provide two studies showing that people can be unable to discriminate audio stimuli, but brain reactions were different. David has also linked to an article showing how phase correction occurred in synchronized motor movements due to unconsciously perceived elements in an auditory stimulus. [a href="index.php?act=findpost&pid=342715"][{POST_SNAPBACK}][/a]


Yes, it is always interesting to learn, and I thank you for providing all these informations.

I had a look at the first one that deals with masking ba/da sounds with noise. In fact, the presence of brain activity does not mean that a reaction process is possible, while a decision one is not.
In the article about finger tapping, I just read the first experiment, with the change in tempo (Figure 2). The change has not been conciously perceived, but I don't agree with interpreting the adjustment of the finger tempo as an unconcious perception of it. It is just the permanent mechanism of adjustment that follows the change.
Looking at the diagrams that show the synchronization error during the unaltered sequences (Figure 2 a), we can see that people's synchronization is better that the change in tempo. The error bars in the left part are smaller that +/- 10 ms. Not adjusting the finger tapping in order to follow the change would have eventually lead to a conciously perceived asynchonicity.
My conclusion is that the change was conciously perceived, but wrongly interpreted as a random variation of the finger tempo, that was, as the diagram shows (figure 2b, 2c, 2d) corrected the beat following the detection. Thus hidden among all the other corrections that were made by the subject in order to keep the right tempo.
People were asked if they perceived a change in tempo, and said that this was not the case. But maybe if they had been asked "did you always tap in phase, or did you readjust your tempo at a given time", people would have answered that they had to readjust in b, c, and d case only.
Furthermore, in music class, we are trained, for rhythm dictation, to ignore small deviations, and only report musically relevant information. These people all had a solid musical background. Maybe they actually perceived the change in tempo, but interpreted the question as "did the score went from allegro to presto" (which was indeed not the case), while the question actually ment "was the conducer as accurate as a metronome" (which was not the case).

Quote
And an unconscious sound can have unconscious effects. Careful about privileging conscious effects though, that is a bit of an illusion.
[a href="index.php?act=findpost&pid=342715"][{POST_SNAPBACK}][/a]


Maybe. But in fact I am more interested in debunking false claims about sound quality. We are surrounded by unfounded information about 15,000 € CD players, or "unlistenable" 320 kbps MP3. And all these claims assume that the difference is conciously audible. Showing that this is not the case can be acheived with ABX tests.
Maybe there are also unconcious differences, that work in these cases, but it is already such a hard work to show that there are no "kickass differences" between two interconnect cables, and that silver interconnect cables do not emphasize treble, that I'm not planning to go beyond ABX (or AXY, or any other decision pattern) for the time being.

What's the problem with double-blind testing?

Reply #116 –
Quote
Maybe. But in fact I am more interested in debunking false claims about sound quality. We are surrounded by unfounded information about 15,000 € CD players, or "unlistenable" 320 kbps MP3. And all these claims assume that the difference is conciously audible. Showing that this is not the case can be acheived with ABX tests.


I agree. There are certainly many false claims out there, and ridiculously priced cables and gear are based largely in pseudoscience. I'm totally with you there. All I'm suspecting is that some of this nonsense might be rooted in auditory processing issues that ABX testing doesn't tap into. I have no interest in arguing for something that real audiophile freaks could latch onto. I just think it's an empirical question worthy of study.

Quote
In fact, the presence of brain activity does not mean that a reaction process is possible, while a decision one is not.


You're right - the point of that article is about how sensory and decision processes are dissociated, and the brain is doing different stuff on noisier stimuli even though decision processes are treating them the same.

I think it is a bit of a stretch to claim that the participants in the tempo synchronization study actually perceived the change, and just needed to be asked differently. For one, with studies like these (and with this researcher), you can rest assured it has been replicated before it was published, and that sort of confound would have been dealt with early. But more importantly, the more parsimonious story, scientifically, is that the participants' motor movements were driven by a system that had access to different information about the stimulus than the cognitive/decision system. This is well established in psychophysics research. The Roelof's effect I've talked about before is a good example.

What's the problem with double-blind testing?

Reply #117 –
Quote
Also, psychoacoustic research already found that quick switch, short length stimuli, blind listening tests have proved to be more much more sensitive that casual listening.


Hey KikeG, do you have some reference for this?

I'm by no means claiming it's not true, but I haven't come across this myself, but would very much like to read any relevant papers on the issue.

What's the problem with double-blind testing?

Reply #118 –
I can't cite any papers, but I've read this many times from people knowledgeable in psychoacoustics.

What's the problem with double-blind testing?

Reply #119 –
I'm new to the forums here, but I must say I'm finding the contributions here very, very interesting. I'm an engineer/scientist by trade, developing technology for a living so I'm interested in getting to the bottom of things. Recently I've been investigating hi-fi with the intention of trying to work out what's worth worrying about and what isn't.

I found this discussion very interesting so I thought I'd try to resurrect a long dead argument. I'm sure that no-one is interested in my opinion, not being a cognitive scientist or having much idea about how lossy compression works, but for what it's worth, I found some arguments from both sides convincing and learned a lot.

I don't think it's reasonable to argue with his assertions (and evidence) showing that ABX testing can't pick up on some unconscious things. Even I've heard of reaction times as a method to by pass the decision gubbins to try to get a better handle on what's going on.

So it seems entirely reasonable to suggest that a test methodology for audio compression that does not rely on conscious inference would be interesting, since it might show effects below the threshold at which they are available for conscious inference.

What I'm not clear on, and I don't think can be taken as a 'given' at this stage (before the proposed experiments) is that such effects will occur for the case of compression (at bitrates above 128kbps). I think the examples that have been shown so far show cases in which a stimulus with a clear mechanism for interfering with perception is present. From what I understand, the principles on which audio compression routines are based avoid such mechanism as far as they are known.

There seems to be an assertion that the brain will certainly recognise a compressed signal as degraded, and therefore work to reconstruct it. I'm not convinced that the brain will be able to infer that there is anything to be reconstructed in the sound, and I reckon the understanding of these processes are not well enough understood to be able to predict that there is likely to be a metabolic overhead associated with compressed audio.

So I can see the point of carrying out some experiments, but I don't think that the outcome can be pre-judged.

I think a lot of the negative reaction may have stemmed from the suggestion that inaudible acoustic differences might 'matter' for a listener's overall experience. I really don't see that this is supported by the evidence, and would be music to the snake oil salesperson's ears. If the metabolic overhead is unconscious, it's hardly been shown to be significant or material to the experience.

Anyway - can anyone report with an update? Have any such experiments been carried out?

What's the problem with double-blind testing?

Reply #120 –
Quote from: Pio2001 on
I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre).


Just because the FPS change, there's no necessity for the pitch or timing to change. People working with films and video have been managing this situation for several decades.

This article explains how FPS changes are have been managed for decades so that they don't affect the length or running speed of film or video:

http://www.zerocut.com/tech/pulldown.html

In the old days, the production paths for sound and video were separated, the video or film was altered using pull down techniques, and the audio was added back into the finished product with  no changes. Or, the sound was added back into the finished product with a different FPS from a more origional source.

These days, most video production software manages things like this automagically.  You just specify the FPS of the finished product, and it is produced in accordance with your spec under the covers, as it were.

In short, your example has a rather serious flaw.


What's the problem with double-blind testing?

Reply #121 –
I too found these discussions interesting.  I think one relevant intuition is this.  Someone has just failed an ABX test where he (or she) switches quickly back and forth between two signals.  He says to himself, "Still, there might be a difference between these sources that matters to me.  When I listen to one signal for a long stretch, perhaps in a relaxed way, I might get a different experience from it than from the other signal.  I might hear things in one that I do not hear in the other.  Or I might get greater pleasure from one than the other.  I might not be very good at comparing those experiences, since they are separated in time.  But what matters to me might be that I have certain experiences, not that I know or be able to report that I do." 

The intuition, then, is that something in those signals could make a difference to perception, even if that difference is not itself perceived, or the subject of awareness or report.

That is the skeptical worry, and I have never seen a really convincing response to it.

Mark

What's the problem with double-blind testing?

Reply #122 –
I have an idea as how to test what djcombes and Mark DeB are trying to get tested... There should be a reality show that lasted for a month or so and the contestants would have to listen to a lot of music and they would never know the source and they would always have to grade their experience level, aswell as guess if it came from original source or if it was a compressed sample.

All in all, I'm sure the results would show that there would be no difference in experience grades between original sources and compressed ones. I am sure that somedays the contestants would grade compressed samples better and then other days they would grade the originals better...

Anyway... such a prolonged test would be interresting and would definetly bring more insight and more evidence for this debate.

Maybe some of the hydrogenaudio forum readers are rich enough or influent enough to make this kind of test possible?

EDIT: typos
lame -V 0

What's the problem with double-blind testing?

Reply #123 –
It's a BS argument simply because there are no time limits to ABX testing.  Listen to X for a day and then Y for another day (or whatever period of time makes you happy), and give your answer; repeat the process 16 times regardless of how long it takes.  If you truly perceive a difference it will be apparent in your score.

What's the problem with double-blind testing?

Reply #124 –
Quote from: greynol on
It's a BS argument simply because there are no time limits to ABX testing.  Listen to X for a day and then Y for another day (or whatever period of time makes you happy), and give your answer; repeat the process 16 times regardless of how long it takes.  If you truly perceive a difference it will be apparent in your score.


Not sure what argument you are referring to, but the skeptic in my example is not supposing that he perceives a difference, only that there might be a difference in what he perceives.

Consider:

(1) You perceive a as X and you perceive b as Y, and X is different from Y.
(2) You perceive that a is different from b.

There is a logical difference between these.  In (2), "different" falls within the scope of "perceives."  In (1), it does not.

The skeptic does not think that (1) implies (2).

EDIT: typo.