Skip to main content ↑ ↓
Topic: DBT Is Flawed Because Bob Stuart Says So (Read 22765 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

DBT Is Flawed Because Bob Stuart Says So

The trouble with ABX testing is always the fact if you want it to be truly objective you have to collect a large enough number of subjective test results to start with, and then apply statistics in order to filter out the unwanted subjectiveness part. This is because every individual test result always has a significant chance of being highly inaccurate, so you need lots of people to participate if you want the final conclusions to be reliable ones.
Often, if not practically always, it is too time consuming and/or too uneconomical to conduct an ABX test properly, i.e. in such way that expectation bias doesn't creep in through the back door etcetera. Experts in auditory neuroscience and psychoacoustics have gathered experimental evidence which appears to indicate the following. Humans who remember different things are perfectly capable of hearing the same sounds differently as a result of remembering different things. For example, Bob Stuart of Meridian, who has a Ph.D in neuroscience, believes that it is perfectly possible for a person to not hear a specific detail in a piece of music when it is played back on one particular system "A", then to discover this specific detail by listening to the same music again on a better, more resolving system "B" next, and then, finally, to turn back to the previous system "A" and always hear this detail on system "A" even though the detail could previously not be heard on system "A". Moreover, Bob Stuart believes rapid switching between sounds can inevitably cause humans to perceive sound objects differently.
 

DBT Is Flawed Because Bob Stuart Says So

Reply #1 –
I'm sorry, but how is any of this "the trouble with abx testing"? Rather this is the trouble with determining what is right and wrong when differences are subtle, and abx testing is the tool that makes this possible.

DBT Is Flawed Because Bob Stuart Says So

Reply #2 –
One trouble is the misapplication of ABX results.  The other is the silly argument that there is a problem with the method of testing because the results can be misapplied.

Regarding Bob Stuart, except for the fast switching part, sure, sounds reasonable.  So?
Is 24-bit/192kHz good enough for your lo-fi vinyl, or do you need 32/384?

DBT Is Flawed Because Bob Stuart Says So

Reply #3 –
Quote from: pdq on
I'm sorry, but how is any of this "the trouble with abx testing"? Rather this is the trouble with determining what is right and wrong when differences are subtle, and abx testing is the tool that makes this possible.

Yes, it is the tool that makes this possible, at least in theory. However, in practice, it is typically more often than not extremely difficult to find enough people who will participate in such testings.

Quote from: greynol on
One trouble is the misapplication of ABX results.  The other is the silly argument that there is a problem with the method of testing because the results can be misapplied.

Regarding Bob Stuart, except for the fast switching part, sure, sounds reasonable.  So?

I don't know why you disagree on the fast switching part, because there appears to be substantial evidence to support it. Experts claim a natural process normally occurs in the brain while we listen; it gradually builds a map of sound objects. When combined with auditory stimulae which the brain can manage to logically fit together with it, this map causes our perception to adapt accordingly. With rapid switching, this natural process does not occur.
My point is, if even world's best experts fail to come to an agreement on what's required to be able to design a reliable ABX test, which IMHO appears to be the case BTW, then who are we to judge those who are skeptical towards the objectivists?

DBT Is Flawed Because Bob Stuart Says So

Reply #4 –
Quote from: audioclaudio on
I don't know why you disagree on the fast switching part, because there appears to be substantial evidence to support it.

I guess I'd have to take your word for it on that one.

Quote
With rapid switching, this natural process does not occur.

You seem to misunderstand an extremely important part about ABX testing: ABX can only be used to objectively demonstrate differences.  A failed test does not prove two things sound the same to all people under all circumstances for all time.

Quote
My point is, if even world's best experts fail to come to an agreement on what's required to be able to design a reliable ABX test, which IMHO appears to be the case BTW, then who are we to judge those who are skeptical towards the objectivists?

I can come up with all sorts of fanciful things that you cannot disprove. Will you be giving me the same credibility?
Is 24-bit/192kHz good enough for your lo-fi vinyl, or do you need 32/384?

DBT Is Flawed Because Bob Stuart Says So

Reply #5 –
Many of these "problems" apply to any listening test where you must compare two sources and decide what you think about them - and that includes sighted listening tests! Therefore it's wrong to raise these "problems" as criticisms of ABX/DBT.

DBT Is Flawed Because Bob Stuart Says So

Reply #6 –
Quote from: audioclaudio on
Quote from: pdq on
I'm sorry, but how is any of this "the trouble with abx testing"? Rather this is the trouble with determining what is right and wrong when differences are subtle, and abx testing is the tool that makes this possible.

Yes, it is the tool that makes this possible, at least in theory. However, in practice, it is typically more often than not extremely difficult to find enough people who will participate in such testings.

It's actually quite simple. Person A claims to be able to hear something. If he then voluntarily takes an ABX test for, say, a dozen or so trials, he either has provided statistically significant results that support his claim, or else his claim is unsubstantiated (though not necessarily false). Unsubstantiated claims remain just that, unsubstantiated.

DBT Is Flawed Because Bob Stuart Says So

Reply #7 –
Quote from: greynol on
Quote from: audioclaudio on
I don't know why you disagree on the fast switching part, because there appears to be substantial evidence to support it.

I guess I'd have to take your word for it on that one.

Well, I said "there appears to be" because I am no expert on the matter. There was a TAS interview by Robert Harley with Bob Stuart about it, which was published some years ago.
Quote from: greynol on
Quote
With rapid switching, this natural process does not occur.

You seem to misunderstand an extremely important part about ABX testing: ABX can only be used to objectively demonstrate differences.  A failed test does not prove two things sound the same to all people under all circumstances for all time.

No, I understand this and I am not trying to argue against it. However, it implies that, if Bob Stuart is correct about the consequence of rapid switching, ABX might not be capable of demonstrating differences which could be easily noticed in listening sessions during which the natural process would be given the time and chance to occur, so, again, this would indicate ABX is generally very often too time consuming and/or too economically unfeasible.
Quote from: greynol on
Quote
My point is, if even world's best experts fail to come to an agreement on what's required to be able to design a reliable ABX test, which IMHO appears to be the case BTW, then who are we to judge those who are skeptical towards the objectivists?

I can come up with all sorts of fanciful things that you cannot disprove. Will you be giving me the same credibility?

This is exactly the problem I was actually trying to describe. If nobody had looked for planets outside the Solar System, nobody would have ever discovered any. That is what science is, or ought to be, all about in the first place. Without science, there can be no objectiveness. If I have to name one thing I have in common with Bob Stuart, it's the fact we both don't believe in Black Magic (and we both love music). 
Quote from: 2Bdecided on
Many of these "problems" apply to any listening test where you must compare two sources and decide what you think about them - and that includes sighted listening tests! Therefore it's wrong to raise these "problems" as criticisms of ABX/DBT.

Yes, I believe that is entirely correct. It partly explains why I usually prefer not to partake in any listening tests of any kind whatsoever. I am in the same camp as Robert Harley about them. Meaning, differences between sounds are magnified when I listen purely for pleasure rather than for making comparisons. I know, it may seem a little bit strange perhaps, but nevertheless, it's true.
Quote from: pdq on
Quote from: audioclaudio on
Quote from: pdq on
I'm sorry, but how is any of this "the trouble with abx testing"? Rather this is the trouble with determining what is right and wrong when differences are subtle, and abx testing is the tool that makes this possible.

Yes, it is the tool that makes this possible, at least in theory. However, in practice, it is typically more often than not extremely difficult to find enough people who will participate in such testings.

It's actually quite simple. Person A claims to be able to hear something. If he then voluntarily takes an ABX test for, say, a dozen or so trials, he either has provided statistically significant results that support his claim, or else his claim is unsubstantiated (though not necessarily false). Unsubstantiated claims remain just that, unsubstantiated.

Obviously. Like I said though, I suspect that alot of these "unsubstantiated claims" remain unsubstantiated only because ABX itself is what can severely impair one's ability to "hear something", and that this could perhaps help to explain why alot of knowledgeable people, including a fair number of experts (there's that dirty word again...), are leaning more towards, or at least are not so very skeptical towards, the subjectiveness part of the picture than some others might find logical.

DBT Is Flawed Because Bob Stuart Says So

Reply #8 –
Quote from: audioclaudio on
If nobody had looked for planets outside the Solar System, nobody would have ever discovered any. That is what science is, or ought to be, all about in the first place.

How does requiring objective confirmation of claims prevent curiosity?

Quote
Meaning, differences between sounds are magnified when I listen purely for pleasure rather than for making comparisons. I know, it may seem a little bit strange perhaps, but nevertheless, it's true.

The needle on my placebo effect meter just pegged.

Funny how otherwise obvious differences vanish under scrutiny, isn't it?

Quote
ABX itself is what can severely impair one's ability to "hear something"


That this is it true is highly questionable.  Next time you present a "fact" I recommend supporting it with evidence rather than anecdotes.

While there may be a limit on one's patience and/or resources which may hinder proper ABX testing, ABX testing does not require arbitrary time limits.  This tired "time limit" excuse still doesn't change where the burden of proof falls when pitting theories that are corroborated with little or no evidence against theories that are corroborated with evidence or are falsifiable but have yet to be falsified.

Regarding "experts" I can find some who say global warming is not real.  I don't know of any in this camp who don't also have some kind of incentive to hold this position, however.  You might consider whether the same can't be said about some of these neuroscientists.  Of course some kind of vested interest doesn't necessarily mean that they are wrong either.  The point is that expert testimony can be cherry-picked.
Is 24-bit/192kHz good enough for your lo-fi vinyl, or do you need 32/384?

DBT Is Flawed Because Bob Stuart Says So

Reply #9 –
Quote from: audioclaudio on
Obviously. Like I said though, I suspect that alot of these "unsubstantiated claims" remain unsubstantiated only because ABX itself is what can severely impair one's ability to "hear something", and that this could perhaps help to explain why alot of knowledgeable people, including a fair number of experts (there's that dirty word again...), are leaning more towards, or at least are not so very skeptical towards, the subjectiveness part of the picture than some others might find logical.

Let's say that someone claims to be able to hear a difference, but only under conditions A, B and C. Therefore ABX won't work.

Very sinple, do the ABX test under conditions A, B and C and let's see what you can do. If you say that one of the conditions is that it takes you an hour or two to hear the difference, fine. Take your time. Get back to me when you are done. I will still need to see ABX test results.

DBT Is Flawed Because Bob Stuart Says So

Reply #10 –
My sincerest apologies for a mistake I made in my previous post.  For those who might not have seen it in original form, I had interpreted "differences between sounds are magnified when I listen purely for pleasure [...] it's true" as universally applying to all people.

FWIW I am currently performing MUSHRA tests under contract and can absolutely assure you with no uncertainty that there are many details that would have otherwise gone overlooked had I been listening just for pleasure.  Perhaps if you were ever to stake your reputation and put a decent wage on the line by attempting to demonstrate that you can hear subtle differences between two things you would likely find the same thing.  Or, perhaps you have comparable experience and have simply come to the opposite conclusion. If so I'd like to hear about it.
Is 24-bit/192kHz good enough for your lo-fi vinyl, or do you need 32/384?

DBT Is Flawed Because Bob Stuart Says So

Reply #11 –
Quote from: greynol on
Quote
If nobody had looked for planets outside the Solar System, nobody would have ever discovered any. That is what science is, or ought to be, all about in the first place.

How does requiring objective confirmation of claims prevent curiosity?

Those who did keep on looking for planets outside the Solar System were about to give up on it because nobody wanted to believe them and therefore nobody cared to invest in "that kind of research". As soon as the objective confirmation hit the news like a bombshell, new questions were raised, as people from all over the world all of a sudden did become very curious about these planets.
Quote from: greynol on
Quote from: audioclaudio on
ABX itself is what can severely impair one's ability to "hear something"

There you go again.

While there may be a limit on one's patience and/or resources which may hinder proper ABX testing, ABX testing does not require arbitrary time limits.  This tired "time limit" excuse still doesn't change how the burden of proof falls when dealing with falsifiable claims vs. unfalsifiable claims.

Regarding "experts" I can find some who say global warming is not real.  I don't know of any in this camp who don't also have some kind of incentive to hold this position, however.  You might consider whether the same can't be said about some of these neuroscientists.  Of course some kind of vested interest doesn't necessarily mean that they are wrong either.  The point is that expert testimony can be cherry-picked.

I agree. I never said ABX testing is an invalid way of testing. If an ABX test fails, all that means is further testing is in order. But demanding further testing from a group of people is one thing. Getting that same group of people to become curious enough to participate in such further testing is still usually a whole lot less doable IMO. It depends on how big the reward is versus how much effort it takes, I guess. Personally, I, find rapid switching in an ABX test more stressful than almost anything.
Quote from: pdq on
Quote from: audioclaudio on
Obviously. Like I said though, I suspect that alot of these "unsubstantiated claims" remain unsubstantiated only because ABX itself is what can severely impair one's ability to "hear something", and that this could perhaps help to explain why alot of knowledgeable people, including a fair number of experts (there's that dirty word again...), are leaning more towards, or at least are not so very skeptical towards, the subjectiveness part of the picture than some others might find logical.

Let's say that someone claims to be able to hear a difference, but only under conditions A, B and C. Therefore ABX won't work.

Very sinple, do the ABX test under conditions A, B and C and let's see what you can do. If you say that one of the conditions is that it takes you an hour or two to hear the difference, fine. Take your time. Get back to me when you are done. I will still need to see ABX test results.

What if condition A says I can only hear a difference when listening purely for pleasure, condition B says I can't listen purely for pleasure while I'm being forced to listen for differences instead, and condition C says ABX won't work because ABX inevitably forces me to listen for differences? That would be a deadlock situation. It's only a hypothesis, sure. But then, how exactly were you going to prove that the difference I am hearing is called a hallucination? Long story short, ABX and DBT are very valuable tools. They just ain't always perfect for the job.
Quote from: greynol on
My sincerest apologies for a mistake I made in my previous post.  For those who might not have seen it in original form, I had interpreted "differences between sounds are magnified when I listen purely for pleasure [...] it's true" as universally applying to all people.

FWIW I am currently performing MUSHRA tests under contract and can absolutely assure you with no uncertainty that there are many details that would have otherwise gone overlooked had I been listening just for pleasure.  Perhaps if you were ever to stake your reputation and put a decent wage on the line by attempting to demonstrate that you can hear subtle differences between two things you would likely find the same thing.  Or, perhaps you have comparable experience and have simply come to the opposite conclusion. If so I'd like to hear about it.

No worries, just because your avatar looks like an extremist doesn't also mean I think you are one.  In fact, my first record that I owned as a kid was the 7" 45 rpm single of the Smurf song.
The differences in sound that are magnified when I listen purely for pleasure are not really audible to me in the sense that I can easily describe them, but they do have a big enough impact on things like fatiguingness and involvingness for me to still notice them in the long run. It can take several weeks, and sometimes it even takes months.

DBT Is Flawed Because Bob Stuart Says So

Reply #12 –
Quote from: audioclaudio on
The differences in sound that are magnified when I listen purely for pleasure are not really audible to me in the sense that I can easily describe them, but they do have a big enough impact on things like fatiguingness and involvingness for me to still notice them in the long run. It can take several weeks, and sometimes it even takes months.

...and they can be shown through ABX testing since ABX testing has no inherent time limits.  Absent of objective data, I call bullshit.

Why should I believe this over placebo effect?  Aside from anecdotes arriving from sighted tests, can you name two independent studies that demonstrate either fatigue or feeling more or less involved is the difference between otherwise identical sounds?  Which do believe has a larger body of objective supporting evidence, placebo effect or fatigue/involvement?  The fatigue/involvement card has been played many times on this forum already and has never made it past "anecdotal."  Here's your opportunity to raise its status to "credible."  Will you rise to the occasion or will you do like everyone before you and simply make excuses for what is continuing to squarely solidify as an excuse?

Let's be clear: if you can qualify perceived differences in sound then you should be able to test for them through ABX.  If removing expectation bias eliminates the ability to perceive these differences then one really must question what is actually influencing the perception, the quality originally theorized or expectation bias.

BTW, fast switching is an tool that may or may not be available to the participant.  When it is available, its use should be optional.
Is 24-bit/192kHz good enough for your lo-fi vinyl, or do you need 32/384?

DBT Is Flawed Because Bob Stuart Says So

Reply #13 –
Quote from: audioclaudio on
What if condition A says I can only hear a difference when listening purely for pleasure, condition B says I can't listen purely for pleasure while I'm being forced to listen for differences instead, and condition C says ABX won't work because ABX inevitably forces me to listen for differences? That would be a deadlock situation. It's only a hypothesis, sure. But then, how exactly were you going to prove that the difference I am hearing is called a hallucination? Long story short, ABX and DBT are very valuable tools. They just ain't always perfect for the job.

You've moved outside science and into "religion", because you have an untestable belief. Untestable, because we know that expectation bias/placebo exists, but you believe there's a real difference that only exists in situations where it's not possible to remove expectation bias / placebo.

Therefore there is no test that can differentiate between an imagined difference, and a "real" difference that only "exists" in situations where an imagined difference cannot be ruled out.

This is heading into the realms of philosophy, or trees falling down in forests that don't make a sound because no one is there to hear them. It's not testable by science. It's not testable at all.

Convenient that, isn't it?


Your most extreme position, that even thinking about whether you hear something (rather than just listening entirely for pleasure) will remove the difference, at first sound a little like the problem of observation changing the particles being observed in physics. Or Schrödinger's cat.. Except, at best, it's about your state-of-mind, and the reality of the audio signal received by your ears exists independently of your state of mind.


I think the convincing thing about blind listening is this: even while listening "blind", many people are still convinced they hear a difference. It's not the act of blinding the sources or being made to compare them that makes the difference go away. It continues to exist in the mind of the observers - they report that they hear it. Their answers show that they do not.

Cheers,
David.

DBT Is Flawed Because Bob Stuart Says So

Reply #15 –
Quote from: dhromed on
Quote from: audioclaudio on
Moreover, Bob Stuart believes rapid switching between sounds can inevitably cause humans to perceive sound objects differently.


I found the source for that. Haven't fully read it yet myself.

Yes, and the page before that. http://www.avguide.com/article/tas-194-mer...t-harley?page=1

DBT Is Flawed Because Bob Stuart Says So

Reply #16 –
Quote from: audioclaudio on
The trouble with ABX testing is always the fact if you want it to be truly objective you have to collect a large enough number of subjective test results to start with, and then apply statistics in order to filter out the unwanted subjectiveness part. This is because every individual test result always has a significant chance of being highly inaccurate, so you need lots of people to participate if you want the final conclusions to be reliable ones.
Often, if not practically always, it is too time consuming and/or too uneconomical to conduct an ABX test properly, i.e. in such way that expectation bias doesn't creep in through the back door etcetera. Experts in auditory neuroscience and psychoacoustics have gathered experimental evidence which appears to indicate the following. Humans who remember different things are perfectly capable of hearing the same sounds differently as a result of remembering different things. For example, Bob Stuart of Meridian, who has a Ph.D in neuroscience, believes that it is perfectly possible for a person to not hear a specific detail in a piece of music when it is played back on one particular system "A", then to discover this specific detail by listening to the same music again on a better, more resolving system "B" next, and then, finally, to turn back to the previous system "A" and always hear this detail on system "A" even though the detail could previously not be heard on system "A". Moreover, Bob Stuart believes rapid switching between sounds can inevitably cause humans to perceive sound objects differently.



So, what do you mean by 'fast switching'?  Do you mean, using short audio samples, or do you mean, making the actual switching moment as instantaneous as possible?

I suspect you mean the former.  If so, that's not 'fast switching'  And btw, how long ago did Bob Stuart get his PhD, and was it specifically in 'neuroscience'?  He's been running Meridian since 1977.  Has he kept up with the neuroscience literature since then?

DBT Is Flawed Because Bob Stuart Says So

Reply #17 –
Quote from: audioclaudio on
Quote from: dhromed on
Quote from: audioclaudio on
Moreover, Bob Stuart believes rapid switching between sounds can inevitably cause humans to perceive sound objects differently.


I found the source for that. Haven't fully read it yet myself.

Yes, and the page before that. http://www.avguide.com/article/tas-194-mer...t-harley?page=1



"What I think is outrageous is to say we understand everything about how the human hearing system works, " 

Well, Bob, who is saying that, really? 

DBT Is Flawed Because Bob Stuart Says So

Reply #18 –
Quote from: krabapple on
Bob Stuart believes rapid switching between sounds can inevitably cause humans to perceive sound objects differently.

Yes, and the page before that. http://www.avguide.com/article/tas-194-mer...t-harley?page=1


"What I think is outrageous is to say we understand everything about how the human hearing system works, " 

Well, Bob, who is saying that, really?


Can we all say "Straw man argument" ;-)

The reference seems very confused or confusing;

Quote
"Robert: That brings to mind a conversation we had at CES about why blind listening tests may not be reliable. You said that when exposed to sound, our brain builds a model over time of what’s creating that sound. The rapid switching in blind testing doesn’t allow that natural process to occur, and we get confused.

Bob: That’s right. Perception happens on lots of different time scales.


Well, yes!

Quote
There’s something called the conscious present, which is a period of time over which some of this integration into an object would happen.


No!  Counscious(ness) is being aware of one's own existence, sensations, thoughts, surroundings, etc

(two big problems: First the correct form of the word was not used, and secondly consciousness acts over time where time is an independent parameter. Time is not consciousness itself.) These guys are obviously way over their head even entering this discussion!

If you were dropped into a concert hall, how long would it take you to really understand what it is you’re hearing? It can take several seconds, or even minutes, before you’re listening fully into the space.

Quote
Sometimes when you’re looking for a difference between A and B, you can hear it quickly. Other times the difference between A and B can come on a time scale of minutes or even longer where you find that you’ve changed something and you don’t notice a change but find that you have a very different connection to the music. But if you are doing quick switching that mechanism gets broken.


Major conflation!  Small details in sound only remain in the brain for a very short period of time - the time scale is seconds at most. You may remember that you heard a small detail that you converted into a memory, but the actual sensation related to the small detail comes and goes almost instantly.


Quote
The problem with A/B switching, or blind listening tests,


Another major error involving conflation - blind tests and A/B switching are completely independent. You can do either independent of the other. Equating them again shows that the people talking are way out of their depth


Quote
is that it doesn’t always eliminate things that we find to be important on a lot of time scales.


Hello! Aren't we talking about hearing small details and not eliminating the perception of them?

Quote
Obviously you can do blind listening on long time scales, and that’s good.


Actually, it has been found that listening on long time scales erases the reliable perception of small details. The most important use of long term listening is to search for short passages of music where the audible difference we are investigating is portrayed most clearly.h


All of the confusion I see above IME is greatly helped by true blind testing. Has either done more than diddle around with blind testing? I don't see it here!

DBT Is Flawed Because Bob Stuart Says So

Reply #19 –
Quote from: audioclaudio on
The trouble with ABX testing is always the fact if you want it to be truly objective you have to collect a large enough number of subjective test results to start with, and then apply statistics in order to filter out the unwanted subjectiveness part.


Wow! I've never seen so many critical mistakes in one sentence!

First off there is no intent that ABX tests be objective. The goals are that they be relevant, reliable and accurate. Tests involving listening by people are inherently subjective and that is a good thing. In fact the goal is to involve subjects (people) and in some sense be as subjective as possible.

The reason for collecting modest numbers of test results is to manage the inherent variations in subjective tests near the threshold of reliable detection. If a test necessarily involves a large number of trials, the effect being studied is probably so small that it is actually not significant to normal listening. When something is clearly audible statistical significance is easily obtained with a fairly small number of trials.

Quote
This is because every individual test result always has a significant chance of being highly inaccurate, so you need lots of people to participate if you want the final conclusions to be reliable ones.


Wrong. The above is only true for very small differences. Trained listeners can be very accurate and reliable if they are listening to a difference that is even just a little bit above the threshold of audibility.

One of the more obvious things about listening tests that one obtains from practical experience is that when they involve things that can be actually heard, they are generally fast and easy.

Most of the anecdotes about zillions of trials and frustrated listeners come from people who in retrospect were trying to hear things that aren't audible.

DBT Is Flawed Because Bob Stuart Says So

Reply #20 –
Quote from: krabapple on
So, what do you mean by 'fast switching'?  Do you mean, using short audio samples, or do you mean, making the actual switching moment as instantaneous as possible?

I suspect you mean the former.  If so, that's not 'fast switching'

From what I can read in the article, it means using short audio samples.
Quote from: krabapple on
And btw, how long ago did Bob Stuart get his PhD, and was it specifically in 'neuroscience'?  He's been running Meridian since 1977.  Has he kept up with the neuroscience literature since then?

He's still a very highly acclaimed expert in psychoacoustics, the "golden ears" type of person, and IIRC he is still one of the most experienced members of the AES (Audio Engineering Society) group. He is also famed for being the inventor of (IMO) breakthrough innovative technologies such as the apodizing filter, which BTW has recently been adopted by Dolby TrueHD, and for the work he has done in the development of the MLP (Meridian Lossless Packing) codec, which is commonly used in DVD-Audio, HD DVD and Blu Ray.
Quote from: krabapple on
"What I think is outrageous is to say we understand everything about how the human hearing system works, " 

Well, Bob, who is saying that, really?

People who believe ABX is ALWAYS the perfect tool under ANY condition, and who demand objective evidence about EVERYTHING, but who, at the same time also, fail to provide objective evidence to backup their belief system?
From the same article:
"The problem with A/B switching, or blind listening tests, is that it doesn’t always eliminate things that we find to be important on a lot of time scales."
A bit further down the paragraph:
"What we’re looking for is not only that we can hear a difference but also that it is more musically satisfying."
You could always try and find objective evidence to show it's bullshit. However... :cough: I wish you the best of luck with that! :cough:

DBT Is Flawed Because Bob Stuart Says So

Reply #21 –
Quote from: audioclaudio on
Quote from: krabapple on
So, what do you mean by 'fast switching'?  Do you mean, using short audio samples, or do you mean, making the actual switching moment as instantaneous as possible?

I suspect you mean the former.  If so, that's not 'fast switching'

From what I can read in the article, it means using short audio samples.


Actually, you need both.


Quote
[quote author= link=msg=804891 date=0]And btw, how long ago did Bob Stuart get his PhD, and was it specifically in 'neuroscience'?  He's been running Meridian since 1977.  Has he kept up with the neuroscience literature since then?

He's still a very highly acclaimed expert in psychoacoustics, the "golden ears" type of person, and IIRC he is still one of the most experienced members of the AES (Audio Engineering Society) group.
[/quote]

In your opinion, which based on your posts seems to be poorly-informed in this area.

Quote
He is also famed for being the inventor of (IMO) breakthrough innovative technologies such as the apodizing filter, which BTW has recently been adopted by Dolby TrueHD, and for the work he has done in the development of the MLP (Meridian Lossless Packing) codec, which is commonly used in DVD-Audio, HD DVD and Blu Ray.


None of which gives someone chops related to doing proper subjective tests.


[quote author=audioclaudio link=msg=0 date=]People who believe ABX is ALWAYS the perfect tool under ANY condition,[/quote]

Straw man argument.

[quote author=audioclaudio link=msg=0 date=]and who demand objective evidence about EVERYTHING[/quote]


Straw man argument compounded by abuse of the word "objective"


[quote author=audioclaudio link=msg=0 date=]but who, at the same time also, fail to provide objective evidence to backup their belief system?[/quote]

I would summarize my belief system in this area as being dominated by the idea that sighted evaluations involving small audible differences are strongly influenced by false positives. What proof of that do you require?

DBT Is Flawed Because Bob Stuart Says So

Reply #22 –
Quote from: Arnold B. Krueger on
Quote from: audioclaudio on
The trouble with ABX testing is always the fact if you want it to be truly objective you have to collect a large enough number of subjective test results to start with, and then apply statistics in order to filter out the unwanted subjectiveness part.


Wow! I've never seen so many critical mistakes in one sentence!

First off there is no intent that ABX tests be objective. The goals are that they be relevant, reliable and accurate. Tests involving listening by people are inherently subjective and that is a good thing. In fact the goal is to involve subjects (people) and in some sense be as subjective as possible.

The reason for collecting modest numbers of test results is to manage the inherent variations in subjective tests near the threshold of reliable detection. If a test necessarily involves a large number of trials, the effect being studied is probably so small that it is actually not significant to normal listening. When something is clearly audible statistical significance is easily obtained with a fairly small number of trials.

You basically just confirmed the whole thing I said. Because the problem with ABX testing is the fact it cannot always be used to produce truly objective results, it cannot always be considered relevant, accurate and reliable.
http://en.wikipedia.org/wiki/Objectivity_(science)
Quote from: Arnold B. Krueger on
Quote
This is because every individual test result always has a significant chance of being highly inaccurate, so you need lots of people to participate if you want the final conclusions to be reliable ones.


Wrong. The above is only true for very small differences. Trained listeners can be very accurate and reliable if they are listening to a difference that is even just a little bit above the threshold of audibility.

That is only your opinion. What you call "small differences" might very well be less small to me.
Quote from: Arnold B. Krueger on
One of the more obvious things about listening tests that one obtains from practical experience is that when they involve things that can be actually heard, they are generally fast and easy.

Most of the anecdotes about zillions of trials and frustrated listeners come from people who in retrospect were trying to hear things that aren't audible.

...Or the bias that ABX fails to eliminate is exactly what's causing the fact things aren't audible.

DBT Is Flawed Because Bob Stuart Says So

Reply #23 –
Quote from: Arnold B. Krueger on
Actually, you need both.

Bob Stuart says it breaks the natural process if short audio samples are used. By breaking the natural process, bias is introduced inevitably. My point is not whether or not Bob Stuart is correct about what he says. It's the simple fact that, although I haven't, ironically you still haven't proven anything at all, either. So please, just stop trying to pretend that you have.
Quote from: Arnold B. Krueger on
Quote
[quote author= link=msg=804891 date=0]And btw, how long ago did Bob Stuart get his PhD, and was it specifically in 'neuroscience'?  He's been running Meridian since 1977.  Has he kept up with the neuroscience literature since then?

He's still a very highly acclaimed expert in psychoacoustics, the "golden ears" type of person, and IIRC he is still one of the most experienced members of the AES (Audio Engineering Society) group.


In your opinion, which based on your posts seems to be poorly-informed in this area.

Quote
He is also famed for being the inventor of (IMO) breakthrough innovative technologies such as the apodizing filter, which BTW has recently been adopted by Dolby TrueHD, and for the work he has done in the development of the MLP (Meridian Lossless Packing) codec, which is commonly used in DVD-Audio, HD DVD and Blu Ray.


None of which gives someone chops related to doing proper subjective tests.[/quote]
As per my above comment, none of your contributions to this thread give you chops about it so far, either...
Quote from: Arnold B. Krueger on
Quote
People who believe ABX is ALWAYS the perfect tool under ANY condition,


Straw man argument.

Quote
and who demand objective evidence about EVERYTHING



Straw man argument compounded by abuse of the word "objective"

You still don't get the sarcasm in any of this part, do you?   
[quote author= link=msg=804914 date=0][quote author=audioclaudio link=msg=0 date=]but who, at the same time also, fail to provide objective evidence to backup their belief system?[/quote]

I would summarize my belief system in this area as being dominated by the idea that sighted evaluations involving small audible differences are strongly influenced by false positives. What proof of that do you require?
[/quote]
I don't require proof of that at all, because as a matter of fact I never disagreed with it. The McGurk effect can never be denied, for example. My stance on sighted evaluations is the same as Bob Stuart's, meaning I am convinced there are at least SOME situations where they CAN be more relevant, accurate and reliable than double blinded evaluations, even if the differences in sound are said to be "small".

DBT Is Flawed Because Bob Stuart Says So

Reply #24 –
audioclaudio and Arnold  : These two last pages of the thread are really difficult to follow, and don't really help at all to make a decision that answers the topic's title.


First of all, I would like to rewind and set a straight and concrete meaning of several words, since it's not really clear what you're against:

- Objective test/Objectivism ( http://en.wikipedia.org/wiki/Objectivism . In this thread, really http://en.wikipedia.org/wiki/Objectivity_%28science%29 ) : The main idea behind this is that the person doing the test does not matter, since the reality exists independently of the subject taking the test (1 meter is 1 meter no matter who checks it) . Objectivity does make possible to repeat tests and *verify* a result.

- Subjective test/Subjectivism ( http://en.wikipedia.org/wiki/Subjectivism ) : The main idea under subjectivism is that we know the world thanks to our sensory system, and as such, only what we can perceive is what exists (ok. This is just one interpretation. Subjectivism has many variants, from phylosophical, to theological).


- ABX Test: An Objective test that evaluates the response (subjective or not) of the participants to the items at test, and obtains a (statistically) representable result that can be considered reliable and, as such, valid for the specific group at which the study was directed.

- ABX Test, application: An ABX test can only say that a difference exists between the two items being evaluated by the determined subgroup at which the study is directed.  (Concretely, this means that it can never be used to say that a difference does not exist. At much, with many failed tests, it can be an indication that such difference is not probable).

In case of Hydrogenaudio, this means that it allows us to *verify* that an *audible difference* between two audio samples (being them two different codecs, two soundcards, two amplifiers....) does exist for the person or people involved in doing the ABX test. If enough tests validate this, then, and only then, can be considered that the difference is real and not due to some error in the test.


Hydrogenaudio is an Objectivism Forum. Let's put that straight and undeniably clear. The high emphasis on ABX tests, the relation with the science world, and the participation of people directly related to the subject of this forum (like the actual coders of audio codecs) is an indication of it.



Reasons why ABX tests are a requirement in order to get objective tests, reliable and representable for the intended audience:


A) Expectation bias ( http://en.wikipedia.org/wiki/Expectation_bias ): A subject, intentionally or not, can be influenced by its subconsciusness: Someone listening to a song in a newly bought equipment does have expectation bias when evaluating its sound quality against an older, less good equipment. (while a difference might exists, the subject could be perceiving his feeling that a difference exists and not the real difference).

B) Representative: To pass an ABX test, it is required to correctly identify a percentage of the trials, and they cannot be cherry picked. ABX pretends to reduce the possibilities of: chance/luck, hidden elements at test (i.e. a difference caused by something that is not the item at test), unconscious influence from other subjects (that's the requirement for the test to be *double* blind).


Setting up an audio ABX test is not necesarily easy, Here are some examples of what can go wrong:

- Playing at different sample rates: soundcards, soundcard drivers, OS audio stack, or even the playback application can cause problems not related to the original audio, but to the different sampling rate being used.
- Volume mismatch: It is important that the samples at test are correctly leveled, so that the subject evaluates the audio, not the difference in audio amplitude. Differences of less than 1dB can be audible.
- Clipping/distortions caused by playback settings/equipment: It is not strange that audio clips digitally today. While the clip in itself could be heard and could determine a limitation or problem depending on the intended use, the problem could be gone just by setting the amplification properly.



When is an ABX necessary:

ABX test, as said above, help to determine that a difference exists. For Hydrogenaudio it is required when an *audible* difference is reported and:

- It is not commonly known that it exists
- It is against the general knowledge that says such difference should not be audible.
- It is not measurable by other means that could clearly determine otherwise (i.e. an audible difference in lossless codecs can be discarded if the codec works and so the problem, if any, exists somewhere else).



To finish this post, I'd like to add that ABX is not intended to avaluate if an item is better than another. In order to determine such thing, it is necessary that they are found to be different. Once that is verified, the difference can be avaluated between both items.

It's been said that ABX does not impose a limitation of how long/short a listening period runs for, neither puts it limits in how many times you switch back an forth to determine a difference/preference.
Also note that the existence of a difference is objective. The avaluation of the difference, while some rules apply, is more subjective.
That's why on the public listening tests, the statistic methods (like MUSHRA) that are applied on the final results of the group try to reduce the influence of subjective decisions.
 
SimplePortal 1.0.0 RC1 © 2008-2020