Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: The Dishonesty of Sighted Listening Tests (Read 28186 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

The Dishonesty of Sighted Listening Tests

http://seanolive.blogspot.com/2009/04/dish...io-product.html

Can't believe I saw this on the Stereophile forums before I saw it here (thanks to Ethan Winer).

Quote
In summary, the sighted and blind loudspeaker listening tests produced significantly different sound quality ratings. The psychological biases in the sighted tests were sufficiently strong that listeners were largely unresponsive to real changes in the sound quality caused by acoustical interactions between the loudspeaker, its position in the room, and the program material. In other words, sighted tests produce dishonest and unreliable measurements of how the product truly sounds. It’s time the audio industry grow up and acknowledge this fact, if they want to gain the trust and respect of consumers.


The Dishonesty of Sighted Listening Tests

Reply #1
Seeing this kind of stuff makes me a happy young lad

Thanks for sharing.

 

The Dishonesty of Sighted Listening Tests

Reply #2
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.

http://forum.stereophile.com/forum/showfla...age=0#Post64883

There's no use for a carebears mode with posts like this:

Quote
So my hearing is better than the advocates of the DBT, who have no idea what they are taking about.


Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.

The Dishonesty of Sighted Listening Tests

Reply #3
It is in fact about 1 or 2 steps removed from RAO.

Pls--
RAO? Not, I think, Royal Army of Oman or renal artery occlusion, though perhaps remedial action opportunity?

Edit: typo

The Dishonesty of Sighted Listening Tests

Reply #4
No, rec.audio.opinion, that legendary cesspool where defenses of double blind tests are sometimes replied to with allegations of child molestation or drug abuse.

The Dishonesty of Sighted Listening Tests

Reply #5
Quote
...just accept the fact that DBTs are hopelessly flawed.

Quote
You have removed the listener from the music by asking only if there is a difference.

Damn it. Now I'm down again. That second quote is just...damn it! Damn it!

Well, in any case, good luck with your endeavor over there at SF. I'll certainly be keeping an eye on the thread.

The Dishonesty of Sighted Listening Tests

Reply #6
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.

http://forum.stereophile.com/forum/showfla...age=0#Post64883

There's no use for a carebears mode with posts like this:

Quote
So my hearing is better than the advocates of the DBT, who have no idea what they are taking about.


Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.




KBK and Fremer are despicable examples of the anti-science audiophile mindset. 


The Dishonesty of Sighted Listening Tests

Reply #7
Nah, Michigan J Frog isn't Mikey - unless he has sockpuppets?

The Dishonesty of Sighted Listening Tests

Reply #8
Hmmm.. it seems that there are two problems to be solved here:

1) Determine the audio system that sounds the best (for which, given a suitable definition of `best', DBT may be a good solution).

2) Determine the audio system that sounds the best, looks the best, and best serves as a status symbol (for which, DBT by itself would not be a good solution).

Whether I've captured the problems perfectly is not the point.  The point is that solutions shouldn't need to be argued over -- either they solve the problem or they don't.  If it's not clear whether a solution solves or best solves a problem, then go back and debate and agree the requirements of the problem to be solved.

  - bandpass  (Prof.)

The Dishonesty of Sighted Listening Tests

Reply #9
Nah, Michigan J Frog isn't Mikey - unless he has sockpuppets?


I wasn't confusing the two -- or referring directly to your two posted quotes -- but I'm happy to add MJF, geoffkait, jvigne, and atkinson to that shortlist.  And now the ones accusing Olive of having a commercial agenda.


(On a more optimistic note, I see there's a 'davidrich' posting there -- I wonder if that's *the* David Rich who does excellent in-depth technical reviews for Sensible Sound and others?  )

The Dishonesty of Sighted Listening Tests

Reply #10
Quote
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.


Why even bother with them? Why not just boycott the website? You know for a fact they are never going to except DBT due to their ignorance and the fact that they are trying to sell products. Why would they listen to the scientific community? It's like getting into a political discussion with you best friend who vehemently disagree with you. While in one senese it's fun to yell at them for being so naive in close minded in another sense you aren't going to change their world view. It's the same thing with DBT's in my opinion. Either they don't know how to conduct them and arn't willing to or are never going to accept them. 

Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.


That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.
budding I.T professional

The Dishonesty of Sighted Listening Tests

Reply #11
Quote
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.
Why even bother with them? Why not just boycott the website? You know for a fact they are never going to except DBT due to their ignorance and the fact that they are trying to sell products. Why would they listen to the scientific community? It's like getting into a political discussion with you best friend who vehemently disagree with you. While in one senese it's fun to yell at them for being so naive in close minded in another sense you aren't going to change their world view. It's the same thing with DBT's in my opinion. Either they don't know how to conduct them and arn't willing to or are never going to accept them. 

I admit, part of it is exasperation on my part at Sean even bothering to post there. Had he not been there and it just was Ethan vs the turks, well, it wouldn't be the first time I avoided a DBT flamewar on Stereophile. But I couldn't just stand there and watch him get flamed to pieces out there by himself. He might get old and cranky and turn into jj or Arny before our very eyes.



(kidding)

Quote
Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.
Right. That said, I'm pretty proud of my reply...

The Dishonesty of Sighted Listening Tests

Reply #12
Quote
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.


Why even bother with them? Why not just boycott the website? You know for a fact they are never going to except DBT due to their ignorance and the fact that they are trying to sell products. Why would they listen to the scientific community? It's like getting into a political discussion with you best friend who vehemently disagree with you. While in one senese it's fun to yell at them for being so naive in close minded in another sense you aren't going to change their world view. It's the same thing with DBT's in my opinion. Either they don't know how to conduct them and arn't willing to or are never going to accept them. 

Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.


That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


Hotshot: That is also my conclusion. It is largely a waste of time to try to convert faith-based audiophiles to accept and apply science towards audio research and evaluation.  If they are manufacturers, consumers, or reviewers, it doesn't serve their own self interests.

Most high-end audio companies and audio magazines simply cannot afford the facilities, staff, and time to conduct proper double-blind listening tests.  In the absence of  scientific-based listening tests or comprehensive perceptually-meaningful objective measurements, there is little choice but to eschew a scientific approach, and rely on unsubstantiated claims about the product's sound quality.

In the end, this approach does little to advance our scientific understanding about sound reproduction, nor does it advance the  state of technology required to improve our listening experience. You cannot sustain a long-term business model based on this approach. I think Gordon Holt (read his interview linked in my blog) knew this back in the 1980's, and the consumer has confirmed he was correct, voting with their pocket book.

My blog posting on Blind versus Sighted Listening tests was not aimed at or intended to change anti-science audiophiles who are mostly a lost cause, and probably not a significant demographic to even worry about. My article was aimed at people genuinely interested in the psychological biases associated with evaluating audio equipment under sighted conditions.  The bias effects are real as shown in the study. Therefore, a blind test is required in order to measure the true sound quality of the loudspeaker.

I don't expect double-blind tests to flourish within audio companies or magazines any time soon. Most high-end audio stores have gone bankrupt and closed their doors, In-store audio demos are mostly a thing of the past.  I would like to see a future where consumers  are provided standardized perceptually meaningful audio specifications that allow them to quickly determine whether product A sounds better than product B based on a set of sound quality attributes.  This would allow the consumer to free themselves from reliance on unsubstantiated manufacturers'/audio reviewers' claims based on uncontrolled, biased sighted listening. It's probably a pipe dream - but it's something worth aiming towards.

The Dishonesty of Sighted Listening Tests

Reply #13
> It is largely a waste of time to try to convert faith-based audiophiles to accept and apply
> science towards audio research and evaluation. If they are manufacturers, consumers, or
> reviewers, it doesn't serve their own self interests.

In the 1970s before the mainstream audiophile marketing sector, home audio consumers were happily purchasing amplifiers with 0.001% THD at 1kHz into an 8 Ohm load rather than ones with 0.002% distortion. Technical perfomance in a "real" sense is rarely going to have much to do with the success of luxury products. What matters is perceived performance.

> Most high-end audio companies and audio magazines simply cannot afford the facilities, staff,
> and time to conduct proper double-blind listening tests.

Utter rubbish. If there was a business case for assessing performance in a scientifically valid manner then it would be done. Gearing up to routinely peform scientifically valid experiments is not a significant issue. The problem is that it would be suicide - audiophile products have to be distinguishable.

> In the absence of scientific-based listening tests or comprehensive perceptually-meaningful
> objective measurements, there is little choice but to eschew a scientific approach, and rely on
> unsubstantiated claims about the product's sound quality.

Rubbish. If a consumer wants to access meaningfull information it exists in the technical journals and by talking to professionals. A passive consumer is going to get fed nonsense but there is no reason an active one cannot access information in order to put together a good hi-fi for a reasonable price.

> In the end, this approach does little to advance our scientific understanding about sound
> reproduction, nor does it advance the state of technology required to improve our listening
> experience.

State funding for scientific research in sound reproductino disappeared about 50 years ago. Since then it has been a development task for a low-tech luxury goods sector.

> You cannot sustain a long-term business model based on this approach.

Business is about persuading people to purchase your products. Technical performance is not wholly irrelevant for luxury goods but it is seriously trumped by perceived performance.

> I think Gordon Holt (read his interview linked in my blog) knew this back in the 1980's, and
> the consumer has confirmed he was correct, voting with their pocket book.

My disciples? What is the basis for giving Gordon Holt's views any weight?

> My blog posting on Blind versus Sighted Listening tests was not aimed at or intended to
> change anti-science audiophiles who are mostly a lost cause,

35 years ago these same lost causes held very different views on the performance of home audio equipment. I would suggest that if it made business sense for the industry to change their views it would be easy enough to achieve.

> My article was aimed at people genuinely interested in the psychological biases associated
> with evaluating audio equipment under sighted conditions. The bias effects are real as shown
> in the study.

Indeed.

> Therefore, a blind test is required in order to measure the true sound quality of the
> loudspeaker.

What does true mean? An audiophile believer sitting in front of the shiny hi-fi with all cues turned on?

> I don't expect double-blind tests to flourish within audio companies or magazines any time
> soon.

I would agree with the magazines (must admit to a degree of admiration for how things are currently organised) but not sure I would agree when it comes to the larger audio companies. Blind tests allow engineers to sort out what is due to what and this is useful when juggling the compromises.

> I would like to see a future where consumers are provided standardized perceptually
> meaningful audio specifications that allow them to quickly determine whether product A sounds
> better than product B based on a set of sound quality attributes.

Why?

Is this in the interests of hi-fi companies in the developed world? If it is not in the interests of hi-fi companies in the developed world, is it still in the interests of the consumers in the developed world?

Globalisation may have changed the answer to this since 1970s when the switch from hi-fi to audiophile occurred.

> This would allow the consumer to free themselves from reliance on unsubstantiated
> manufacturers'/audio reviewers' claims based on uncontrolled, biased sighted listening.

Ummm... You have read a few of the posts in the Stereophile thread? You are aware of the proportion of Sterophile-like to HydrogenAudio-like forums?

> It's probably a pipe dream - but it's something worth aiming towards.

Why?

I would sugget the rational thing to do is to accept the reason why people become audiophiles, to ignore them and perhaps work towards creating a small community for people with a hobby interest in the sound quality of home audio.

Please feel free to respond robustly.

The Dishonesty of Sighted Listening Tests

Reply #14

<<Utter rubbish. If there was a business case for assessing performance in a scientifically valid manner then it would be done. Gearing up to routinely peform scientifically valid experiments is not a significant issue>>

Are you sure? Contrary to what you claim, the investment in anechoic chambers, listening labs, equipment, speaker movers, training listeners, engineering/scientific staff IS a significant issue and cost. And if it were not for the scientific assessment of loudspeakers done by Floyd Toole at the NRC -- being continued at Harman --  we wouldn't have learned how to do proper measurements of loudspeakers and correlate those measurements to listeners' preference ratings. Companies like Revel, PSB, Paradigm, etc. indeed much loudspeaker industry have all benefited from that scientific research, as well as audio magazines who began to do loudspeakers measurements off-axis after that research was published. The success of the Canadian loudspeaker industry is in large part due to scientific assessment of performance, and they readily acknowledge it in their marketing literature. I would argue that is an example of a business case where scientific assessment of performance has paid off.

<<Rubbish. If a consumer wants to access meaningfull information it exists in the technical journals and by talking to professionals. A passive consumer is going to get fed nonsense but there is no reason an active one cannot access information in order to put together a good hi-fi for a reasonable price.>>

Again, are you sure about that?  Show me the technical journals (J AES ?) I can go to find out which loudspeaker is the best  one to purchase? Technical journals like AES or JASA don't publish the names of the products. Also, please give some names of professionals where a consumer can  get unbiased, comprehensive scientific subjective and objective performance data that tells them which is the best loudspeaker to purchase for say $2000?

<<< What is the basis for giving Gordon Holt's views any weight?>>>

His  credentials and experience in the high-end industry surely make his opinion as worthy as yours, mine or anyone elses'.

<<What does true mean? An audiophile believer sitting in front of the shiny hi-fi with all cues turned on?>>

In the context of a listening test, "true" means the listener is only responding to acoustical changes caused by the variable being manipulated. The marketing department deals with how the customer responds to the shininess of the hi-fi -- not me. I  only care about the sound quality.


< . Blind tests allow engineers to sort out what is due to what and this is useful when juggling the compromises>>

And why wouldn't it also be a useful tool for reviewers or consumers for sorting out which products make the best compromises?

> I would like to see a future where consumers are provided standardized perceptually
> meaningful audio specifications that allow them to quickly determine whether product A sounds
> better than product B based on a set of sound quality attributes.

>>Why?>>

Because it would provide accurate and reliable information that would make it easier for consumers to choose the loudspeaker that best matches their taste and application. The current loudspeaker specifications are meaningless.  As Toole likes to say in his book, "there is more useful information on the side of a tire." The industry bemoans that consumers don't care about high quality audio anymore. Part of the problem falls upon us because we have failed to educate them, and give them compelling,  trustworthy, accessible data that clearly shows them there is a significant difference in sound quality between a $200. HTIB and a $3000 home theatre setup.

The Dishonesty of Sighted Listening Tests

Reply #15
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


The last thing we need to be are the Richard Dawkinses of DBT.

That said, anyone who jumps into that discussion deserves a medal.

The Dishonesty of Sighted Listening Tests

Reply #16
[quote name='Axon' date='Apr 11 2009, 14:09' post='627018']

<<I admit, part of it is exasperation on my part at Sean even bothering to post there. Had he not been there and it just was Ethan vs the turks, well, it wouldn't be the first time I avoided a DBT flamewar on Stereophile. But I couldn't just stand there and watch him get flamed to pieces out there by himself. He might get old and cranky and turn into jj or Arny before our very eyes.>>

Axon,
Just for the record: I'm already old and cranky, at least according to my wife and kids.  When the anti-blind- test flamers come up with some compelling arguments and scientific data to back them up,  then I will start to take them seriously, and respond.

Cheers
Sean
Audio Musings

The Dishonesty of Sighted Listening Tests

Reply #17
Quote
The last thing we need to be are the Richard Dawkinses of DBT.


 


Quote
When the anti-blind- test flamers come up with some compelling arguments and scientific data to back them up, then I will start to take them seriously, and respond.


I still believe that are both ignorant and naive. They simply won't produce that kind of evidence, because they don't know how to do the appropriate DBTs and they are simply lazy so they put up this shall we say this "wall" between you and everyone else with these "sighted listening tests" that are either non-scientific or pseudo-scientific. Anyway I just wanted to chime in and say I enjoy reading your informative blog posts on DBT with speakers. It's very informative! I am learning all of this cool new stuff. 
budding I.T professional

The Dishonesty of Sighted Listening Tests

Reply #18
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


The last thing we need to be are the Richard Dawkinses of DBT.


I disagree.  We can only *wish* we had a bulldog of his professional calibre and name recognition, defending 'our' branch of science as well as he does evolutionary biology.

If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman fallacy.

Quote
That said, anyone who jumps into that discussion deserves a medal.


OK, is mine in the mail? 

The Dishonesty of Sighted Listening Tests

Reply #19
If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman fallacy.

This could get very OT, but I think there is a relevance. Dawkins' failure to engage with sophisticated theisms is a real weakness in his intellectual position, and the No True Scotsman fallacy is irrelevant, unless we accept Dawkins' claim to define real theism--which is itself a No True Scotsmanism. I don't want to pursue that any further, partly because it's OT and partly because I have no clue about whether or not the universe is genuinely intelligible (to believe which roughly = Deism), or whether or not any human values correspond to values in some way inherent in the universe (to believe which roughly = Theism).

What is relevant is that Dawkins' style is deeply unpersuasive to the undecided; in the same way, to slag off audiophools, and still worse, to assume that the audience of audiophool groups are themselves all audiophools, is unpersuasive to those who could be brought to rationality. Look at the reception AV-OCD got here when he asked a simple question, the answer to which is not, in fact self-evident. How do the people who don't know much get to realise that it's HA that's on the side of Truth and Enlightenment, if we come on like a bunch of yahoos?

Which is not to say that you, krabapple, have been being a yahoo, at all, just a general remark that seems to have become a bit relevant at the moment.

The Dishonesty of Sighted Listening Tests

Reply #20
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


The last thing we need to be are the Richard Dawkinses of DBT.


I disagree.  We can only *wish* we had a bulldog of his professional calibre and name recognition, defending 'our' branch of science as well as he does evolutionary biology.

If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman fallacy


I'm referring to the fact that he's a smug ass. And I say that as an evolutionary theory loving athiest. He gives athiests a bad name just like people who talk about audio listening issues while dismissing ABX. Just my opinion.

The Dishonesty of Sighted Listening Tests

Reply #21
No, rec.audio.opinion, that legendary cesspool where defenses of double blind tests are sometimes replied to with allegations of child molestation or drug abuse.


Not to mention coprophilia, coprophagia, deviant sexual practices, terrorists, ...

The best cesspool company in the world could not pump out rec.audio.opinion.
-----
J. D. (jj) Johnston

The Dishonesty of Sighted Listening Tests

Reply #22
He might get old and cranky and turn into jj or Arny before our very eyes.

Eh?

Trust me, he's already there.
-----
J. D. (jj) Johnston

The Dishonesty of Sighted Listening Tests

Reply #23
Hotshot: That is also my conclusion. It is largely a waste of time to try to convert faith-based audiophiles to accept and apply science towards audio research and evaluation.


Well, I wandered over there and gave you all the support I could muster without suggesting that they were first cousins to creationists.

I was as polite as I could manage given the heights of arrogance and depths of hatred of science that simply floated away from the thread.
-----
J. D. (jj) Johnston

The Dishonesty of Sighted Listening Tests

Reply #24
If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman fallacy.

This could get very OT, but I think there is a relevance. Dawkins' failure to engage with sophisticated theisms is a real weakness in his intellectual position, and the No True Scotsman fallacy is irrelevant, unless we accept Dawkins' claim to define real theism--which is itself a No True Scotsmanism. I don't want to pursue that any further, partly because it's OT and partly because I have no clue about whether or not the universe is genuinely intelligible (to believe which roughly = Deism), or whether or not any human values correspond to values in some way inherent in the universe (to believe which roughly = Theism).



Dawkins is addressing the dominant cultural paradigms of God-belief -- not the scholastic or Deistic or Einsteinian arguments for a highly abstract God (which he actually *specifically notes* he is not addressing, in his book)  .  The theologically minded claim that, well, what the common folk believe in isn't the REAL God-- hence the No True Scotsman.

Quote
What is relevant is that Dawkins' style is deeply unpersuasive to the undecided;



And you know this how?  There is testimony from readers to the contrary.