HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Axon on 2009-04-11 03:51:26

Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-11 03:51:26
http://seanolive.blogspot.com/2009/04/dish...io-product.html (http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html)

Can't believe I saw this on the Stereophile forums before I saw it here (thanks to Ethan Winer).

Quote
In summary, the sighted and blind loudspeaker listening tests produced significantly different sound quality ratings. The psychological biases in the sighted tests were sufficiently strong that listeners were largely unresponsive to real changes in the sound quality caused by acoustical interactions between the loudspeaker, its position in the room, and the program material. In other words, sighted tests produce dishonest and unreliable measurements of how the product truly sounds. It’s time the audio industry grow up and acknowledge this fact, if they want to gain the trust and respect of consumers.

Title: The Dishonesty of Sighted Listening Tests
Post by: Ron Jones on 2009-04-11 04:17:36
Seeing this kind of stuff makes me a happy young lad

Thanks for sharing.
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-11 05:19:34
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.

http://forum.stereophile.com/forum/showfla...age=0#Post64883 (http://forum.stereophile.com/forum/showflat.php?Cat=0&Number=64883&an=0&page=0#Post64883)

There's no use for a carebears mode with posts like this:

Quote
So my hearing is better than the advocates of the DBT, who have no idea what they are taking about.


Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.
Title: The Dishonesty of Sighted Listening Tests
Post by: MichaelW on 2009-04-11 05:33:35
It is in fact about 1 or 2 steps removed from RAO.

Pls--
RAO? Not, I think, Royal Army of Oman or renal artery occlusion, though perhaps remedial action opportunity?

Edit: typo
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-11 05:36:44
No, rec.audio.opinion, that legendary cesspool where defenses of double blind tests are sometimes replied to with allegations of child molestation or drug abuse.
Title: The Dishonesty of Sighted Listening Tests
Post by: Ron Jones on 2009-04-11 05:43:11
Quote
...just accept the fact that DBTs are hopelessly flawed.

Quote
You have removed the listener from the music by asking only if there is a difference.

Damn it. Now I'm down again. That second quote is just...damn it! Damn it!

Well, in any case, good luck with your endeavor over there at SF. I'll certainly be keeping an eye on the thread.
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-11 05:52:16
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.

http://forum.stereophile.com/forum/showfla...age=0#Post64883 (http://forum.stereophile.com/forum/showflat.php?Cat=0&Number=64883&an=0&page=0#Post64883)

There's no use for a carebears mode with posts like this:

Quote
So my hearing is better than the advocates of the DBT, who have no idea what they are taking about.


Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.




KBK and Fremer are despicable examples of the anti-science audiophile mindset. 

Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-11 07:42:30
Nah, Michigan J Frog isn't Mikey - unless he has sockpuppets?
Title: The Dishonesty of Sighted Listening Tests
Post by: bandpass on 2009-04-11 09:42:30
Hmmm.. it seems that there are two problems to be solved here:

1) Determine the audio system that sounds the best (for which, given a suitable definition of `best', DBT may be a good solution).

2) Determine the audio system that sounds the best, looks the best, and best serves as a status symbol (for which, DBT by itself would not be a good solution).

Whether I've captured the problems perfectly is not the point.  The point is that solutions shouldn't need to be argued over -- either they solve the problem or they don't.  If it's not clear whether a solution solves or best solves a problem, then go back and debate and agree the requirements of the problem to be solved.

  - bandpass  (Prof.)
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-11 16:52:34
Nah, Michigan J Frog isn't Mikey - unless he has sockpuppets?


I wasn't confusing the two -- or referring directly to your two posted quotes -- but I'm happy to add MJF, geoffkait, jvigne, and atkinson to that shortlist.  And now the ones accusing Olive of having a commercial agenda.


(On a more optimistic note, I see there's a 'davidrich' posting there -- I wonder if that's *the* David Rich who does excellent in-depth technical reviews for Sensible Sound and others?  )
Title: The Dishonesty of Sighted Listening Tests
Post by: HotshotGG on 2009-04-11 19:52:41
Quote
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.


Why even bother with them? Why not just boycott the website? You know for a fact they are never going to except DBT due to their ignorance and the fact that they are trying to sell products. Why would they listen to the scientific community? It's like getting into a political discussion with you best friend who vehemently disagree with you. While in one senese it's fun to yell at them for being so naive in close minded in another sense you aren't going to change their world view. It's the same thing with DBT's in my opinion. Either they don't know how to conduct them and arn't willing to or are never going to accept them. 

Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.


That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-11 22:09:01
Quote
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.
Why even bother with them? Why not just boycott the website? You know for a fact they are never going to except DBT due to their ignorance and the fact that they are trying to sell products. Why would they listen to the scientific community? It's like getting into a political discussion with you best friend who vehemently disagree with you. While in one senese it's fun to yell at them for being so naive in close minded in another sense you aren't going to change their world view. It's the same thing with DBT's in my opinion. Either they don't know how to conduct them and arn't willing to or are never going to accept them. 

I admit, part of it is exasperation on my part at Sean even bothering to post there. Had he not been there and it just was Ethan vs the turks, well, it wouldn't be the first time I avoided a DBT flamewar on Stereophile. But I couldn't just stand there and watch him get flamed to pieces out there by himself. He might get old and cranky and turn into jj or Arny before our very eyes.



(kidding)

Quote
Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.
Right. That said, I'm pretty proud of my reply (http://forum.stereophile.com/forum/showflat.php?Cat=0&Number=64991&Main=64883#Post64932)...
Title: The Dishonesty of Sighted Listening Tests
Post by: solive on 2009-04-11 23:08:01
Quote
FWIW, there's a truly incredible thread playing out on Stereophile over this; I feel the need to provide some muscle for Sean (who is rather crazily entering that hornet's nest). Note, this is not the place for calm, evenhanded, or respectful discussion. It is in fact about 1 or 2 steps removed from RAO.


Why even bother with them? Why not just boycott the website? You know for a fact they are never going to except DBT due to their ignorance and the fact that they are trying to sell products. Why would they listen to the scientific community? It's like getting into a political discussion with you best friend who vehemently disagree with you. While in one senese it's fun to yell at them for being so naive in close minded in another sense you aren't going to change their world view. It's the same thing with DBT's in my opinion. Either they don't know how to conduct them and arn't willing to or are never going to accept them. 

Quote
Olive's useless time-wasting, money-gobbing tests only prove that double blind tests disorient the listener, and are an unreliable means of testing audio gear.


That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


Hotshot: That is also my conclusion. It is largely a waste of time to try to convert faith-based audiophiles to accept and apply science towards audio research and evaluation.  If they are manufacturers, consumers, or reviewers, it doesn't serve their own self interests.

Most high-end audio companies and audio magazines simply cannot afford the facilities, staff, and time to conduct proper double-blind listening tests.  In the absence of  scientific-based listening tests or comprehensive perceptually-meaningful objective measurements, there is little choice but to eschew a scientific approach, and rely on unsubstantiated claims about the product's sound quality.

In the end, this approach does little to advance our scientific understanding about sound reproduction, nor does it advance the  state of technology required to improve our listening experience. You cannot sustain a long-term business model based on this approach. I think Gordon Holt (read his interview (http://www.stereophile.com/asweseeit/1107awsi/) linked in my blog) knew this back in the 1980's, and the consumer has confirmed he was correct, voting with their pocket book.

My blog posting on Blind versus Sighted Listening tests was not aimed at or intended to change anti-science audiophiles who are mostly a lost cause, and probably not a significant demographic to even worry about. My article was aimed at people genuinely interested in the psychological biases associated with evaluating audio equipment under sighted conditions.  The bias effects are real as shown in the study. Therefore, a blind test is required in order to measure the true sound quality of the loudspeaker.

I don't expect double-blind tests to flourish within audio companies or magazines any time soon. Most high-end audio stores have gone bankrupt and closed their doors, In-store audio demos are mostly a thing of the past.  I would like to see a future where consumers  are provided standardized perceptually meaningful audio specifications (http://seanolive.blogspot.com/2009/01/what-loudspeaker-specifications-are.html) that allow them to quickly determine whether product A sounds better than product B based on a set of sound quality attributes.  This would allow the consumer to free themselves from reliance on unsubstantiated manufacturers'/audio reviewers' claims based on uncontrolled, biased sighted listening. It's probably a pipe dream - but it's something worth aiming towards.
Title: The Dishonesty of Sighted Listening Tests
Post by: honestguv on 2009-04-12 00:58:31
> It is largely a waste of time to try to convert faith-based audiophiles to accept and apply
> science towards audio research and evaluation. If they are manufacturers, consumers, or
> reviewers, it doesn't serve their own self interests.

In the 1970s before the mainstream audiophile marketing sector, home audio consumers were happily purchasing amplifiers with 0.001% THD at 1kHz into an 8 Ohm load rather than ones with 0.002% distortion. Technical perfomance in a "real" sense is rarely going to have much to do with the success of luxury products. What matters is perceived performance.

> Most high-end audio companies and audio magazines simply cannot afford the facilities, staff,
> and time to conduct proper double-blind listening tests.

Utter rubbish. If there was a business case for assessing performance in a scientifically valid manner then it would be done. Gearing up to routinely peform scientifically valid experiments is not a significant issue. The problem is that it would be suicide - audiophile products have to be distinguishable.

> In the absence of scientific-based listening tests or comprehensive perceptually-meaningful
> objective measurements, there is little choice but to eschew a scientific approach, and rely on
> unsubstantiated claims about the product's sound quality.

Rubbish. If a consumer wants to access meaningfull information it exists in the technical journals and by talking to professionals. A passive consumer is going to get fed nonsense but there is no reason an active one cannot access information in order to put together a good hi-fi for a reasonable price.

> In the end, this approach does little to advance our scientific understanding about sound
> reproduction, nor does it advance the state of technology required to improve our listening
> experience.

State funding for scientific research in sound reproductino disappeared about 50 years ago. Since then it has been a development task for a low-tech luxury goods sector.

> You cannot sustain a long-term business model based on this approach.

Business is about persuading people to purchase your products. Technical performance is not wholly irrelevant for luxury goods but it is seriously trumped by perceived performance.

> I think Gordon Holt (read his interview linked in my blog) knew this back in the 1980's, and
> the consumer has confirmed he was correct, voting with their pocket book.

My disciples? What is the basis for giving Gordon Holt's views any weight?

> My blog posting on Blind versus Sighted Listening tests was not aimed at or intended to
> change anti-science audiophiles who are mostly a lost cause,

35 years ago these same lost causes held very different views on the performance of home audio equipment. I would suggest that if it made business sense for the industry to change their views it would be easy enough to achieve.

> My article was aimed at people genuinely interested in the psychological biases associated
> with evaluating audio equipment under sighted conditions. The bias effects are real as shown
> in the study.

Indeed.

> Therefore, a blind test is required in order to measure the true sound quality of the
> loudspeaker.

What does true mean? An audiophile believer sitting in front of the shiny hi-fi with all cues turned on?

> I don't expect double-blind tests to flourish within audio companies or magazines any time
> soon.

I would agree with the magazines (must admit to a degree of admiration for how things are currently organised) but not sure I would agree when it comes to the larger audio companies. Blind tests allow engineers to sort out what is due to what and this is useful when juggling the compromises.

> I would like to see a future where consumers are provided standardized perceptually
> meaningful audio specifications that allow them to quickly determine whether product A sounds
> better than product B based on a set of sound quality attributes.

Why?

Is this in the interests of hi-fi companies in the developed world? If it is not in the interests of hi-fi companies in the developed world, is it still in the interests of the consumers in the developed world?

Globalisation may have changed the answer to this since 1970s when the switch from hi-fi to audiophile occurred.

> This would allow the consumer to free themselves from reliance on unsubstantiated
> manufacturers'/audio reviewers' claims based on uncontrolled, biased sighted listening.

Ummm... You have read a few of the posts in the Stereophile thread? You are aware of the proportion of Sterophile-like to HydrogenAudio-like forums?

> It's probably a pipe dream - but it's something worth aiming towards.

Why?

I would sugget the rational thing to do is to accept the reason why people become audiophiles, to ignore them and perhaps work towards creating a small community for people with a hobby interest in the sound quality of home audio.

Please feel free to respond robustly.
Title: The Dishonesty of Sighted Listening Tests
Post by: solive on 2009-04-12 02:47:03

<<Utter rubbish. If there was a business case for assessing performance in a scientifically valid manner then it would be done. Gearing up to routinely peform scientifically valid experiments is not a significant issue>>

Are you sure? Contrary to what you claim, the investment in anechoic chambers, listening labs, equipment, speaker movers, training listeners, engineering/scientific staff IS a significant issue and cost. And if it were not for the scientific assessment of loudspeakers done by Floyd Toole at the NRC -- being continued at Harman --  we wouldn't have learned how to do proper measurements of loudspeakers and correlate those measurements to listeners' preference ratings. Companies like Revel, PSB, Paradigm, etc. indeed much loudspeaker industry have all benefited from that scientific research, as well as audio magazines who began to do loudspeakers measurements off-axis after that research was published. The success of the Canadian loudspeaker industry is in large part due to scientific assessment of performance, and they readily acknowledge it in their marketing literature. I would argue that is an example of a business case where scientific assessment of performance has paid off.

<<Rubbish. If a consumer wants to access meaningfull information it exists in the technical journals and by talking to professionals. A passive consumer is going to get fed nonsense but there is no reason an active one cannot access information in order to put together a good hi-fi for a reasonable price.>>

Again, are you sure about that?  Show me the technical journals (J AES ?) I can go to find out which loudspeaker is the best  one to purchase? Technical journals like AES or JASA don't publish the names of the products. Also, please give some names of professionals where a consumer can  get unbiased, comprehensive scientific subjective and objective performance data that tells them which is the best loudspeaker to purchase for say $2000?

<<< What is the basis for giving Gordon Holt's views any weight?>>>

His  credentials and experience in the high-end industry surely make his opinion as worthy as yours, mine or anyone elses'.

<<What does true mean? An audiophile believer sitting in front of the shiny hi-fi with all cues turned on?>>

In the context of a listening test, "true" means the listener is only responding to acoustical changes caused by the variable being manipulated. The marketing department deals with how the customer responds to the shininess of the hi-fi -- not me. I  only care about the sound quality.


< . Blind tests allow engineers to sort out what is due to what and this is useful when juggling the compromises>>

And why wouldn't it also be a useful tool for reviewers or consumers for sorting out which products make the best compromises?

> I would like to see a future where consumers are provided standardized perceptually
> meaningful audio specifications that allow them to quickly determine whether product A sounds
> better than product B based on a set of sound quality attributes.

>>Why?>>

Because it would provide accurate and reliable information that would make it easier for consumers to choose the loudspeaker that best matches their taste and application. The current loudspeaker specifications are meaningless.  As Toole likes to say in his book, "there is more useful information on the side of a tire." The industry bemoans that consumers don't care about high quality audio anymore. Part of the problem falls upon us because we have failed to educate them, and give them compelling,  trustworthy, accessible data that clearly shows them there is a significant difference in sound quality between a $200. HTIB and a $3000 home theatre setup.
Title: The Dishonesty of Sighted Listening Tests
Post by: gorgekko on 2009-04-12 19:50:07
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


The last thing we need to be are the Richard Dawkinses of DBT.

That said, anyone who jumps into that discussion deserves a medal.
Title: The Dishonesty of Sighted Listening Tests
Post by: solive on 2009-04-12 22:44:29
[quote name='Axon' date='Apr 11 2009, 14:09' post='627018']

<<I admit, part of it is exasperation on my part at Sean even bothering to post there. Had he not been there and it just was Ethan vs the turks, well, it wouldn't be the first time I avoided a DBT flamewar on Stereophile. But I couldn't just stand there and watch him get flamed to pieces out there by himself. He might get old and cranky and turn into jj or Arny before our very eyes.>>

Axon,
Just for the record: I'm already old and cranky, at least according to my wife and kids.  When the anti-blind- test flamers come up with some compelling arguments and scientific data to back them up,  then I will start to take them seriously, and respond.

Cheers
Sean
Audio Musings (http://seanolive.blogspot.com)
Title: The Dishonesty of Sighted Listening Tests
Post by: HotshotGG on 2009-04-12 23:56:35
Quote
The last thing we need to be are the Richard Dawkinses of DBT.


 


Quote
When the anti-blind- test flamers come up with some compelling arguments and scientific data to back them up, then I will start to take them seriously, and respond.


I still believe that are both ignorant and naive. They simply won't produce that kind of evidence, because they don't know how to do the appropriate DBTs and they are simply lazy so they put up this shall we say this "wall" between you and everyone else with these "sighted listening tests" that are either non-scientific or pseudo-scientific. Anyway I just wanted to chime in and say I enjoy reading your informative blog posts on DBT with speakers. It's very informative! I am learning all of this cool new stuff. 
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-13 00:04:59
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


The last thing we need to be are the Richard Dawkinses of DBT.


I disagree.  We can only *wish* we had a bulldog of his professional calibre and name recognition, defending 'our' branch of science as well as he does evolutionary biology.

If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman (http://en.wikipedia.org/wiki/No_true_Scotsman) fallacy.

Quote
That said, anyone who jumps into that discussion deserves a medal.


OK, is mine in the mail? 
Title: The Dishonesty of Sighted Listening Tests
Post by: MichaelW on 2009-04-13 05:34:17
If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman (http://en.wikipedia.org/wiki/No_true_Scotsman) fallacy.

This could get very OT, but I think there is a relevance. Dawkins' failure to engage with sophisticated theisms is a real weakness in his intellectual position, and the No True Scotsman fallacy is irrelevant, unless we accept Dawkins' claim to define real theism--which is itself a No True Scotsmanism. I don't want to pursue that any further, partly because it's OT and partly because I have no clue about whether or not the universe is genuinely intelligible (to believe which roughly = Deism), or whether or not any human values correspond to values in some way inherent in the universe (to believe which roughly = Theism).

What is relevant is that Dawkins' style is deeply unpersuasive to the undecided; in the same way, to slag off audiophools, and still worse, to assume that the audience of audiophool groups are themselves all audiophools, is unpersuasive to those who could be brought to rationality. Look at the reception AV-OCD got here when he asked a simple question, the answer to which is not, in fact self-evident. How do the people who don't know much get to realise that it's HA that's on the side of Truth and Enlightenment, if we come on like a bunch of yahoos?

Which is not to say that you, krabapple, have been being a yahoo, at all, just a general remark that seems to have become a bit relevant at the moment.
Title: The Dishonesty of Sighted Listening Tests
Post by: gorgekko on 2009-04-13 06:02:08
That's sacrilegious. I can't think of a good analogy, but that's like saying Richard Dawkins doesn't understand anything about Evolutionary Biology.


The last thing we need to be are the Richard Dawkinses of DBT.


I disagree.  We can only *wish* we had a bulldog of his professional calibre and name recognition, defending 'our' branch of science as well as he does evolutionary biology.

If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman (http://en.wikipedia.org/wiki/No_true_Scotsman) fallacy


I'm referring to the fact that he's a smug ass. And I say that as an evolutionary theory loving athiest. He gives athiests a bad name just like people who talk about audio listening issues while dismissing ABX. Just my opinion.
Title: The Dishonesty of Sighted Listening Tests
Post by: Woodinville on 2009-04-13 09:23:43
No, rec.audio.opinion, that legendary cesspool where defenses of double blind tests are sometimes replied to with allegations of child molestation or drug abuse.


Not to mention coprophilia, coprophagia, deviant sexual practices, terrorists, ...

The best cesspool company in the world could not pump out rec.audio.opinion.
Title: The Dishonesty of Sighted Listening Tests
Post by: Woodinville on 2009-04-13 09:25:12
He might get old and cranky and turn into jj or Arny before our very eyes.

Eh?

Trust me, he's already there.
Title: The Dishonesty of Sighted Listening Tests
Post by: Woodinville on 2009-04-13 09:28:19
Hotshot: That is also my conclusion. It is largely a waste of time to try to convert faith-based audiophiles to accept and apply science towards audio research and evaluation.


Well, I wandered over there and gave you all the support I could muster without suggesting that they were first cousins to creationists.

I was as polite as I could manage given the heights of arrogance and depths of hatred of science that simply floated away from the thread.
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-13 17:40:18
If you're referring to the fact that in 'The God Delusion' he fails to consider 'sophisticated' theological arguments for religious beliefs in any depth, I refer you to the No True Scotsman (http://en.wikipedia.org/wiki/No_true_Scotsman) fallacy.

This could get very OT, but I think there is a relevance. Dawkins' failure to engage with sophisticated theisms is a real weakness in his intellectual position, and the No True Scotsman fallacy is irrelevant, unless we accept Dawkins' claim to define real theism--which is itself a No True Scotsmanism. I don't want to pursue that any further, partly because it's OT and partly because I have no clue about whether or not the universe is genuinely intelligible (to believe which roughly = Deism), or whether or not any human values correspond to values in some way inherent in the universe (to believe which roughly = Theism).



Dawkins is addressing the dominant cultural paradigms of God-belief -- not the scholastic or Deistic or Einsteinian arguments for a highly abstract God (which he actually *specifically notes* he is not addressing, in his book)  .  The theologically minded claim that, well, what the common folk believe in isn't the REAL God-- hence the No True Scotsman.

Quote
What is relevant is that Dawkins' style is deeply unpersuasive to the undecided;



And you know this how?  There is testimony from readers to the contrary.
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-13 17:48:30
I'm referring to the fact that he's a smug ass.


Are you American?  I ask because IME, the combination of articulate speech plus a middle/upperclass British accent plus being *right* more often than not on a 'controversial' issue, gets interpreted as 'smugness' by some of my countrymen.


Quote
And I say that as an evolutionary theory loving athiest. He gives athiests a bad name just like people who talk about audio listening issues while dismissing ABX.


er..what?  Sorry, I don't see that as a good analogy at all...you're comparing Dawkins to audio 'subjectivists'??

Quote
Just my opinion.



Wait, I thought it was 'fact'?  Make up your mind.   
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-13 19:32:42
Axon,
Just for the record: I'm already old and cranky, at least according to my wife and kids.  When the anti-blind- test flamers come up with some compelling arguments and scientific data to back them up,  then I will start to take them seriously, and respond.


Eh? Trust me, he's already there.

OK, OK - seems like we have enough "golden ears" to establish Sean's age and crankiness. No blind testing necessary. Heh.


Well, I wandered over there and gave you all the support I could muster without suggesting that they were first cousins to creationists.

I was as polite as I could manage given the heights of arrogance and depths of hatred of science that simply floated away from the thread.
Thanks, and I had the same thoughts as you regarding creationism. It took a very substantial amount of self-control to avoid trotting out the jesus horse (http://www.b3tards.com/u/758f65c94bf4f6553036/jesusdino2.jpg).

The phrase "audio creationism" seems strangely apt, now that I think about it.

But, I'm sure we could take that farther than even some HA members might be comfortable with, sooooo...
Title: The Dishonesty of Sighted Listening Tests
Post by: MichaelW on 2009-04-13 21:49:40
Dawkins is addressing the dominant cultural paradigms of God-belief -- not the scholastic or Deistic or Einsteinian arguments for a highly abstract God (which he actually *specifically notes* he is not addressing, in his book)  .


Are you American? Naive Creationism is not the dominant paradigm of God-belief in most of the world.

But I must stop now.
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-13 22:28:26
Dawkins is addressing the dominant cultural paradigms of God-belief -- not the scholastic or Deistic or Einsteinian arguments for a highly abstract God (which he actually *specifically notes* he is not addressing, in his book)  .


Are you American? Naive Creationism is not the dominant paradigm of God-belief in most of the world.

But I must stop now.
It's more prevalent (http://science.drvinson.net/polls) than one might think. Of course, in the US, it truly is dominant (http://www.gallup.com/poll/21814/Evolution-Creationism-Intelligent-Design.aspx)(barely non-majority, but punching far above its weight politically).

I haven't read Dawkins yet, but I was immensely entertained by Hitchens's brilliant and thickly insulting style. Right up your alley, krab. I sorta kept it hidden while reading it over Christmas, but a relative (a liberal Catholic who hates Benedict so much I am halfway expecting her to start celebrating Guy Fawkes Day, effigies and all) chuckled when I told her about it.

Mods: should this be split off into Off Topic?
Title: The Dishonesty of Sighted Listening Tests
Post by: MichaelW on 2009-04-14 01:26:13
OK, I think I can keep it relevant to HA.

I have no problems with Dawkins, Hitchens et al. slagging off stupid beliefs. Nor do I have any problem with atheism (me, Camus's vision is persuasive, if overly hopeful). However, it seems to me just plain wrong to claim you're making an argument for atheism when you're attacking one particular form of belief (however prevalent: US hyper-evangelicalism has got nothing to do with, for instance, the beliefs of Hindus).

The analogy is what actually happens when someone who doesn't yet have a firm grip on the difficulty of hearing straight comes here and asks a question like, for instance, "Where are the ABX tests for FLAC?" The poser of that question got accused of promoting the belief that lossless isn't lossless, when he was 1. asking a question, 2. rather doubtful of the claim, 3. aware that the only way anything could happen was in the playback. And, in fact, long ago with feeble puters and bad soundcards, it seems there could be problems in the playback. But someone who wasn't as firmly grounded as OP in that thread could well have concluded that asking for ABX for FLAC was like suggesting that the Emperor was stark naked.

Of course, it depends whether the idea is to spread an appreciation for sound methodology and rational thought. There's no reason why anyone should have that objective, and jeering at the stupid and ill-informed is maybe fun enough. Truly, I mean that: there is no obligation to stop the ignorant being parted from their money, nor to stop self-appointed arbiters of sound exercising their little czardoms. But good to know what one is actually doing.
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-14 04:28:50
It took a very substantial amount of self-control to avoid trotting out the jesus horse (http://www.b3tards.com/u/758f65c94bf4f6553036/jesusdino2.jpg).


That's too small to be a Jesus horse.  You must have meant this (http://farm1.static.flickr.com/128/414998399_4b1b06b1b8_b.jpg)
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-14 04:35:20
Dawkins is addressing the dominant cultural paradigms of God-belief -- not the scholastic or Deistic or Einsteinian arguments for a highly abstract God (which he actually *specifically notes* he is not addressing, in his book)  .


Are you American? Naive Creationism is not the dominant paradigm of God-belief in most of the world.


You sure about that?  You think most people are more Deist, say, than not?  (I can hardly wait for you to distinguish for me between 'naive' and 'sophisticated' creationism; I'm guess the anthropic principle will make an appearance?)

Anyway, we're talking about two different things.  I was referring to 'The God Delusion', which isn't mainly about evolution vs creationism.

If anything , disdaining Dawkins on the basis of his arguments against creationism (rather than the existence of God), is even more bizarre.



Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-14 04:50:16
OK, I think I can keep it relevant to HA.

I have no problems with Dawkins, Hitchens et al. slagging off stupid beliefs. Nor do I have any problem with atheism (me, Camus's vision is persuasive, if overly hopeful). However, it seems to me just plain wrong to claim you're making an argument for atheism when you're attacking one particular form of belief (however prevalent: US hyper-evangelicalism has got nothing to do with, for instance, the beliefs of Hindus).


LOL.  *Are you sure about that?*

'Fundamentalist'-style religious belief certainly exists among Hindus -- politically powerful ones at that.  Christian-style "evangelicalism' isn't necessary for it to exist.

Have you *read* The God Delusion, or only read *about* it?  The arguments therein are mainly, but not *only* focused on the Judeo-CHristian-Islamic monotheistic triad (and btw do you consider these three together to be ' one particular form of belief'? That makes you more like Dawkins than perhaps you realize. ) .  A discussion of polytheism, including Hinduism, begins on p. 32.  It ends like so:

Quote
I’m not attacking any particular version of God or gods. I’m attacking God, all gods, anything and everything supernatural, wherever and whenever they have been or will be invented.
.



Quote
The analogy is what actually happens when someone who doesn't yet have a firm grip on the difficulty of hearing straight comes here and asks a question like, for instance, "Where are the ABX tests for FLAC?" The poser of that question got accused of promoting the belief that lossless isn't lossless, when he was 1. asking a question, 2. rather doubtful of the claim, 3. aware that the only way anything could happen was in the playback. And, in fact, long ago with feeble puters and bad soundcards, it seems there could be problems in the playback. But someone who wasn't as firmly grounded as OP in that thread could well have concluded that asking for ABX for FLAC was like suggesting that the Emperor was stark naked.


Sorry, you've lost me.  The analogy of Dawkins put forth by gordongekko  was  to an someone who ' talks about audio listening issues while dismissing ABX'.  So, what is 'ABX' analogous to here -- supposedly good arguments for the existence of God?

Dawkins *is* suggesting that the Emperor is naked.
Title: The Dishonesty of Sighted Listening Tests
Post by: 2Bdecided on 2009-04-14 16:15:32
Dawkins appears to hate religion more than he loves science. That's what I gathered when he was given five minutes to say whatever he liked on Radio 4 the other year.

Quite sad really - I was hoping for five minutes of inspirational science - we could do with some of that on the BBC. There's enough religion bashing already (though of course there's some religion too).


I fear we've become a little too quick to jump on people here at HA. I also fear it's becoming a little like a club where people compete to prove that they are "in" it by seeing who can give the most "on message" answer - the result being that one person asks about discrete stereo, and he gets 18 responses telling him why he should use joint stereo. Really, come on, one response is enough!

We shouldn't assume that all the prejudices of stereophile are present in every newbie's first post!


In the world where specialist shops are dying, and people buy most things through the internet, I like the idea of figuring out some useful performance metrics and attaching them to loudspeakers. It's a challenge, but the journey of a thousand miles starts with the first step etc. If it can push companies to solve real problems, so much the better.


The real tragedy with the subjectivist crowd is that if an equivalent amount of energy was put into blind testing, we could have got so much further with audio than we have. However, I don't think it's worth making this point to the people posting to the stereophile forum - and I doubt many lurkers hack it there for long (it's a bit mental, and people don't actually care any more), though I could be wrong.

Cheers,
David.
Title: The Dishonesty of Sighted Listening Tests
Post by: botface on 2009-04-14 17:30:28
I fear we've become a little too quick to jump on people here at HA. I also fear it's becoming a little like a club where people compete to prove that they are "in" it by seeing who can give the most "on message" answer - the result being that one person asks about discrete stereo, and he gets 18 responses telling him why he should use joint stereo. Really, come on, one response is enough!

Spot on.
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-14 18:39:11
Quote
I fear we've become a little too quick to jump on people here at HA. I also fear it's becoming a little like a club where people compete to prove that they are "in" it by seeing who can give the most "on message" answer - the result being that one person asks about discrete stereo, and he gets 18 responses telling him why he should use joint stereo. Really, come on, one response is enough! We shouldn't assume that all the prejudices of stereophile are present in every newbie's first post!
You won't see a disagreement from me on that! (http://www.hydrogenaudio.org/forums/index.php?showtopic=70272&st=40)

That said... how does that apply to this thread, specifically?

Quote
The real tragedy with the subjectivist crowd is that if an equivalent amount of energy was put into blind testing, we could have got so much further with audio than we have. However, I don't think it's worth making this point to the people posting to the stereophile forum - and I doubt many lurkers hack it there for long (it's a bit mental, and people don't actually care any more), though I could be wrong.
True, but the lurkers only leave insofar as controversy and/or ignorance is exposed. I suspect the main posters there have a substantial amount of influence on lurkers who otherwise are not aware of the DBT debate.
Title: The Dishonesty of Sighted Listening Tests
Post by: gorgekko on 2009-04-14 21:32:32
I'm referring to the fact that he's a smug ass.


Are you American?  I ask because IME, the combination of articulate speech plus a middle/upperclass British accent plus being *right* more often than not on a 'controversial' issue, gets interpreted as 'smugness' by some of my countrymen.


Nope, I'm a Canadian. And a Anglophile at that so I have no problem with upperclass British accents. Hell, my grandmother sounded like Her Majesty.

I just don't like the man. I think he does as much damage to his cause as he does the intelligent designers/creationist.
Title: The Dishonesty of Sighted Listening Tests
Post by: MichaelW on 2009-04-14 22:27:01
I fear we've become a little too quick to jump on people here at HA.

Yes indeed, David: that's the point I was trying to raise (though I probably muddied the issue: sorry about that). Having a giggle at directional cables and magic pebbles is good clean fun, but some of the people who come here are either just uninformed (which everyone is about most things) or, more troubling to them, they've got information which is out of date. And things do get out of date very quickly in the digital world.

Anyone who's done any teaching knows that there are some questions which intelligent people tend to ask at particular points in their learning. They've been asked so often that there are either well-understood answers, or it's clear there is no simple answer. It can be tedious to keep on dealing with them, but it's not the questioner's problem that they've been asked before.
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-15 06:51:49
Dawkins appears to hate religion more than he loves science. That's what I gathered when he was given five minutes to say whatever he liked on Radio 4 the other year.


Was  the other year, the year The God Delusion was published, by chance?

C'mon, you're taking five minutes of output from someone who's been one of the two best-known public champions of evolutionary biology since the 1970s, authoring a string of science books since then, and on that basis deciding Dawkins 'hates religion more than he loves science'? 


Quote
The real tragedy with the subjectivist crowd is that if an equivalent amount of energy was put into blind testing, we could have got so much further with audio than we have.

However, I don't think it's worth making this point to the people posting to the stereophile forum - and I doubt many lurkers hack it there for long (it's a bit mental, and people don't actually care any more), though I could be wrong.



I was shocked (pleasantly) to see Gordon Holt, of all people (being the founder of Stereophile) express much the same sentiment about the lost opportunity for blind testing, in an Q&A with Stereophile's current editor.
Title: The Dishonesty of Sighted Listening Tests
Post by: HotshotGG on 2009-04-15 14:12:55
What did I do?  . Next time I will refrain from using Richard Dawkins as an analogy to this on going DBT war it was probably a very bad example. The only reason it came to mind is I took numerous Philosophy classes in college and at one time was majoring in Philosophy before my new job took me into IT. I think this is a kind of topic that would be useful in the off-topic section of the forum rather then in Sean Olive's critique of Sighted Listening Tests. I was trying use the analogy comparing scientific DBT to faith based audiophiles. My attention wasn't to spark any Philosophical or Political debate although it often comes down to that sometimes on here. I too would be happy discuss both just in the off topic section of the forum. 
Title: The Dishonesty of Sighted Listening Tests
Post by: Axon on 2009-04-15 19:38:55
I was shocked (pleasantly) to see Gordon Holt, of all people (being the founder of Stereophile) express much the same sentiment about the lost opportunity for blind testing, in an Q&A with Stereophile's current editor.

I was also pleasantly shocked to see JA actually publish it
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-15 19:47:58
I was shocked (pleasantly) to see Gordon Holt, of all people (being the founder of Stereophile) express much the same sentiment about the lost opportunity for blind testing, in an Q&A with Stereophile's current editor.

I was also pleasantly shocked to see JA actually publish it



I was further amused by the various vitriolic and bitterly betrayed reactions to it...even today, as on that idiotic forum thread.
Title: The Dishonesty of Sighted Listening Tests
Post by: Woodinville on 2009-04-18 08:23:27
I was shocked (pleasantly) to see Gordon Holt, of all people (being the founder of Stereophile) express much the same sentiment about the lost opportunity for blind testing, in an Q&A with Stereophile's current editor.

I was also pleasantly shocked to see JA actually publish it



I was further amused by the various vitriolic and bitterly betrayed reactions to it...even today, as on that idiotic forum thread.


Check out Jan Vigne's latest (as of now) response to me at the stereophile forum. I begin to suspect that the gentleman is disturbed. I urge you NOT to reply there.
Title: The Dishonesty of Sighted Listening Tests
Post by: singaiya on 2009-04-18 17:56:46
Are actual blind people useful for listening tests? I thought when you lose one sense, your others become more sensitive. One one hand I can see why they may not be representative of "normal" hearing, but on the other hand they may be helpful for exploring the outermost boundaries of hearing. Perhaps it depends on what is being tested.

Maybe it's too off topic, but I've been curious about it.
Title: The Dishonesty of Sighted Listening Tests
Post by: pdq on 2009-04-18 22:17:56
Often someone with flawed hearing will be more sensitive to certain kinds of encoding artifacts than someone with so-called normal hearing.
Title: The Dishonesty of Sighted Listening Tests
Post by: ff123 on 2009-04-19 03:58:43
Are actual blind people useful for listening tests? I thought when you lose one sense, your others become more sensitive. One one hand I can see why they may not be representative of "normal" hearing, but on the other hand they may be helpful for exploring the outermost boundaries of hearing. Perhaps it depends on what is being tested.

Maybe it's too off topic, but I've been curious about it.


Some blind people can do some amazing things by clicking their tongues and listening to the echos coming back.

http://www.youtube.com/watch?v=uobuBc2GO0o (http://www.youtube.com/watch?v=uobuBc2GO0o)

Maybe their highly developed sense of sound-space could be useful for improving surround-sound systems (passive locating).  It's hard for me to even think of a reproducing system which could hope to fool one of these people.
Title: The Dishonesty of Sighted Listening Tests
Post by: ShowsOn on 2009-04-19 05:28:35
Dawkins appears to hate religion more than he loves science.

Yeah, I mean he only has  a Ph.D. in Zoology.
Title: The Dishonesty of Sighted Listening Tests
Post by: singaiya on 2009-04-19 18:30:22
Are actual blind people useful for listening tests? I thought when you lose one sense, your others become more sensitive. One one hand I can see why they may not be representative of "normal" hearing, but on the other hand they may be helpful for exploring the outermost boundaries of hearing. Perhaps it depends on what is being tested.

Maybe it's too off topic, but I've been curious about it.


Some blind people can do some amazing things by clicking their tongues and listening to the echos coming back.

http://www.youtube.com/watch?v=uobuBc2GO0o (http://www.youtube.com/watch?v=uobuBc2GO0o)


That's amazing, thanks for the link!
Title: The Dishonesty of Sighted Listening Tests
Post by: honestguv on 2009-04-22 20:34:30
quote name='solive' date='Apr 12 2009, 03:47' post='627052'

Apologies for the break.

>> If there was a business case for assessing performance in a scientifically valid manner then
>> it would be done. Gearing up to routinely peform scientifically valid experiments is not a
>> significant issue
>
> Contrary to what you claim, the investment in anechoic chambers, listening labs, equipment,
> speaker movers, training listeners, engineering/scientific staff IS a significant issue and
> cost.

This was not the claim. The claim was that if there was a business case it would be done. Around the end of the stereo boom and before the mainstream audiophile got going it was not unusual for acoustic professionals and their facilities to be involved with the assessment of hi-fi equipment at least in the UK. The home audio market was quite different at this time and what made business sense was also different.

> The success of the Canadian loudspeaker industry is in large part due to scientific assessment
> of performance, and they readily acknowledge it in their marketing literature. I would argue
> that is an example of a business case where scientific assessment of performance has paid off.

I am not sure the "large part" would stand up to close scrutiny. People choose to purchase particular loudspeakers for a wide range of reasons and I suspect there might be an element of your seeing what you want to see?

>> A passive consumer is going to get fed nonsense but there is no reason an active one cannot
>> access information in order to put together a good hi-fi for a reasonable price.
>
> Again, are you sure about that? Show me the technical journals (J AES ?) I can go to find out which
> loudspeaker is the best one to purchase? Technical journals like AES or JASA don't publish the names of
> the products. Also, please give some names of professionals where a consumer can get unbiased,
> comprehensive scientific subjective and objective performance data that tells them which is the best
> loudspeaker to purchase for say $2000?

It is a passive consumer that needs the type of information you suggest that tells them what brands to purchase based on paper ratings. If it was in the interests of the industry to provide such information it would have been done long ago. Given that it is clearly not in the interests of the industry, how much taxpayers money should be spent by the state to provide this information and how much damage would it do in terms of jobs, tax revenue and the like?

An active consumer primarily needs access to knowledge about what is important and why. This knowledge is available from non-audiophile sources for those that consider the effort worthwhile.

> His credentials and experience in the high-end industry surely make his opinion as worthy as yours,
> mine or anyone elses'.

It is hard to reply without knowing what you mean by opinion.

You appeared to put him forward as someone whose statements should be given weight. It may be interesting that he has changed his view somewhat in his old age and that it has shifted in the "correct" direction but is the basis on which he forms his judgements the rational one based on reproducible evidence? Do some of his other statements reinforce or counter the view that he is rational?

> In the context of a listening test, "true" means the listener is only responding to acoustical
> changes caused by the variable being manipulated. The marketing department deals with how the
> customer responds to the shininess of the hi-fi -- not me. I only care about the sound quality.

I would suggest this is a rather peculiar use of the word true. Somebody sitting at home listening to their hi-fi has no way to get at your "true". For example, I would argue that the "natural" sighted listening rating of speaker S in the link at the top was the true one and not the "artificial" blind listening result.

If removing the visual cues is important for a true assessment then what about the room?

>> Blind tests allow engineers to sort out what is due to what and this is useful when juggling the
>> compromises>>
>
> And why wouldn't it also be a useful tool for reviewers or consumers for sorting out which products
> make the best compromises?

An audiophile reviewer writes audiophile reviews for audiophile publications. Commercial audiophile publications are primarily vehicles for audiophile advertising. If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.

It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.

> Because it would provide accurate and reliable information that would make it easier for consumers to
> choose the loudspeaker that best matches their taste and application.

Is this a good or bad thing for a luxury products industry?

Is this a good or a bad thing for the consumers of luxury products? How important are the illusions?

> The industry bemoans that consumers don't care about high quality audio anymore.

The industry works with and to an extent modifies what audiophile consumers want. The less audiophiles are interested in (expensive) technical performance and more interested in (cheaper) perceived perfomance the bigger the profit margins. In the UK companies like Linn and Naim have been very successfull doing this over the life of the audiophile sector while most of the more established companies from the hi-fi boom that tried to hang onto to the old ways largely disappeared although a few of the brands are still around. Whole sectors of the audiophile industry like cables and capacitors are completely based on perceived performance.

Non-audiophile consumers have a much wider choice of entertainment sources today and I suspect the silliness of the audiophile sector is going to have limited appeal.

> Part of the problem falls upon us because we have failed to educate them, and give them compelling,
> trustworthy, accessible data that clearly shows them there is a significant difference in sound
> quality between a $200. HTIB and a $3000 home theatre setup.

Why is it the industries job to educate consumers? Surely it is the industries job to make profits for their shareholders, pay tax to the state, keep their workforce happy and persuade consumers to purchase their products. Does educating consumers of luxury products help or hinder this?

Consumers buy what they want. They do not buy what knowledgeable people consider best for them but what what they want.
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-22 23:22:34
Consumers buy what they want. They do not buy what knowledgeable people consider best for them but what what they want.


That begs the question:  where does the particular 'want' come from?  Consumer preference doesn't form de novo in a vaccuum.

Consumers DO respond to the recommendations of people they BELIEVE are more knowledgeable than themselves.  That's one
component of marketing, and I'd say it's an especially significant one in audio, a hobby with a strongly technical component.

One problem is that an uninformed consumer -- not to mention the lazy journalist -- has little way of knowing
whether an 'expert' is really an expert, and whether they are being fed science or pseudoscience.  How many would-be audiophiles
who have bought the 'expert' Robert Harley's books,  have been able to  tell which of his claims are absurdities, and which are
well-grounded in fact?
Title: The Dishonesty of Sighted Listening Tests
Post by: rpp3po on 2009-04-23 00:21:04
An audiophile reviewer writes audiophile reviews for audiophile publications. Commercial audiophile publications are primarily vehicles for audiophile advertising. If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.

It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.


I think that is exactly the point. Honest audio review does not have a business case. Imagine the 'perfect' objective and honest audio magazine. They would have to tell you in one CD player review after another that both measurements and blind tests aren't significantly different from 99% of the other tested units. Then again and again they could prove the bullshitness of cable voodoo.

The only interesting area would be properly setup blind speaker testing. But what kind of a business case would that be? The better your reviews the sooner you lose your subscribers - happily knowing that they have bought the objectively best gear for their budget and everything else was voodoo. Maybe five years from then your potential customer may buy one ore two issues again for upgrade advice, but that's all!

Now look at the business case for voodoo reviewers! Their subscribers may eagerly await every new monthly issues just to get inspired how they could 'optimize' their setup even more. You know, finding the proper cable for exactly your combination of amp and speaker is almost art!  It can take several tries to find perfection. And that's only the beginning. Jitter must be controlled, crossovers must be modded, and so on. A never ending journey. Just the day when you think that the speaker cable is finally the perfect match for your setup you start thinking about replacing your amp with a tube driven one. Your neighbor's sounds just so much warmer....
Title: The Dishonesty of Sighted Listening Tests
Post by: krabapple on 2009-04-23 01:06:20
An audiophile reviewer writes audiophile reviews for audiophile publications. Commercial audiophile publications are primarily vehicles for audiophile advertising. If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.

It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.


I think that is exactly the point. Honest audio review does not have a business case. Imagine the 'perfect' objective and honest audio magazine. They would have to tell you in one CD player review after another that both measurements and blind tests aren't significantly different from 99% of the other tested units. Then again and again they could prove the bullshitness of cable voodoo.

The only interesting area would be properly setup blind speaker testing. But what kind of a business case would that be? The better your reviews the sooner you lose your subscribers - happily knowing that they have bought the objectively best gear for their budget and everything else was voodoo. Maybe five years from then your potential customer may buy one ore two issues again for upgrade advice, but that's all!


In addition to loudspeakers, there's also room acoustics, networked audio and computer/audio interfaces, and the ever-burgeoning feature sets of AVRs (esp. digital 'room correction') and players.

A paper or online journal that focused on this stuff -- stuff that actually makes an audible difference -- and had the financial power to present it in a graphically slick, well-written and well-researched way -- could perhaps find a niche.

But given that formerly 'objectivist' leaning glossy Sound & Vision has gone flabby, while the plucky but dowdy Sensible Sound appears to be on hiatus,  it appears the market for no-nonsense audio journalism not all that hot.  Though I'd guess that in an economic downturn, people would be *pleased* to know what great sound they could get for relatively little money.
Title: The Dishonesty of Sighted Listening Tests
Post by: solive on 2009-04-23 04:02:33
[quote name='honestguv' date='Apr 22 2009, 12:34' post='629015']
quote name='solive' date='Apr 12 2009, 03:47' post='627052'

<<This was not the claim. The claim was that if there was a business case it would be done. >>

In the very next sentence the poster also claimed "Gearing up to routinely peform scientifically valid experiments is not a significant issue" and that was the claim which I replied is not true: gearing up to do valid listening tests is a serious investment, which I believe pays for itself over the long term. Whether you believe there is a good business case for this approach  is not relevant to me.  I happen to believe (and so does my company) that a scientific approach towards audio product research, development, testing and validation is the best approach for staying in business over the long-term.  Please remember that over 70% of our business is automotive audio with customers like BMW, Mercedes, Toyota/Lexus and Hyundai/Genesis, who don't tolerate the laissez-faire approach towards audio testing to which you subscribe. If you tell these companies that your audio system sounds better than the competition, they want to see hard scientific valid data to back it up.  We cannot simply "pull some numbers out of a hat" and say, "it sounds really good -- take my word for it."

.<<Given that it is clearly not in the interests of the industry, how much taxpayers money should be spent by the state to provide this information and how much damage would it do in terms of jobs, tax revenue and the like?>>

Where did I suggest that the state provide consumers valid product information??  I suggested that private companies and audio review magazines might provide this information voluntarily-- not the goverment!

<<An active consumer primarily needs access to knowledge about what is important and why. This knowledge is available from non-audiophile sources for those that consider the effort worthwhile. >>

OK. Again, I ask you where exactly are these "non-audiophile sources"??  Is it classified FBI information?? 

<<It is hard to reply without knowing what you mean by opinion.>>

Opinion 1 a: a view, judgment, or appraisal formed in the mind about a particular matter (Websters)  Read the Gordon Holt interview where he states his opinion. Read it, and if you still find it hard to reply don't fret too much. You don't  have to reply to every point I make unless you just like to be argumentative..

<<You appeared to put him forward as someone whose statements should be given weight. It may be interesting that he has changed his view somewhat in his old age and that it has shifted in the "correct" direction but is the basis on which he forms his judgements the rational one based on reproducible evidence? Do some of his other statements reinforce or counter the view that he is rational?>>

I'll let you figure that out. I have more scientific testing to do..

<<I would suggest this is a rather peculiar use of the word true. >>

Again I refer you to the Websters: TRUE (1) being in accordance with the actual state of affairs <true description> (2): conformable to an essential reality. I think my use of the word "True" is perfectly correct in the context of  describing the effect that visual biases have on the perception of sound. If you disagree please take this up with Websters - not me.

<<If removing the visual cues is important for a true assessment then what about the room?>>

Glad you asked. We have acoustically transparent curtains that hide the boundaries of the walls. We also have a binaural room scanning device that completely eliminates all visual cues. You can read about it in my blog (http://seanolive.blogspot.com)

<<If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.>>

Perhaps. Scientific reviews of $1000 per ft wires won't help sell advertising or make interesting reading because there is either nothing much to say -- or something really bad to say. When really bad things are said, the magazine risks being sued by the company.

<<It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.>>

Try Consumer Reports - for the past 2 years, they have  started to do loudspeaker ratings based on comprehensive anechoic measurements -- Other organizations like CEA are considering similar specifications for loudspeakers, although the industry is very reluctant.

<<Is this a good or bad thing for a luxury products industry?>>

Regarding perceptually meaningful product specifications, it depends on your perspective. If you make a really good product, the answer is  "YES". If you make a crappy product then the answer is "NO." If you are a consumer then the answer is "YES" -- unless you are unfortunate to already own the crappy product.


<<The industry works with and to an extent modifies what audiophile consumers want. The less audiophiles are interested in (expensive) technical performance and more interested in (cheaper) perceived perfomance the bigger the profit margins>>

So you are saying there is more money to be made in deceiving audiophiles  because you can charge them a lot more for something that has poor technical performance. And you have no moral qualms about that?
What I don't understand is why you think technical performance needs to be expensive. I've measured and tested many $500 loudspeakers that have technically superior performance (and better sound) than $11,000 loudspeakers. The correlation between price and technical/sound quality performance with loudspeakers is very low.  My explanation for this  is that most companies that sell very expensive speakers don't sell many of them, which means their R&D budgets for proper design, testing and validation are very low. The other problem is that they are able to get away with it.

<<Non-audiophile consumers have a much wider choice of entertainment sources today and I suspect the silliness of the audiophile sector is going to have limited appeal.>>
Agreed. So Gordon Holt was correct.


<<Why is it the industries job to educate consumers? Surely it is the industries job to make profits for their shareholders, pay tax to the state, keep their workforce happy and persuade consumers to purchase their products. Does educating consumers of luxury products help or hinder this?>.

Do you think keeping the consumer ignorant and deceiving  them is a much better approach?  That really turned out well for Enron, AIG,etc,etc  Educating consumers about your product makes good business sense because consumers are more likely to purchase your product, and employees and shareholders are more likely to invest in the company.  The cost of education is part of the price for doing business, and in fact increases business. It has nothing to do with the state - which you  seem to be obsessed about.  Many companies belong to audio trade associations (AES, CEA, ALMA) which also help pay for the cost of educating consumers about new CE technologies like Blu-ray,etc,etc..

<< Consumers buy what they want. They do not buy what knowledgeable people consider best for them but what what they want.>>

You really are quite naive about how consumers make purchase decisions, and the audio business in general.  If consumers just bought what they want -- there would be no media advertising, marketing budgets, audio magazines, celebrity endorsements, or store sales people on commission to influence and manipulate the consumer's purchase decision.  In Japan, the audio consumer is greatly influenced by what experts say (ie the reviewer) because that is the cultural norm. There is a great opportunity to influence the consumer's purchase decision in a positive way by providing them useful information about the performance of the product. This seems to be  completely lost on you, as you seem to think  there is more profit in deception, ignorance and lies --
Title: The Dishonesty of Sighted Listening Tests
Post by: MichaelW on 2009-04-23 04:15:13
Magazines don't make most of their money from subscribers/newstand purchasers; they make it from advertisers. Which explains practically everything, yes?
Title: The Dishonesty of Sighted Listening Tests
Post by: singaiya on 2009-04-23 04:33:40
<<Why is it the industries job to educate consumers? Surely it is the industries job to make profits for their shareholders, pay tax to the state, keep their workforce happy and persuade consumers to purchase their products. Does educating consumers of luxury products help or hinder this?>.

Do you think keeping the consumer ignorant and deceiving  them is a much better approach?  That really turned out well for Enron, AIG,etc,etc  Educating consumers about your product makes good business sense because consumers are more likely to purchase your product, and employees and shareholders are more likely to invest in the company.  The cost of education is part of the price for doing business, and in fact increases business. It has nothing to do with the state - which you  seem to be obsessed about.  Many companies belong to audio trade associations (AES, CEA, ALMA) which also help pay for the cost of educating consumers about new CE technologies like Blu-ray,etc,etc..


I agree totally. Though differences like this (it appears) may indicate differences in philosophical perspectives that are more basic; indicating a divide on where someone's coming from - industry or science. Two areas not necessarily mutually exclusive, but which have been uncomfortable together in the consumer realm.
Not sure if that comment makes sense, but in any case, solive you make the point way better than I could!
Title: The Dishonesty of Sighted Listening Tests
Post by: rpp3po on 2009-04-23 15:39:00
I happen to believe (and so does my company) that a scientific approach towards audio product research, development, testing and validation is the best approach for staying in business over the long-term.  Please remember that over 70% of our business is automotive audio with customers like BMW, Mercedes, Toyota/Lexus and Hyundai/Genesis, who don't tolerate the laissez-faire approach towards audio testing to which you subscribe. If you tell these companies that your audio system sounds better than the competition, they want to see hard scientific valid data to back it up.  We cannot simply "pull some numbers out of a hat" and say, "it sounds really good -- take my word for it.


For the domain of professional publications and consulting I certainly do believe in the business case of objective audio evaluation. Not for consumers, though, because of the described difficulties regarding regular and lasting customer retention.
Title: The Dishonesty of Sighted Listening Tests
Post by: Arnold B. Krueger on 2009-04-23 16:35:01
Why is it the industries job to educate consumers?


Educating consumers is a well-known and highly effective means of maximzing shareholder value.

It's all about human nature. People will buy things they don't understand at all, but generally only under duress. An example of such a thing would be medications.

Generally people what to understand something about the how and why of anything they buy. Why do some prefer Coke over Pepsi? It's the one that tastes best to me. Why do some prefer one loudspeaker over another? It's the one that sounds best (for the money) to me.

Why does a consumer think that one speaker sounds better than another? If he knows some reasons why a speaker should sound better, many consumers will perceive that it sounds better when it is demonstrated to them.

Education is a means to get the consumer's mind thinking favorable thoughts about your product, even before he has any personal experience with it. It's a way to keep him thinking favorable thoughts about your product even if it is harder to buy or more expensive, or heaven forbid sounds bad in the show room.
Title: The Dishonesty of Sighted Listening Tests
Post by: pdq on 2009-04-23 17:02:05
Magazines don't make most of their money from subscribers/newstand purchasers; they make it from advertisers. Which explains practically everything, yes?

Which is why Consumer Reports, which has no advertising revenue, is far more likely to provide unbiased information.
Title: The Dishonesty of Sighted Listening Tests
Post by: honestguv on 2009-05-01 13:24:51
quote name='solive' date='Apr 23 2009, 05:02' post='629100'

Apologies again for the break. You do not seem to be responding in the spirit hoped, perhaps I should be briefer.

> Please remember that over 70% of our business is automotive audio with customers like BMW,
> Mercedes, Toyota/Lexus and Hyundai/Genesis, who don't tolerate the laissez-faire approach
> towards audio testing to which you subscribe.

I subscribe? Curiously, one of the reason for not posting was a visit to Torino (not in your list) although the sound discussed was not from the radio.

> Where did I suggest that the state provide consumers valid product information?? I suggested
> that private companies and audio review magazines might provide this information
> voluntarily-- not the goverment!

Since it is not in the interests of the industry (unless you dispute this?) then it has to be funded by the state. In the 1970s, at least in Europe with which I am familiar, there was a growing move by the state in this direction but then it all went very quiet when the home audio industry started to move from hi-fi to audiophile. All that fuss about stretching specified Watts and then not a peep about magic one-way gold-plated audiophile speaker cables. Why was that?

>> If removing the visual cues is important for a true assessment then what about the room?
>
> Glad you asked. We have acoustically transparent curtains that hide the boundaries of the
> walls. We also have a binaural room scanning device that completely eliminates all visual
> cues. You can read about it in my blog

That was not the question intended. If audiophiles should not include their visual/knowlege cues in their assessment of loudspeakers then should they substitute some other room response when judging the "true" performance of their speakers?

>> Is this a good or bad thing for a luxury products industry?
>
> Regarding perceptually meaningful product specifications, it depends on your perspective. If
> you make a really good product, the answer is "YES". If you make a crappy product then the
> answer is "NO." If you are a consumer then the answer is "YES" -- unless you are unfortunate
> to already own the crappy product.

How much are luxury products about technical performance and how much are they about perceived performance (e.g. status). How a professional in a lab or studio judges the abilities of hardware is substantially different to how the audiophiles on Stereophile's forum judge the abilities of hardware. The latter may well believe a lot of nonsense about how the world around them works but it is a valid part of how they judge.

> So you are saying there is more money to be made in deceiving audiophiles because you can
> charge them a lot more for something that has poor technical performance. And you have no
> moral qualms about that?

Is it morally wrong to give people what they want? The posters on Stereophile's forum want audio equipment to be magical and will work hard to push away and ignore information that goes against this. It is no different with Bose and B&O equipment in the consumer audio sector. Customers are purchasing far more of what goes with the brand than they are sound quality. I have no problem with this.

What I would be uncomforatble with is the consequences of state involvement in imposing full and fair specifications for luxury products like home audio equipment. So long as the state supports education and access to valid information exists then, in my judgement, that is where the line should be drawn.

> What I don't understand is why you think technical performance needs to be expensive.

Good technical performance for loudspeakers requires substantial cabinets, magenetic material, production control, limited demand, and the like which is going to keep costs up. Contrast this with, say, Bose products where more emphasis is placed on the marketing.

> Do you think keeping the consumer ignorant and deceiving them is a much better approach?

Consumers have access to good and bad information and can choose who and what to believe. If better consumer information about luxury products is to be provided then it will have to be funded. I think we may disagree about the funding.

> Educating consumers about your product makes good business sense because consumers are more
> likely to purchase your product, and employees and shareholders are more likely to invest in
> the company. The cost of education is part of the price for doing business, and in fact
> increases business.

This is not a general rule when it comes to luxury goods. A company may try to carve out a niche in the home audio sector along these lines but since the switch to audiophile in the 70s the track record of such attempts has not been good. If you are in the business of manufacturing low-tech products and trying to sell them on technical performance for a price then the manufacturers in the developed world are going to be at a disadvantage. Perhaps less today than in the 70s.

> It has nothing to do with the state - which you seem to be obsessed about. Many companies
> belong to audio trade associations (AES, CEA, ALMA) which also help pay for the cost of
> educating consumers about new CE technologies like Blu-ray,etc,etc..

It makes business sense for companies to pool resources when introducing new technologies. It would only make business sense for loudspeaker manufacturers to pool resources and educate consumers if they all gained from this. Would Bose gain from educating consumers? Of course not, educating consumers would put manufacturing costs up by raising the required level of performance, make existing marketing less effective and move the product in the direction of a commodity.

> If consumers just bought what they want -- there would be no media advertising, marketing
> budgets, audio magazines, celebrity endorsements, or store sales people on commission to
> influence and manipulate the consumer's purchase decision.

This makes no sense. The things you list are intended to influence what consumers want.

> This seems to be completely lost on you, as you seem to think there is more profit in
> deception, ignorance and lies --

Deception, ignorance and lies depends on how you look at the world. Because of our education and signing up for a rational view of how the world works, you and I may recognise deception, ignorance and lies in audiophile marketing and products but audiophile believers do not. Is our view worth more to them than their own? If so, why do they not accept it when it is presented to them?

Concerning the profits, after the stereo boom, the home audio companies in the developed world had the choice of changing or going out of business. There was no competing with the products from the developing world on a basis of technical performance for the price. I think the companies that adopted a marketing lead rather than a technical lead approach were wise although, a bit like you now, I certainly did not think that at the time.