Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: The Dishonesty of Sighted Listening Tests (Read 28341 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

The Dishonesty of Sighted Listening Tests

Reply #50 –
Quote from: honestguv on
An audiophile reviewer writes audiophile reviews for audiophile publications. Commercial audiophile publications are primarily vehicles for audiophile advertising. If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.

It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.


I think that is exactly the point. Honest audio review does not have a business case. Imagine the 'perfect' objective and honest audio magazine. They would have to tell you in one CD player review after another that both measurements and blind tests aren't significantly different from 99% of the other tested units. Then again and again they could prove the bullshitness of cable voodoo.

The only interesting area would be properly setup blind speaker testing. But what kind of a business case would that be? The better your reviews the sooner you lose your subscribers - happily knowing that they have bought the objectively best gear for their budget and everything else was voodoo. Maybe five years from then your potential customer may buy one ore two issues again for upgrade advice, but that's all!

Now look at the business case for voodoo reviewers! Their subscribers may eagerly await every new monthly issues just to get inspired how they could 'optimize' their setup even more. You know, finding the proper cable for exactly your combination of amp and speaker is almost art!  It can take several tries to find perfection. And that's only the beginning. Jitter must be controlled, crossovers must be modded, and so on. A never ending journey. Just the day when you think that the speaker cable is finally the perfect match for your setup you start thinking about replacing your amp with a tube driven one. Your neighbor's sounds just so much warmer....

The Dishonesty of Sighted Listening Tests

Reply #51 –
Quote from: rpp3po on
Quote from: honestguv on
An audiophile reviewer writes audiophile reviews for audiophile publications. Commercial audiophile publications are primarily vehicles for audiophile advertising. If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.

It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.


I think that is exactly the point. Honest audio review does not have a business case. Imagine the 'perfect' objective and honest audio magazine. They would have to tell you in one CD player review after another that both measurements and blind tests aren't significantly different from 99% of the other tested units. Then again and again they could prove the bullshitness of cable voodoo.

The only interesting area would be properly setup blind speaker testing. But what kind of a business case would that be? The better your reviews the sooner you lose your subscribers - happily knowing that they have bought the objectively best gear for their budget and everything else was voodoo. Maybe five years from then your potential customer may buy one ore two issues again for upgrade advice, but that's all!


In addition to loudspeakers, there's also room acoustics, networked audio and computer/audio interfaces, and the ever-burgeoning feature sets of AVRs (esp. digital 'room correction') and players.

A paper or online journal that focused on this stuff -- stuff that actually makes an audible difference -- and had the financial power to present it in a graphically slick, well-written and well-researched way -- could perhaps find a niche.

But given that formerly 'objectivist' leaning glossy Sound & Vision has gone flabby, while the plucky but dowdy Sensible Sound appears to be on hiatus,  it appears the market for no-nonsense audio journalism not all that hot.  Though I'd guess that in an economic downturn, people would be *pleased* to know what great sound they could get for relatively little money.

The Dishonesty of Sighted Listening Tests

Reply #52 –
[quote name='honestguv' date='Apr 22 2009, 12:34' post='629015']
quote name='solive' date='Apr 12 2009, 03:47' post='627052'

<<This was not the claim. The claim was that if there was a business case it would be done. >>

In the very next sentence the poster also claimed "Gearing up to routinely peform scientifically valid experiments is not a significant issue" and that was the claim which I replied is not true: gearing up to do valid listening tests is a serious investment, which I believe pays for itself over the long term. Whether you believe there is a good business case for this approach  is not relevant to me.  I happen to believe (and so does my company) that a scientific approach towards audio product research, development, testing and validation is the best approach for staying in business over the long-term.  Please remember that over 70% of our business is automotive audio with customers like BMW, Mercedes, Toyota/Lexus and Hyundai/Genesis, who don't tolerate the laissez-faire approach towards audio testing to which you subscribe. If you tell these companies that your audio system sounds better than the competition, they want to see hard scientific valid data to back it up.  We cannot simply "pull some numbers out of a hat" and say, "it sounds really good -- take my word for it."

.<<Given that it is clearly not in the interests of the industry, how much taxpayers money should be spent by the state to provide this information and how much damage would it do in terms of jobs, tax revenue and the like?>>

Where did I suggest that the state provide consumers valid product information??  I suggested that private companies and audio review magazines might provide this information voluntarily-- not the goverment!

<<An active consumer primarily needs access to knowledge about what is important and why. This knowledge is available from non-audiophile sources for those that consider the effort worthwhile. >>

OK. Again, I ask you where exactly are these "non-audiophile sources"??  Is it classified FBI information?? 

<<It is hard to reply without knowing what you mean by opinion.>>

Opinion 1 a: a view, judgment, or appraisal formed in the mind about a particular matter (Websters)  Read the Gordon Holt interview where he states his opinion. Read it, and if you still find it hard to reply don't fret too much. You don't  have to reply to every point I make unless you just like to be argumentative..

<<You appeared to put him forward as someone whose statements should be given weight. It may be interesting that he has changed his view somewhat in his old age and that it has shifted in the "correct" direction but is the basis on which he forms his judgements the rational one based on reproducible evidence? Do some of his other statements reinforce or counter the view that he is rational?>>

I'll let you figure that out. I have more scientific testing to do..

<<I would suggest this is a rather peculiar use of the word true. >>

Again I refer you to the Websters: TRUE (1) being in accordance with the actual state of affairs <true description> (2): conformable to an essential reality. I think my use of the word "True" is perfectly correct in the context of  describing the effect that visual biases have on the perception of sound. If you disagree please take this up with Websters - not me.

<<If removing the visual cues is important for a true assessment then what about the room?>>

Glad you asked. We have acoustically transparent curtains that hide the boundaries of the walls. We also have a binaural room scanning device that completely eliminates all visual cues. You can read about it in my blog

<<If it was in the interests of the audiophile publications to revert to more traditional/rational means of assessing the performance of hardware then the reviewing would follow.>>

Perhaps. Scientific reviews of $1000 per ft wires won't help sell advertising or make interesting reading because there is either nothing much to say -- or something really bad to say. When really bad things are said, the magazine risks being sued by the company.

<<It would be a useful tool for some consumers but I can see no plausible way it is going to get funded.>>

Try Consumer Reports - for the past 2 years, they have  started to do loudspeaker ratings based on comprehensive anechoic measurements -- Other organizations like CEA are considering similar specifications for loudspeakers, although the industry is very reluctant.

<<Is this a good or bad thing for a luxury products industry?>>

Regarding perceptually meaningful product specifications, it depends on your perspective. If you make a really good product, the answer is  "YES". If you make a crappy product then the answer is "NO." If you are a consumer then the answer is "YES" -- unless you are unfortunate to already own the crappy product.


<<The industry works with and to an extent modifies what audiophile consumers want. The less audiophiles are interested in (expensive) technical performance and more interested in (cheaper) perceived perfomance the bigger the profit margins>>

So you are saying there is more money to be made in deceiving audiophiles  because you can charge them a lot more for something that has poor technical performance. And you have no moral qualms about that?
What I don't understand is why you think technical performance needs to be expensive. I've measured and tested many $500 loudspeakers that have technically superior performance (and better sound) than $11,000 loudspeakers. The correlation between price and technical/sound quality performance with loudspeakers is very low.  My explanation for this  is that most companies that sell very expensive speakers don't sell many of them, which means their R&D budgets for proper design, testing and validation are very low. The other problem is that they are able to get away with it.

<<Non-audiophile consumers have a much wider choice of entertainment sources today and I suspect the silliness of the audiophile sector is going to have limited appeal.>>
Agreed. So Gordon Holt was correct.


<<Why is it the industries job to educate consumers? Surely it is the industries job to make profits for their shareholders, pay tax to the state, keep their workforce happy and persuade consumers to purchase their products. Does educating consumers of luxury products help or hinder this?>.

Do you think keeping the consumer ignorant and deceiving  them is a much better approach?  That really turned out well for Enron, AIG,etc,etc  Educating consumers about your product makes good business sense because consumers are more likely to purchase your product, and employees and shareholders are more likely to invest in the company.  The cost of education is part of the price for doing business, and in fact increases business. It has nothing to do with the state - which you  seem to be obsessed about.  Many companies belong to audio trade associations (AES, CEA, ALMA) which also help pay for the cost of educating consumers about new CE technologies like Blu-ray,etc,etc..

<< Consumers buy what they want. They do not buy what knowledgeable people consider best for them but what what they want.>>

You really are quite naive about how consumers make purchase decisions, and the audio business in general.  If consumers just bought what they want -- there would be no media advertising, marketing budgets, audio magazines, celebrity endorsements, or store sales people on commission to influence and manipulate the consumer's purchase decision.  In Japan, the audio consumer is greatly influenced by what experts say (ie the reviewer) because that is the cultural norm. There is a great opportunity to influence the consumer's purchase decision in a positive way by providing them useful information about the performance of the product. This seems to be  completely lost on you, as you seem to think  there is more profit in deception, ignorance and lies --
 

The Dishonesty of Sighted Listening Tests

Reply #53 –
Magazines don't make most of their money from subscribers/newstand purchasers; they make it from advertisers. Which explains practically everything, yes?

The Dishonesty of Sighted Listening Tests

Reply #54 –
Quote from: solive on
<<Why is it the industries job to educate consumers? Surely it is the industries job to make profits for their shareholders, pay tax to the state, keep their workforce happy and persuade consumers to purchase their products. Does educating consumers of luxury products help or hinder this?>.

Do you think keeping the consumer ignorant and deceiving  them is a much better approach?  That really turned out well for Enron, AIG,etc,etc  Educating consumers about your product makes good business sense because consumers are more likely to purchase your product, and employees and shareholders are more likely to invest in the company.  The cost of education is part of the price for doing business, and in fact increases business. It has nothing to do with the state - which you  seem to be obsessed about.  Many companies belong to audio trade associations (AES, CEA, ALMA) which also help pay for the cost of educating consumers about new CE technologies like Blu-ray,etc,etc..


I agree totally. Though differences like this (it appears) may indicate differences in philosophical perspectives that are more basic; indicating a divide on where someone's coming from - industry or science. Two areas not necessarily mutually exclusive, but which have been uncomfortable together in the consumer realm.
Not sure if that comment makes sense, but in any case, solive you make the point way better than I could!

The Dishonesty of Sighted Listening Tests

Reply #55 –
Quote from: solive on
I happen to believe (and so does my company) that a scientific approach towards audio product research, development, testing and validation is the best approach for staying in business over the long-term.  Please remember that over 70% of our business is automotive audio with customers like BMW, Mercedes, Toyota/Lexus and Hyundai/Genesis, who don't tolerate the laissez-faire approach towards audio testing to which you subscribe. If you tell these companies that your audio system sounds better than the competition, they want to see hard scientific valid data to back it up.  We cannot simply "pull some numbers out of a hat" and say, "it sounds really good -- take my word for it.


For the domain of professional publications and consulting I certainly do believe in the business case of objective audio evaluation. Not for consumers, though, because of the described difficulties regarding regular and lasting customer retention.

The Dishonesty of Sighted Listening Tests

Reply #56 –
Quote from: honestguv on
Why is it the industries job to educate consumers?


Educating consumers is a well-known and highly effective means of maximzing shareholder value.

It's all about human nature. People will buy things they don't understand at all, but generally only under duress. An example of such a thing would be medications.

Generally people what to understand something about the how and why of anything they buy. Why do some prefer Coke over Pepsi? It's the one that tastes best to me. Why do some prefer one loudspeaker over another? It's the one that sounds best (for the money) to me.

Why does a consumer think that one speaker sounds better than another? If he knows some reasons why a speaker should sound better, many consumers will perceive that it sounds better when it is demonstrated to them.

Education is a means to get the consumer's mind thinking favorable thoughts about your product, even before he has any personal experience with it. It's a way to keep him thinking favorable thoughts about your product even if it is harder to buy or more expensive, or heaven forbid sounds bad in the show room.

The Dishonesty of Sighted Listening Tests

Reply #57 –
Quote from: MichaelW on
Magazines don't make most of their money from subscribers/newstand purchasers; they make it from advertisers. Which explains practically everything, yes?

Which is why Consumer Reports, which has no advertising revenue, is far more likely to provide unbiased information.

The Dishonesty of Sighted Listening Tests

Reply #58 –
quote name='solive' date='Apr 23 2009, 05:02' post='629100'

Apologies again for the break. You do not seem to be responding in the spirit hoped, perhaps I should be briefer.

> Please remember that over 70% of our business is automotive audio with customers like BMW,
> Mercedes, Toyota/Lexus and Hyundai/Genesis, who don't tolerate the laissez-faire approach
> towards audio testing to which you subscribe.

I subscribe? Curiously, one of the reason for not posting was a visit to Torino (not in your list) although the sound discussed was not from the radio.

> Where did I suggest that the state provide consumers valid product information?? I suggested
> that private companies and audio review magazines might provide this information
> voluntarily-- not the goverment!

Since it is not in the interests of the industry (unless you dispute this?) then it has to be funded by the state. In the 1970s, at least in Europe with which I am familiar, there was a growing move by the state in this direction but then it all went very quiet when the home audio industry started to move from hi-fi to audiophile. All that fuss about stretching specified Watts and then not a peep about magic one-way gold-plated audiophile speaker cables. Why was that?

>> If removing the visual cues is important for a true assessment then what about the room?
>
> Glad you asked. We have acoustically transparent curtains that hide the boundaries of the
> walls. We also have a binaural room scanning device that completely eliminates all visual
> cues. You can read about it in my blog

That was not the question intended. If audiophiles should not include their visual/knowlege cues in their assessment of loudspeakers then should they substitute some other room response when judging the "true" performance of their speakers?

>> Is this a good or bad thing for a luxury products industry?
>
> Regarding perceptually meaningful product specifications, it depends on your perspective. If
> you make a really good product, the answer is "YES". If you make a crappy product then the
> answer is "NO." If you are a consumer then the answer is "YES" -- unless you are unfortunate
> to already own the crappy product.

How much are luxury products about technical performance and how much are they about perceived performance (e.g. status). How a professional in a lab or studio judges the abilities of hardware is substantially different to how the audiophiles on Stereophile's forum judge the abilities of hardware. The latter may well believe a lot of nonsense about how the world around them works but it is a valid part of how they judge.

> So you are saying there is more money to be made in deceiving audiophiles because you can
> charge them a lot more for something that has poor technical performance. And you have no
> moral qualms about that?

Is it morally wrong to give people what they want? The posters on Stereophile's forum want audio equipment to be magical and will work hard to push away and ignore information that goes against this. It is no different with Bose and B&O equipment in the consumer audio sector. Customers are purchasing far more of what goes with the brand than they are sound quality. I have no problem with this.

What I would be uncomforatble with is the consequences of state involvement in imposing full and fair specifications for luxury products like home audio equipment. So long as the state supports education and access to valid information exists then, in my judgement, that is where the line should be drawn.

> What I don't understand is why you think technical performance needs to be expensive.

Good technical performance for loudspeakers requires substantial cabinets, magenetic material, production control, limited demand, and the like which is going to keep costs up. Contrast this with, say, Bose products where more emphasis is placed on the marketing.

> Do you think keeping the consumer ignorant and deceiving them is a much better approach?

Consumers have access to good and bad information and can choose who and what to believe. If better consumer information about luxury products is to be provided then it will have to be funded. I think we may disagree about the funding.

> Educating consumers about your product makes good business sense because consumers are more
> likely to purchase your product, and employees and shareholders are more likely to invest in
> the company. The cost of education is part of the price for doing business, and in fact
> increases business.

This is not a general rule when it comes to luxury goods. A company may try to carve out a niche in the home audio sector along these lines but since the switch to audiophile in the 70s the track record of such attempts has not been good. If you are in the business of manufacturing low-tech products and trying to sell them on technical performance for a price then the manufacturers in the developed world are going to be at a disadvantage. Perhaps less today than in the 70s.

> It has nothing to do with the state - which you seem to be obsessed about. Many companies
> belong to audio trade associations (AES, CEA, ALMA) which also help pay for the cost of
> educating consumers about new CE technologies like Blu-ray,etc,etc..

It makes business sense for companies to pool resources when introducing new technologies. It would only make business sense for loudspeaker manufacturers to pool resources and educate consumers if they all gained from this. Would Bose gain from educating consumers? Of course not, educating consumers would put manufacturing costs up by raising the required level of performance, make existing marketing less effective and move the product in the direction of a commodity.

> If consumers just bought what they want -- there would be no media advertising, marketing
> budgets, audio magazines, celebrity endorsements, or store sales people on commission to
> influence and manipulate the consumer's purchase decision.

This makes no sense. The things you list are intended to influence what consumers want.

> This seems to be completely lost on you, as you seem to think there is more profit in
> deception, ignorance and lies --

Deception, ignorance and lies depends on how you look at the world. Because of our education and signing up for a rational view of how the world works, you and I may recognise deception, ignorance and lies in audiophile marketing and products but audiophile believers do not. Is our view worth more to them than their own? If so, why do they not accept it when it is presented to them?

Concerning the profits, after the stereo boom, the home audio companies in the developed world had the choice of changing or going out of business. There was no competing with the products from the developing world on a basis of technical performance for the price. I think the companies that adopted a marketing lead rather than a technical lead approach were wise although, a bit like you now, I certainly did not think that at the time.