Here is an audio magazine that supports the use of blind listening tests  when reviewing audio products.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listening Tests

Cheers,
Sean Olive
Director of Acoustic Research, R&D Group
Harman International

Thank you for posting the articles. I especially liked that he addressed non-technical issues about the setup of blind tests. Something which "objectively minded" people, like those on ha.org rarely discuss in more detail. This guy is just straight forwardly aware, honest and investigative, instead of repeating a specific dogma. Again, thanks for the links.

Well, he may get quite far, but when it comes to speakers, i think that there is one significant limitation which he cannot overcome: A first impression of a speaker isn't the same, as a 40th impression of a speaker. What i mean with this is that if one spends some time with getting familiar with a speaker, the impression may be different than the first one. Unfortunatelly, what this means for designing a listening test, is that the setup has to be cheap and easy to build. One single room in one location may not be enough to get more than a first impression of a statistically significant amount of people (unless you have all the testers living nearby and lots of time).

I'm not so sure about listeners'  sound quality judgments on loudspeakers changing over time, if the listeners well-trained, and  the test is well-controlled and sensitive to the effects being measured (timbre, spatial, dynamic/distortion attributes).

I've repeated controlled loudspeaker tests over several months with trained listeners and found their ratings to be pretty stable (~ 0.25 a preference rating on an 11-point scale). I often hear this "extended listening argument" used to dismiss the ecological validity of the tests I conduct, yet I've seen no substantiated proof of this yet. Other people claim the speakers need to be "broken-in" before they can be evaluated, and we know this to be largely B.S (I've actually tested this, and Floyd Toole talks about it in his book on p. 353). Within reason, people will adapt to a loudspeaker's faults over time (especially when the loudspeaker is heard in isolation), which suggests that prolonged listening tests may in fact produce a less sensitive measure of its sound quality.

Thanks for linking to part two of Doug Schneider's article, Mr. Olive! Two questions regarding Harman's testing room with the speaker shuffler:

1. How fast does the shuffler work? I mean, how long does it take to mute speaker A under test, replace it by speaker B, and turn speaker B on?
I'm asking because when doing blind tests of audio codecs at Fraunhofer, I can switch instantaneously (or with a delay of, say, 0.5 sec at most) between stimuli. As I found out, this is essential to identify very subtle differences between the codecs under test. Of course, instantaneous switching with identical speaker position is impossible, but I assume that for speakers, "the faster the better" also holds.

2. Am I correct in assuming that there are anti-sound-reflective curtains on every side of the room? It's hard to see in the picture.

Chris

Quote from: solive on 2009-06-14 03:23:57

I'm not so sure about listeners'  sound quality judgments on loudspeakers changing over time, if the listeners well-trained, and  the test is well-controlled and sensitive to the effects being measured (timbre, spatial, dynamic/distortion attributes).

I've repeated controlled loudspeaker tests over several months with trained listeners and found their ratings to be pretty stable (~ 0.25 a preference rating on an 11-point scale). I often hear this "extended listening argument" used to dismiss the ecological validity of the tests I conduct, yet I've seen no substantiated proof of this yet. Other people claim the speakers need to be "broken-in" before they can be evaluated, and we know this to be largely B.S (I've actually tested this, and Floyd Toole talks about it in his book on p. 353). Within reason, people will adapt to a loudspeaker's faults over time (especially when the loudspeaker is heard in isolation), which suggests that prolonged listening tests may in fact produce a less sensitive measure of its sound quality.

Quote

Well, IMO the "break in"-effect usually is the mentioned "getting familiar with it"-aspect OR (non-exclusive) pure imagination. For well trained listeners, you may very well be correct that it matters less, but for testers who don't do speaker evaluation regularily, i think it matters (myself included there. I find it hard to judge a speaker quickly, because of anxiety and not knowing yet what to listen for [edited]

Regarding listening tests, i think there is another aspect which often isn't considered. People who argue for blind testing often imitate a specific testing methodology and setup, without checking if for a specific purpose, that methodology actually is efficient or needed at all. Part of this seems to come from a "perfectionist" intention to exclude any possible errors, regardless of if that kind of rigor really is needed for a specific task.

A simple example: It is often mentioned, that magazine editors are influenced by knowing the product which they review. The objectivist answer to that is a full blown double-blind test. I however would ask: Would for that specific scenario it perhaps be sufficient, if the reviewer simply would not know what he is reviewing? Imagine a setup like this: A rigid "cover" is put over each speaker which is to be reviewed. The cover looks exactly the same for every speaker. In other words: The speakers get "anonymized", similiar to a blank product packaging. Besides of that limitation, the reviewer can do whatever he wants. He can move the speakers around, etc. Furthermore, always at least three testers will review a speaker, so that contradictory data can be noticed. Would speaker reviews perhaps look very different if it were done that way? And remember: this is not a typical DBT setup!

Similarily, one could ask if a "fully automated" setup really is needed. If the product package is anonymized, a second person which as well does not know the identity of the speakers, can simply do the switching. And behold: The cost and effort of creating such a setup, is something which any reviewer can afford. Yes, it is not as strictly controlled as is typical for audio hardware DBTs - but i would ask: Is it for plain "reviewing" perhaps "good enough"?

1. The shuffler takes about 3 seconds to move different speakers into position. It would be nice if it was faster, but so far, this hasn't proven to be a limiting factor in our tests since audible differences between the speakers are much larger than CODECS -

2. The listener is surrounded by black curtains intended to hide the identities/locations of the  front, side and surround loudspeakers . The curtains are acoustically transparent - not "anti-sound-reflective".

Cheers
Sean Olive
Audio Musings

Or just put them on some kind of "waggon" and add a mechanic to adjust the height of the platform on which the speaker stands without the need to touch the speaker directly. If the waggon is heavy enough, an adjustment of weight because of speaker-type may not even be needed.