Skip to main content
Topic: The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni (Read 7832 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #1
Thank you for this link! I really hope that GoodSound can collect the resources to perform a blind test for every upcoming review.

I haven't gotten to reading your own blog (the one cited in the above link), but I will.

Chris
If I don't reply to your reply, it means I agree with you.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #2
A wonderful amount of good sense is spoken in that editorial!

Cheers,
David.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #3
Indeed! An excellent editorial.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #4
Very well written... I hope one day he addresses "transparent lossy" encodes like Lame -V2 and up. Think I'll drop him a line to that regard...

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #5
Here is an audio magazine that supports the use of blind listening tests  when reviewing audio products.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listening Tests

Cheers,
Sean Olive
Director of Acoustic Research, R&D Group
Harman International


The editor posted a Part 2 to the article, which I came across yesterday. There is a nice photo of our loudspeaker listening lab with the speaker shuffler.


Cheers
Sean Olive
Audio Musings

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #6
Thank you for posting the articles. I especially liked that he addressed non-technical issues about the setup of blind tests. Something which "objectively minded" people, like those on ha.org rarely discuss in more detail. This guy is just straight forwardly aware, honest and investigative, instead of repeating a specific dogma. Again, thanks for the links.
I am arrogant and I can afford it because I deliver.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #7
Thank you for posting the articles. I especially liked that he addressed non-technical issues about the setup of blind tests. Something which "objectively minded" people, like those on ha.org rarely discuss in more detail. This guy is just straight forwardly aware, honest and investigative, instead of repeating a specific dogma. Again, thanks for the links.


Agreed. Doug is one of the more professional audio editors/publishers in the audio business. Sound Stage currently has the best loudspeaker measurements (done at NRC) of any audio magazine, period.  If you know how to interpret those measurements, you can put the subjective reviews into their proper context. I wish him good luck in trying to implement more scientific listening tests in the future.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #8
Well, he may get quite far, but when it comes to speakers, i think that there is one significant limitation which he cannot overcome: A first impression of a speaker isn't the same, as a 40th impression of a speaker. What i mean with this is that if one spends some time with getting familiar with a speaker, the impression may be different than the first one. Unfortunatelly, what this means for designing a listening test, is that the setup has to be cheap and easy to build. One single room in one location may not be enough to get more than a first impression of a statistically significant amount of people (unless you have all the testers living nearby and lots of time).
I am arrogant and I can afford it because I deliver.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #9
Well, he may get quite far, but when it comes to speakers, i think that there is one significant limitation which he cannot overcome: A first impression of a speaker isn't the same, as a 40th impression of a speaker. What i mean with this is that if one spends some time with getting familiar with a speaker, the impression may be different than the first one. Unfortunatelly, what this means for designing a listening test, is that the setup has to be cheap and easy to build. One single room in one location may not be enough to get more than a first impression of a statistically significant amount of people (unless you have all the testers living nearby and lots of time).


You raise a valid point about the listeners. Unless the reviewers have identical setups, or the listeners are living nearby, it will be difficult to get statistically reliable results  since the nuisance variables will not be sufficiently controlled (i.e. different listeners will be hearing different sounds at their ears, so their ratings may well be different). This is why we adopted a standardized listening room and setup at Harman International to train listeners and conduct listening tests in different parts of the world under similar acoustical conditions, and pool the results, if needed.

I'm not so sure about listeners'  sound quality judgments on loudspeakers changing over time, if the listeners well-trained, and  the test is well-controlled and sensitive to the effects being measured (timbre, spatial, dynamic/distortion attributes).

I've repeated controlled loudspeaker tests over several months with trained listeners and found their ratings to be pretty stable (~ 0.25 a preference rating on an 11-point scale). I often hear this "extended listening argument" used to dismiss the ecological validity of the tests I conduct, yet I've seen no substantiated proof of this yet. Other people claim the speakers need to be "broken-in" before they can be evaluated, and we know this to be largely B.S (I've actually tested this, and Floyd Toole talks about it in his book on p. 353). Within reason, people will adapt to a loudspeaker's faults over time (especially when the loudspeaker is heard in isolation), which suggests that prolonged listening tests may in fact produce a less sensitive measure of its sound quality.

Cheers
Sean Olive
Audio Musings

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #10
Thanks for linking to part two of Doug Schneider's article, Mr. Olive! Two questions regarding Harman's testing room with the speaker shuffler:

1. How fast does the shuffler work? I mean, how long does it take to mute speaker A under test, replace it by speaker B, and turn speaker B on?
I'm asking because when doing blind tests of audio codecs at Fraunhofer, I can switch instantaneously (or with a delay of, say, 0.5 sec at most) between stimuli. As I found out, this is essential to identify very subtle differences between the codecs under test. Of course, instantaneous switching with identical speaker position is impossible, but I assume that for speakers, "the faster the better" also holds.

2. Am I correct in assuming that there are anti-sound-reflective curtains on every side of the room? It's hard to see in the picture.

Chris
If I don't reply to your reply, it means I agree with you.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #11
I'm not so sure about listeners'  sound quality judgments on loudspeakers changing over time, if the listeners well-trained, and  the test is well-controlled and sensitive to the effects being measured (timbre, spatial, dynamic/distortion attributes).

I've repeated controlled loudspeaker tests over several months with trained listeners and found their ratings to be pretty stable (~ 0.25 a preference rating on an 11-point scale). I often hear this "extended listening argument" used to dismiss the ecological validity of the tests I conduct, yet I've seen no substantiated proof of this yet. Other people claim the speakers need to be "broken-in" before they can be evaluated, and we know this to be largely B.S (I've actually tested this, and Floyd Toole talks about it in his book on p. 353). Within reason, people will adapt to a loudspeaker's faults over time (especially when the loudspeaker is heard in isolation), which suggests that prolonged listening tests may in fact produce a less sensitive measure of its sound quality.


Well, IMO the "break in"-effect usually is the mentioned "getting familiar with it"-aspect OR (non-exclusive) pure imagination. For well trained listeners, you may very well be correct that it matters less, but for testers who don't do speaker evaluation regularily, i think it matters (myself included there. I find it hard to judge a speaker quickly, because of anxiety and not knowing yet what to listen for. My current speaker set (studio monitors) for example have tweeters, which i at first liked, but later considered too "aggressive" in the 8-12KHz range (hihats, etc). Then again, i already noticed in the past, that i seem to be oversensitive at that freq range).

Regarding listening tests, i think there is another aspect which often isn't considered. People who argue for blind testing often imitate a specific testing methodology and setup, without checking if for a specific purpose, that methodology actually is efficient or needed at all. Part of this seems to come from a "perfectionist" intention to exclude any possible errors, regardless of if that kind of rigor really is needed for a specific task.

A simple example: It is often mentioned, that magazine editors are influenced by knowing the product which they review. The objectivist answer to that is a full blown double-blind test. I however would ask: Would for that specific scenario it perhaps be sufficient, if the reviewer simply would not know what he is reviewing? Imagine a setup like this: A rigid "cover" is put over each speaker which is to be reviewed. The cover looks exactly the same for every speaker. In other words: The speakers get "anonymized", similiar to a blank product packaging. Besides of that limitation, the reviewer can do whatever he wants. He can move the speakers around, etc. Furthermore, always at least three testers will review a speaker, so that contradictory data can be noticed. Would speaker reviews perhaps look very different if it were done that way? And remember: this is not a typical DBT setup!

Similarily, one could ask if a "fully automated" setup really is needed. If the product package is anonymized, a second person which as well does not know the identity of the speakers, can simply do the switching. And behold: The cost and effort of creating such a setup, is something which any reviewer can afford. Yes, it is not as strictly controlled as is typical for audio hardware DBTs - but i would ask: Is it for plain "reviewing" perhaps "good enough"?
I am arrogant and I can afford it because I deliver.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #12
Thanks for linking to part two of Doug Schneider's article, Mr. Olive! Two questions regarding Harman's testing room with the speaker shuffler:

1. How fast does the shuffler work? I mean, how long does it take to mute speaker A under test, replace it by speaker B, and turn speaker B on?
I'm asking because when doing blind tests of audio codecs at Fraunhofer, I can switch instantaneously (or with a delay of, say, 0.5 sec at most) between stimuli. As I found out, this is essential to identify very subtle differences between the codecs under test. Of course, instantaneous switching with identical speaker position is impossible, but I assume that for speakers, "the faster the better" also holds.

2. Am I correct in assuming that there are anti-sound-reflective curtains on every side of the room? It's hard to see in the picture.

Chris


Hi Chris:

1. The shuffler takes about 3 seconds to move different speakers into position. It would be nice if it was faster, but so far, this hasn't proven to be a limiting factor in our tests since audible differences between the speakers are much larger than CODECS -

2. The listener is surrounded by black curtains intended to hide the identities/locations of the  front, side and surround loudspeakers . The curtains are acoustically transparent - not "anti-sound-reflective".

Cheers
Sean Olive
Audio Musings

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #13
I'm not so sure about listeners'  sound quality judgments on loudspeakers changing over time, if the listeners well-trained, and  the test is well-controlled and sensitive to the effects being measured (timbre, spatial, dynamic/distortion attributes).

I've repeated controlled loudspeaker tests over several months with trained listeners and found their ratings to be pretty stable (~ 0.25 a preference rating on an 11-point scale). I often hear this "extended listening argument" used to dismiss the ecological validity of the tests I conduct, yet I've seen no substantiated proof of this yet. Other people claim the speakers need to be "broken-in" before they can be evaluated, and we know this to be largely B.S (I've actually tested this, and Floyd Toole talks about it in his book on p. 353). Within reason, people will adapt to a loudspeaker's faults over time (especially when the loudspeaker is heard in isolation), which suggests that prolonged listening tests may in fact produce a less sensitive measure of its sound quality.


Quote
Well, IMO the "break in"-effect usually is the mentioned "getting familiar with it"-aspect OR (non-exclusive) pure imagination. For well trained listeners, you may very well be correct that it matters less, but for testers who don't do speaker evaluation regularily, i think it matters (myself included there. I find it hard to judge a speaker quickly, because of anxiety and not knowing yet what to listen for [edited]


Regarding listening tests, i think there is another aspect which often isn't considered. People who argue for blind testing often imitate a specific testing methodology and setup, without checking if for a specific purpose, that methodology actually is efficient or needed at all. Part of this seems to come from a "perfectionist" intention to exclude any possible errors, regardless of if that kind of rigor really is needed for a specific task.

A simple example: It is often mentioned, that magazine editors are influenced by knowing the product which they review. The objectivist answer to that is a full blown double-blind test. I however would ask: Would for that specific scenario it perhaps be sufficient, if the reviewer simply would not know what he is reviewing? Imagine a setup like this: A rigid "cover" is put over each speaker which is to be reviewed. The cover looks exactly the same for every speaker. In other words: The speakers get "anonymized", similiar to a blank product packaging. Besides of that limitation, the reviewer can do whatever he wants. He can move the speakers around, etc. Furthermore, always at least three testers will review a speaker, so that contradictory data can be noticed. Would speaker reviews perhaps look very different if it were done that way? And remember: this is not a typical DBT setup!

Similarily, one could ask if a "fully automated" setup really is needed. If the product package is anonymized, a second person which as well does not know the identity of the speakers, can simply do the switching. And behold: The cost and effort of creating such a setup, is something which any reviewer can afford. Yes, it is not as strictly controlled as is typical for audio hardware DBTs - but i would ask: Is it for plain "reviewing" perhaps "good enough"?


So, I think we agree on the point that with proper listener training - the need for prolonged exposure to the loudspeaker is less of an issue.  None of my listeners after training feel anxiety or pressure when doing tests because they are familiar with the process and feel confident doing the task.

Your second point about hiding the identities of the loudspeaker using some rigid enclosure/cover sounds like an interesting proposition. If the enclosures/covers made all of the speakers under test look identical, and had no acoustic effect on the loudspeaker it might work. Of course, as soon as you lifted the speakers you would know how much they cost since audiophiles associate high-end sound with high weight loudspeakers and amplifiers.

I suppose you could add some bricks or sand to the lighter weight loudspeakers so the speakers all weigh the same. The weight issue would not be a bias  if, as you suggest, there was 2nd person to swap the positions of the loudspeakers. This would have to be done with the listener removed from the room since any grunting or swearing during the positioning of the  high-end speakers would send a biasing cue to the listener that the speakers are high-end and expensive

Cheers
Sean Olive
Audio Musings

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #14
Or just put them on some kind of "waggon" and add a mechanic to adjust the height of the platform on which the speaker stands without the need to touch the speaker directly. If the waggon is heavy enough, an adjustment of weight because of speaker-type may not even be needed.
I am arrogant and I can afford it because I deliver.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #15
1. The shuffler takes about 3 seconds to move different speakers into position. It would be nice if it was faster, but so far, this hasn't proven to be a limiting factor in our tests since audible differences between the speakers are much larger than CODECS -

2. The listener is surrounded by black curtains intended to hide the identities/locations of the  front, side and surround loudspeakers . The curtains are acoustically transparent - not "anti-sound-reflective".

Cheers
Sean Olive
Audio Musings

I see. Thanks!

Chris
If I don't reply to your reply, it means I agree with you.

The Misinformed Misleading the Uninformed -- A Bit About Blind Listeni

Reply #16
Or just put them on some kind of "waggon" and add a mechanic to adjust the height of the platform on which the speaker stands without the need to touch the speaker directly. If the waggon is heavy enough, an adjustment of weight because of speaker-type may not even be needed.


Or the reviewers could invest in a binaural room scanning measurement and playback system that would control listener/loudspeaker position effects and sighted biases.

They could email the BRS scans to all the reviewers on staff who could do the tests over headphones eliminating the issue of different room effects.  I've started a separate post on this topic.


Cheers
Sean Olive
Audio Musings

 
SimplePortal 1.0.0 RC1 © 2008-2020