Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Overcoming the Perception Problem (Read 70679 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Overcoming the Perception Problem

Reply #25
I think perhaps you underestimate the general level of the public's intelligence. Anyone who buys a piece of audio equipment knows that most visitors to their house will point out that this system sounds pretty similar to the last one they were excited about.
I am a DJ and heavily-involved in my local electronic music scene. The amount of disinformation and outright lies that people who are professionally involved in audio reproduction believe is absolutely mind-boggling, even (especially?) without their paychecks revolving around selling hardware.

Overcoming the Perception Problem

Reply #26
Deprived of reference points, acuity suffers. Not badly enough to become deaf, obviously - but badly enough to diminish large variables to small ones, and make small ones vanish entirely - which isn't a bad one-line summary of DBT perception results: particularly with reference to hearing which - being driven by feebler mental horsepower - is more prone to suggestion (and more in need of supporting frameworks) than sight (hence McGurk).
Are you claiming people are more prone to hearing things that aren't there than seeing things that aren't there - in respect of the brain "filling in gaps" to create what turns out to be an incorrect model of the real world? I'm not sure that's true.


Quote
Either this is wrong (as you say), or truly accurate testing of this type will come later, when we can directly, mechanically examine - and analyse - brain response without tampering with the subject's psychological state. Certainly not a 'never possible to know' scenario.
...but they've already hacked cat's heads about to find out what signals went in/out of the auditory nerve long before it was possible to do this in a humane way. It's relevant because they have similar cochlea to us, and it's found that, like us, the losses (what you can't hear) derive from the air to neural transduction process.

So we've got cut up cats, predictions from physiology, and blind tests on humans all delivering the same "what difference is just audible" data - but you don't trust the blind tests?


Yet you'll trust the brain response. That's strange, since no one is doubting that placebo is a real brain response - it's just not a response to what you hear!  And if we measure some brain response when people are not aware of what they're listening to (i.e. some response to A that is absent to B), either this will be associated with a conscious audible difference, or not. If not, who cares. If so, then having reported hearing it when not knowing what they were listening to, they've just passed a blind test!

What does the brain scan add? Think this through. Draw a flowchart of the possibilities if it helps.

Cheers,
David.

Overcoming the Perception Problem

Reply #27
'Knowing what you are listening to' is the troublesome variable.

Exactly. It's very troublesome. So we remove that variable in order to get better data.

Deprived of reference points, acuity suffers

You mean the "reference point" of knowing the brand of amplifier playing? Of knowing the encoder? We remove them, because they muddy the data. The only reference point you need is the first piece of audio that plays in your test.

 

Overcoming the Perception Problem

Reply #28
@item:
Perhaps you could share with us a little about who you are so that we can put your point of view into proper perspective.

What do you mean by 'proper perspective'? Am I looking at an ad hominem warmup or a chat-up line?]

Let's just focus about the first part of my question and not worry about the second part.

Please share with us a little about who you are and who you might represent.

Quote
FWIW, as a professional tester I can tell you that I actually pay closer attention to detail when I am consciously involved in a test, despite DBT skeptics and snake oil salesmen telling me that I can't or don't.

That's exactly the point: test conditions create an environment in which you have to 'pay closer attention' - in reality, listen in an entirely different way, disorientated and deprived of cues.

Disorientated and deprived of cues?  You appear not to realize that double-blind testing can (and often does!) provide for the listener to audition the subjects/samples that are known as they are known.  To elaborate on the part that you quoted, during testing I pay closer attention to details explicitly to listen for differences.  If I am listening casually, I do just that, which is to say relax and enjoy.  I have no doubt that my casual listening is done so with less acuity. To put it another way, when was the last time someone accused you of paying close attention because you forgot something you were told and were expected to remember?

If part of my enjoyment has to do with knowing that my equipment or sample is XYZ then that is my business.  If I think I am actually perceiving something differently as a result I respect the forum and refrain from discussing it unless I can comply with its rules.

Quote
For a psych test, that's inadmissable.
Perhaps, perhaps not; I really don't care either way.

Quote
Again, the purpose of DBT is to remove subjectivity as a factor. It can't legitimately be applied with any degree of precision to a study of subjectivity.
...which is nonsense and another obvious display of ignorance.  MUSHRA is well accepted as a double blind test that provides for subjective grading.

Quote
Negative DBT results in the physiological domain are always open to question, but in this domain they aren't even interesting, and it's an embarrassment to the cause to see such faith placed in them.
According to you.  Personally I am really not interested in counting angels dancing on the head of a pin in order to describe the possibility of a difference when there exists no verified objective test data to confirm it.  This is especially the case when the difference is characterized as "night-and-day."

Back on the idea of perceiving differences as a result of having a-priori knowledge of what is being heard, I don't think anyone with enough understanding is denying its power to foster truly tangible differences in the mind.  We simply aren't interested in reading about them here.  There are plenty of other places where you can indulge yourself.

Overcoming the Perception Problem

Reply #29
The sole, specific point I'm making is that DBT is rarely used in perception testing for obvious reasons outlined above, and attempting to smear its credibility from the physiological domain is intellectually dishonest.


And as I told you before,  double blind protocols are common in pain perception studies, where self-reports of perception are the outputs.  You going to address all that, or punt again? 

Quote
And that the abundance of negative results indicates coarse granularity in the test method as much as it supports any particular paradigm.


It 'indicates' that to you because you believe it should, not because it's necessarily true. And as I said before, this amounts to nothing more than an argument from incredulity, if not ignorance.

As 2bdecided already noted, DBTs can support audio signal level differences down to the physical limits of human hearing. Their *granularity* is quite good, if the protocol is good, your windy assertions notwithstanding. 

And now, Mr Joined-in-August,  I trust you won't mind if I sit back and await your predictable departure (back?) to more woozy audio forums, where you'll claim you were driven out of HA for 'unorthodoxy' or 'thinking outside the box'.

Overcoming the Perception Problem

Reply #30
While I certainly agree that putting humans in a lab-situation might change their behaviour and even their perception (although as Greynol points out, it is often likely to sharpen awareness), I still am not getting why this should ever be an argument against attempting to obtain a controlled environment. Certainly there are cases where it isn't feasible. E.g., if I spend a month rebuilding my living room with to get rid of vibrations, damping standing waves (and getting extra insulation against the cold outside), there is no practically feasible way to ABX the old room and the new one [note*]. But I still don't see why it the best practice of research would be to have the listening tests being carried out after hearing me boasting of how much work it was and how much I paid for it.

@item,
could you suggest a design of experiment where blinding the listener or the test administrator would be outright bad for the experiment?


  • but then, I cannot imagine the moderators throwing TOS#8 at me if I claim that it sounds much better and the only evidence I can post is the plot which indicates the +9dB bass hump is down to +3. TOS#8 has is not unreasonable as to the measures to be taken, there is an explicit “to the best of their ability” reservation.

Overcoming the Perception Problem

Reply #31
Abstract:
A positive DBT result establishes reliably that two outcomes or entities differ.
A negative means - equally - either a) the two objects are identical, or b) that the method doesn't permit resolution of their differences.


No, you are wrong at the second part. You don't set up DBT of lossy audio encoders on supertweeters, with FR form 35 kHz to 80 kHz. That method is not going to give anything useful. It's idiotic. Do you really think that peope can't make valid DBT audio tests? It has been done in the past numerous times.

If you set up test correctly, as expected, there will be only "yes, I can definitely hear the difference between A and B", and "no, I can't hear shit". It doesn't matter if the difference in reality is so subtle you can't hear it (lossless vs high bitrate lossy), because the only thing you are testing is if you can hear that difference, and the results are YES and NO. There is no MAYBE UNDER CERTAIN CIRCUMSTANCES, because you then recreate those circumstances and repeat the test. As many times you want, as long as you need.

The trouble is when you realize that you have all that 24/96 music, with fancy expensive amp and loudspeakers hand made from Siberian wood which grew in Tunguska at the crater (not to forget hand-made speaker drivers), and you can't hear the difference from 128 kbit aac file and that high bitrate lossless file. Then the imagination kicks in.

And you, sir, are the product of that mentality, which is contagious.
Error 404; signature server not available.

Overcoming the Perception Problem

Reply #32
It's like The Princess and the Pea. Audiophiles all want to be Princesses, able to feel the pea no matter how many mattresses are between it and them.

Similarly, no matter how good the audio is, and how small a change you make, that change is always audible.

"I can hear a difference" = "I'm a real princess - I can feel the pea"!



I don't believe even a "Real Princess" could feel that pea - it only happens in fairytales, so I guess that means most hi-fi magazines are just that: fairytales.

Cheers,
David.

Overcoming the Perception Problem

Reply #33
If you set up test correctly, as expected, there will be only "yes, I can definitely hear the difference between A and B", and "no, I can't hear shit". It doesn't matter if the difference in reality is so subtle you can't hear it (lossless vs high bitrate lossy), because the only thing you are testing is if you can hear that difference, and the results are YES and NO.


Ehem ... this is not how statistical tests work.

And even if it were, your description would only be valid if this “you” is what is supposed to be tested, and arguably not even then.

Overcoming the Perception Problem

Reply #34
If ABXing negatively alters one's ability to hear differences, it's only a problem if you're using negative results to prove that there is no difference, which is a fallacy in any case: while a positive ABX result shows with a high degree of probability that there IS an audible difference, a negative result never proves anything.

I don't have a problem, however, using the utter lack of positive results as probable evidence of inaudibility, when the alleged difference is claimed to be obvious ("night and day") during sighted tests. Rather, I'll ask "where's your proof?", and shrug when you fail to come up with one.

The effects of sighted bias on what is heard are at least as damaging as the supposedly negative effects on perception of ABX tests. The big difference (no pun intended) is that while sighted tests prove nothing interesting with any degree of certainty, positive ABX results show a high level of probability of unambiguous audibility. Can you think of a better methodology?


Overcoming the Perception Problem

Reply #36
Not everyone is as much a netizen as most of us. Perhaps he just went camping.

Overcoming the Perception Problem

Reply #37
If ABXing negatively alters one's ability to hear differences, it's only a problem if you're using negative results to prove that there is no difference, which is a fallacy in any case: while a positive ABX result shows with a high degree of probability that there IS an audible difference, a negative result never proves anything.


Basically,  what a 'negative' ABX results means is that the hypothesis 'there is an audible difference' was not supported, with a 'p' chance (typically  1 in 20) that an audible difference nevertheless exists . 

Overcoming the Perception Problem

Reply #38
While the OP's reasoning and claimed inference from his cited studies are certainly flawed, he touches a valid point: Double-blind testing of isolated senses puts a subject into an artificial mode of perception, that is different from our usual perception of the world, which is always a multisensory blend. It should also be out of question, that the human brain makes excessive use of inter sensoric correlation while forming a consolidated mental representation of the outside world.

An ABX test can tell you, what an attentive mind, with artificially blocked non-auditory senses, can differentiate at best through the remaining, isolated auditory channel. I do not question, that this is probably as objective as it can get, when the auditory channel alone is all you want to map. I do question how much can be inferred regarding to the experience of actual listening situations in peoples' homes, where not only other senses (seeing your carefully composed system), but also a history of attached memories, associations and whatnot are constantly part of your perception of the world. I'm not claiming that aiming to be objective in a subjective environment is senseless, but it certainly might also make sense, that well situated, older men write magazines for each other, reporting about their experiences of trying to transform their surplus of dollars back to some sense of meaning. Subjective prose, that isn't castrated by some blinded, pain-in-the ass protocol as ABX, might be an actually more "objective" guide to identify an perfectly matching audio system for a member of any common enough group of individuals.

Long term HA usage might turn your mind into something, that has become unable to extract joy from owning expensive audio gear. A history of personal ABX comparisons can attach enough associations, so that this road is just closed. Not everyone might like that and I have come to a point, where I think that both is fine. I do also have come to the belief, that a man convinced that his gold cables sound better in a sighted test, even when he is unable to verify the same results blindly, is probably not lying to us.

PS & BTW Could it be shown already that results from double-blind listening tests correlate with the results of sighted tests over a large enough pool of listeners and different setups?

Overcoming the Perception Problem

Reply #39
Double-blind testing of isolated senses puts a subject into an artificial mode of perception [...]
An ABX test can tell you, what an attentive mind, with artificially blocked non-auditory senses, can differentiate at best through the remaining, isolated auditory channel.  [...]
I do question how much can be inferred regarding to the experience of actual listening situations in peoples' homes,[...]

A double-blind test does not need to be any different for the subject than a sighted test. The only point is that the subject should not be aware which of the two experiences that are being evaluated is he taking.

For example, if he likes to listen with headphones, do so. If he likes to listen laying in the bed, or in a coach, looking at some big speakers, with high or low illumination, with drinks or not,... There's no limitation on that, as long as the same is used for each experience. To be even less distracted, just take another person to run the ABX program for you (i.e. maintaining it double blind, not substituting the ABX program). If it takes 5 minutes, or five hours (because you want to clearly forget about doing the ABX) is up to the subject.


I do also have come to the belief, that a man convinced that his gold cables sound better in a sighted test, even when he is unable to verify the same results blindly, is not lying to us.

If you ommited the word "sound", I could give it a pass, but since sound implies a specific event that originates outside, and is perceived by the ear, then I cannot agree.  One thing is a feeling, and another a perception.  Your mind state can make you more receptive to perceptions, but if that is the case, then you can pass an ABX. Else, there's no proof you're actually perceiving a difference, while there would be reasons to believe the placebo effect is working.


Overcoming the Perception Problem

Reply #40
I would be careful not to limit the word perceive.  Thoughts, feelings, differences in the mind are real and are perceived as such, even if the experience causing the perception came from somewhere other than the ears.  When the word sound is being used then that clearly implies that the experience is coming from the ears and only the ears.

...or am I going overboard in making the terminology more placebophile-friendly?

Overcoming the Perception Problem

Reply #41
Long term HA usage might turn your mind into something, that has become unable to extract joy from owning expensive audio gear.

On the contrary, I enjoy my audio gear knowing that I didn't get fleeced by myself or by someone else.

I don't know if you can actually break down your definition of "Long term HA usage" into specific activities, but ABX doesn't affect musical experiences in any way. If and when you question the quality of your gear, or want to espouse its superiority, you speak the scientific lingua franca by relating your ABX results. Only after that is established can you go on to sighted tests or spectral analysis to bring up discussions of any flavour as to why there were or not any differences.

Most people don't care for comparisons for a variety of reasons. They don't see the need to make claims, and therefore don't need to perform any tests to verify their claims. But in such cases, even a casual discussion of audio gear among friends would really be a discussion of the variety of placebos that people fall prey to.

If the difference is really night and day, it should empower you that your ABX results would come up with the necessary statistical significance in the minimum number of trials and a very short test duration, just to shut the mouths of your sceptics. Yes? And if you had the financial resources to purchase expensive audio gear, it wouldn't be a pain to get some quality switching equipment to facilitate your testing. I see anything otherwise as scientific laziness.


Overcoming the Perception Problem

Reply #43
A double-blind test does not need to be any different for the subject than a sighted test. The only point is that the subject should not be aware which of the two experiences that are being evaluated is he taking.

Indeed, using the word 'blind' (with its natural connotations of being disconcerting, stressful, etc.), where what is actually meant is lack of knowledge or awareness, is highly misleading, and something which many an audiophile builds an argument based upon.  The scientific community would do well to come up with a better term: nescient testing perhaps—testing from a position of nescience.

Overcoming the Perception Problem

Reply #44
While the OP's reasoning and claimed inference from his cited studies are certainly flawed, he touches a valid point: Double-blind testing of isolated senses puts a subject into an artificial mode of perception, that is different from our usual perception of the world, which is always a multisensory blend.


Except,  DBT doesn't do that.

Overcoming the Perception Problem

Reply #45
Among my friends, we have been blind testing ... hm, certain consumer-grade goods. The scientific standards are of course abysmal, and it is evident to anyone & everyone that as the evening passes, the experiments become increasingly worthless (it could be noted though, that the  test is likely getting increasingly more double-blind). We know that professionals try to mitigate the effect in question by spitting out the [fermented|distilled] [barley water|grape juice], a move which in our case would rather constitute a major bias-inducing effect to those who have not already tasted that particular glass.

Anyway, one of us routinely flunks the blind tests with a major smile and a “look at how much money I just saved”. A high-end salesman's worst nightmare, that guy, but I cannot help thinking sometimes that the testing itself actually makes him go for the null. Interesting to see though, how high his hit rate is when he proclaims that “this is awful, it's gotta be the expensive one”.


(Maybe we should one day actually blind test whether the sound in his living room actually sucks as much as we think it does, or whether it is just the placebo from having seen him gradually move his speakers way into the corners over the years, but we should maybe try to keep that test apart from the one I just described.)

Overcoming the Perception Problem

Reply #46
@Porcus:

Long term HA usage might turn your mind into something, that has become unable to extract joy from owning expensive audio gear.
Oh no, if I had the money and no moral compulsion to do anything better with it, I'd get great joy from shiny audio gear. It's jewellery or decoration or an amazing human creation. I can already get joy from not-that-cheap and not-audibly-perfect audio products by enjoying them for what they are.

Whether I'd ever pay to upgrade 96kHz audio files to 192kHz files is another thing. You can't see the difference, and you can't hear the difference, so...

David.

Overcoming the Perception Problem

Reply #47
Ehem ... this is not how statistical tests work.
And even if it were, your description would only be valid if this “you” is what is supposed to be tested, and arguably not even then.


When doing personal test of codec or parameters, I am testing them for my usage, for myself. And then it's either "i can hear the difference" or " i can't", I don't bother if someone else can hear it. And I don't care, really, if these music files sound different for anyone else - I am using them, not trading them, not uploading them.
And I understand that these results can't be solely used as the general truth - "130 kbit tvbr aac file is enough for everyone", for example, but for my usage, these tests are more than statistically enough.

I am sorry if I am missing the point - but isn't the point of ABX test to see if YOU can hear the difference between two files? It is not for viewing the difference between files - simple filesize or binary comparation can be used for that.
Error 404; signature server not available.

Overcoming the Perception Problem

Reply #48
Ehem ... this is not how statistical tests work.
And even if it were, your description would only be valid if this “you” is what is supposed to be tested, and arguably not even then.


When doing personal test of codec or parameters, I am testing them for my usage, for myself. And then it's either "i can hear the difference" or " i can't"


A (one-sided) statistical test would have two outcomes: Either that the alternative hypothesis has passed and should be accepted, or do not accept the alternative hypothesis (then, keep the null). Up to the threshold of the test (confidence could of course always be an artifact of chance), the latter should be the conclusion if there is not positive evidence for the alternative (in this case, “you can hear the difference”).

That should occur if
(I) You have no better chances than a random draw
(II) Your chances are better than the random draw, but you don't have sufficient data to prove it.

Notice that before testing, you are always in one of those two situations. Even if it will turn out when the samples are played that you get a 100/100, then you are in case II until the fact is established.



I am sorry if I am missing the point - but isn't the point of ABX test to see if YOU can hear the difference between two files?


Not necessarily. Suppose I want to establish a “sufficiently good” (for whatever purpose) end-user format. Then I am not satisfied with your score on your music, unless I am only targetting you as a customer. Even if your music does not have nasty enough artifacts for you to detect (or find annoying), it might be different with other ears and other signals. (Of course, you then need to use the appropriate method (test / design of experiment) to check whether the accuracy is better than random, but that is a practical obstacle.)

If 5 percent of the listeners hear differences on 10 percent of their music collection, is then the format “transparent”? I think not. It may be good enough for the purpose, by all means, but it does not mean that there are no audible differences.

Overcoming the Perception Problem

Reply #49
Except,  DBT doesn't do that.


Why do sighted test regularly lead to different results, then? Just calling it "bias, that should be eliminated" doesn't change the fact.

Imagine the following test setup: A test subject is presented music supposedly sourced from either a Sansa Clip or his favorite Burmester rack. You present an expensive looking switch to him, that's basically a dummy and that only inserts a small pause, but connects to the Clip at all times. Now imagine, you'd get a statistically significant result, that the subject rates the sound quality consistently higher, when he believes it to be coming from his Burmester rack / not coming from the Sansa Clip.

Now do a second test, this time double blind with both sources actually connected. Imagine the subject now fails to identify a difference.

What can we draw from this, especially when the subject was a honest type, sincerely motivated to rate the quality exactly as he perceived it in the first setup, without trying to prove or defying anything?

First, HA habit, the subject should stop claiming, that his Burmester setup sounds better than a Sansa Clip, as proven by the DBT. HA usually stops here.

But maybe one shouldn't. The belief, that sound was coming from a impressively crafted sound system, was able to significantly alter the subjects perception. In addition, the subjects usual mode of listening is reflected much better in the first setup than in the second (DBT).