Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How do you listen to an ABX test? (Read 344512 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How do you listen to an ABX test?

Reply #1525 –
Quote from: jkeny on
Quote from: Porcus on
[...]

Quote from: Porcus on
Quote from: jkeny on
It has no way of determining if any null result is valid.


I still do not get what you think is a "valid" conclusion that the experiment could not reject the null.


[...]
What I was trying to get jkeny into elaborating on, is whether he actually agrees that it is "valid" to keep the null until data tells otherwise.

Thus far jkeny has dodged even the most basic concepts, so I doubt I will ever get to what he means by "control" for
  • .

What Jakob is telling you is that a null (or any result) can only be the result of a "valid" test - it can't be the outcome of a test where you sit 100 monkeys in front of a keyboard who are trained to continuously hit the A button. If you don't understand this, you fail to understand the basics of what a "test" means


I know my basic stats, thank you, but as of now it suffices for me to note that again you dodge the question.

How do you listen to an ABX test?

Reply #1526 –
Quote from: xnor on
@mods: I'm seriously wondering why this stupidity has been tolerated for so long in this thread. You locked the other one down pretty quickly after jkeny started ... being himself.


Now, now. No kids of your own? Don't get angry with John, he honestly believes he can "hear" Santa and Organic DACs and Leprechauns and power supplies and whatnot. He trusts his other infantile friends, who told him that they heard Leprechauns under their beds, he "listened" very careful for the instant glaring....excuse me, long term sounds they make...and "verified" that they were real. Like any 4-5yr old intellect, they cannot discern between perceptions and physical reality, between preference and sound. Their perceptions are unerring representations of physical reality.
Adult blind/controlled sound tests are "stupidity".
Which leads to incredibly amusing and entertaining threads like these. 
Closing it would be touted as a victory for the D-K gang (in their view). John went to a sane rational adult site and took it to them, forcing them to silence him. Or some other similar D-K "rationalization". 

cheers,

AJ
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1527 –
Quote from: Jakob1863 on
Quote from: Porcus on
<snip>

That was not really an answer to the question (except what you write in parentheses highlight the issue with sighted tests - they do not test the sound).


Sorry, at that point i did not understand your question. What you expressed in parentheses is misleading, because you simply don´t know if a "sighted test" tests the sound. You might assume that it did not, but you don´t really know.


A sighted test for differences tests not sound but at best sound and vision, as vision gives the identification that one asks for. Testing for sound and vision is not testing for sound.


Quote from: Jakob1863 on
"Keeping the null" is normally not a valid consideration, because, as said before, the "null " will not really be tested.


Uh ... so when one rejects the null, one does so without having ever tested it? The standard procedure is to test whether the data are compatible with the null.



Quote from: Jakob1863 on
Quote
Thus far jkeny has dodged even the most basic concepts, so I doubt I will ever get to what he means by "control" for
  • .


Maybe it would be much more productive to refrain from all this personal stuff to get the interesting points settled.


Yeah, I scratched the surface just to see if there is any fruitful content in jkeny's personal pet issues, and from his answers I agree that they pollute the discussion beyond what is productive to carry on with at the moment.

How do you listen to an ABX test?

Reply #1528 –
Quote from: Jakob1863 on
Maybe it would be much more productive to refrain from all this personal stuff to get the interesting points settled.

Sure Jakob2, I'll help you.

Q: How do you listen to an ABX test?

A: (jkeny) I don't. I'm a chronic peek-o-phile.

Interesting point? Hardly.
But perhaps to you Jakob1863?
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1529 –
Quote from: Porcus on
A sighted test for differences tests not sound but at best sound and vision, as vision gives the identification that one asks for. Testing for sound and vision is not testing for sound.


Let's put it this way: sighted listening includes some sort of 'placebo effect', so is not an evaluation of audible differences. It's not even a test. It's self-disqualifying as such, but of course jkeny denies that because of the sophisticated QA mechanism he uses: his opinion.


So we have to do blind testing.
We have several methods that we know work, but as with any other test care must be taken that the conditions fit your test goal.

In a personal test, I suggested to check your system and hearing with low bitrate mp3s first.
In more formal tests we have participant selection, training ...

So what we are left with is the implications of jkeny's earlier statements: the audiophiles are able but not willing - they first hear differences but then deliberately fail ('cheat' in jkeny's terms) - or they lie about hearing differences in the first place.

Or the more reasonable explanation: the participants didn't hear audible differences. The differences they perceived during sighted evaluation are the result of bias/placebo effects.


Which is why it is absolutely ludicrous to accept claims of audible differences that are the result of sighted listening, especially if you filter them by your opinion (like jkeny does).
The rational position to take is not to accept the claim, until those making the claims can provide solid evidence.
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1530 –
Quote from: Porcus on
<snip>
A sighted test for differences tests not sound but at best sound and vision, as vision gives the identification that one asks for. Testing for sound and vision is not testing for sound.


Vision might give the identification, but if it has or if a listener did hear a difference you can´t decide. Vision is just one bias effect and as you can´t seperate your brain from your "listening device" you still have to deal with a lot of bias effects after excluding "vision" (means after blinding) .
So, unless a test is perfect it surely will never test "sound alone" .
That is one of the reasons why ITU-BS.1116-2 included so much about preparation of the participants and the usage of controls.


Quote
Uh ... so when one rejects the null, one does so without having ever tested it? The standard procedure is to test whether the data are compatible with the null.


It is standard to consider/analyze the data under the assumption that the null hypothesis is true. As Cohen writes it:
Quote
What we want to know is “Given these data, what is the probability that H0 is true?” But as most of us know, what it tells us is “Given that H0 is true, what is the probability of these (or more extreme) data?” These are not the same


which means that Prob(Ho/Data) is not the same as Prob(Data/Ho)  (Ho means null hypothesis)

At that point the other considerations are necessary, see for example the mentioned book of Meilgaard on the topic of sample sizes needed (which is directly related to the argument of Leventhal and of course Cohen) .

How do you listen to an ABX test?

Reply #1531 –
Quote from: Porcus on
Quote from: Jakob1863 on
"Keeping the null" is normally not a valid consideration, because, as said before, the "null " will not really be tested.


Uh ... so when one rejects the null, one does so without having ever tested it? The standard procedure is to test whether the data are compatible with the null.


This is highly curious indeed.
Not only do we stay with the null, we actually presume it to be true until we have sufficient evidence to the contrary.
During the test we also check, assuming the null to be true, how probable the test results are. Only if the results are improbable given the null hypothesis, we reject it.

So it seems to be complete nonsense to say that this is "not a valid consideration".
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1532 –
Quote from: Jakob1863 on
Vision might give the identification, but if it has or if a listener did hear a difference you can´t decide.

Thanks for restating the obvious captain, as if saying something different. Yes, sighted "listening" is not a "test". Of hearing or sound.

Quote from: Jakob1863 on
Vision is just one bias effect and as you can´t seperate your brain from your "listening device" you still have to deal with a lot of bias effects after excluding "vision" (means after blinding) .

Thanks again Capt, blinding is not for "separating" the brain, but simply removing the visual biases of the testee from the DUT.

Quote from: Jakob1863 on
So, unless a test is perfect it surely will never test "sound alone" .

"Perfection" is a strawman figment of the believer imagiination. No claims of "perfect" anywhere here.

Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1533 –
Quote from: ajinfla on
<snip>
Thanks for restating the obvious captain, as if saying something different. Yes, sighted "listening" is not a "test". Of hearing or sound.


Sorry for the repetition, but you simply don´t know if it is a test of "hearing or sound" or not as you can´t decide it.
You might assume (or strongly believe) that it is not, but you can´t know....

As a matter of fact, "sighted listening" means listening under the potential influence of "vision effects" (and a lot of other bias effects too)
"unsighted listening" or "blind listening" means listening under the potential influence of a lot of other bias effects but not the "vision effect bias".

Quote
Thanks again Capt, blinding is not for "separating" the brain, but simply removing the visual biases of the testee from the DUT.


Or, iow, its simply removing _one_ bias effect.

Quote
Quote from: Jakob1863 on
So, unless a test is perfect it surely will never test "sound alone" .

"Perfection" is a strawman figment of the believer imagiination. No claims of "perfect" anywhere here.

Exactly as you can´t claim perfection you have to conclude that every listening test is not a test of "sound alone" but instead a test of "sound and name_the_potential_bias_effect" .

That´s why using appropriate controls and carefull design is so important.
"Block out what you can and randomize what you can´t block out"

How do you listen to an ABX test?

Reply #1534 –
This is complete nonsense and has already been pointed out before as well. Do a search before ...

Knowledge about which DUT you are listening to has little to do with vision, and it will pull in a plethora of biases based on price, design, looks, specs, brand name .... group pressure, expectations, prejudices etc. and will also cause a placebo-like effect. A sighted evaluation therefore is completely and utterly useless to come up with evidence for audible differences.

It's the same reason we don't do sighted tests with placebo sleeping pills or other medicine - because easily a vast majority of people can respond that the placebo works. We observe a much stronger effect of this in groups, for example like in audiophile circlejerks.


Given that, even a single ABX test with high "preponderance of false negatives" () is several magnitudes to infinitely more useful than any of the claims audiophiles come up with during sighted listening - regardless if jkeny's opinion matches it or not.
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1535 –
Quote from: Jakob1863 on
That´s why using appropriate controls and carefull design is so important.
"Block out what you can and randomize what you can´t block out"


And again, for the umpteenth time, in audiophile culture and mass media reporting we aren't dealing with naive listeners (often: college students, the favorite 'lab mice' of university psychology departments)  taking part in an academic trial, who don't know if they'll hear a difference or not. We aren't exploring the fringes of perception, we aren't doing proof of concept.

We are dealing with people,. like JKeny, who already claim , often publicly and with complete assurance, to hear *substantial* or even *night and day* differences between DUTs and formats that by objective criteria should not display such differences.

So you and jkeny, who, I gather are both from the commercial side of hi-end audio , should just stop this newly fervent, highly convenient,  and wholly farcical advocacy for ultimate experimental rigor.  False negatives are not the problem (and not demonstrably 'preponderant') as  you assert they are, in this culture and context.  The use of *sighted evaluation* is.  You guys wouldn't give a damn about Type II errors except that DBT results so often undercut the foundations of high-end belief systems that support your livelihoods.
 

How do you listen to an ABX test?

Reply #1536 –
Quote from: xnor on
Knowledge about which DUT you are listening to has little to do with vision
Talking about vision is just misleading. Someone who is medically 100% blind, but is told which device they are listening to, is taking a "sighted" listening test. Someone with 20/20 vision and their eyes open, but who does not know what they are listening to, is taking a "blind" listening test.

I wouldn't bother to point it out, but some of the protagonists in this discussion might not even understand this.


You and krabapple are spot-on as always, but I wonder why you still bother. If the people you're talking to want to learn anything, it's how to avoid the truth. I doubt impartial people seeking enlightenment read 62-page threads. So while it's nice to be right, I think you're well past wasting your time.

Some of the information might sit nicely in the wiki. No, I'm not volunteering.

Cheers,
David.

How do you listen to an ABX test?

Reply #1537 –
Quote from: 2Bdecided on
You and krabapple are spot-on as always, but I wonder why you still bother. If the people you're talking to want to learn anything, it's how to avoid the truth. I doubt impartial people seeking enlightenment read 62-page threads. So while it's nice to be right, I think you're well past wasting your time.

Some of the information might sit nicely in the wiki. No, I'm not volunteering.

Cheers,
David.
Does this thread really bring anything new to the table? I'm truly wondering... The original question and its answers were sensical, but it drifted off to pointless debates not shortly afterwards.
It's only audiophile if it's inconvenient.

How do you listen to an ABX test?

Reply #1538 –
Quote from: jkeny on
As I say, keep posting - it is very revealing!

The only thing this thread is revealing is your own ignorance. Your questions and errors in logic have been answered time and time again, but you simply. refuse. to listen.

Didn't you say yourself that you don't believe anyone's mind can be changed by a discussion on the Internet? If that's true, why are you even here? You clearly already know that the "facts" held true here are different from the "fact" you hold true.

P.S.: That's not how "facts" work.

Quote from: jkeny on
So, Arny, let's examine your logic - you want false positive & false negative statistics for something you don't consider a test - sighted evaluations.

You consider these statistics important for a test but don't have false negative statistics for ABX tests.

Clearly you consider sighted testing to be an important, valid methodology for determining auditory differences. Show your work. Tell us how often sighted evaluation produces false positives and false negatives. That's what he's asking for.

Since you still don't seem to get it: One of the primary reasons this community prefers and mandates blind ABX tests is because they remove the factor of bias that comes with knowing whether the tested sample belongs to a given encoding/DAC/source/etc. The placebo effect is practically a subset of expectation bias.

In an ABX test, you're taking two known quantities (or unknown quantities if the test is double-blind) that may or may not have a difference and comparing them to a third unknown quantity selected from the previous two for the purpose of demonstrating whether or not the tester can identify which of those two quantities it is. That's all it is, boiled down to simplicity.

For this purpose, false negatives are merely less relevant than false positives. False negatives imply that the tester is incapable of detecting a difference even if there is one. False positives imply that the tester detects a difference exists where there is none. One of those two "false" results creates something that doesn't actually exist, while the other does not.

You really should be able to understand why that one is more of a concern than the other by now. We're not denying that false negatives can occur nor are we denying that they matter. You, however, would burn down the forest because one tree is rotten.

Quote from: Kohlrabi on
Does this thread really bring anything new to the table? I'm truly wondering... The original question and its answers were sensical, but it drifted off to pointless debates not shortly afterwards.

It really doesn't. It was already driven off the rails by SoundAndMachine before jkeny joined the fray.

How do you listen to an ABX test?

Reply #1539 –
Quote from: Jakob1863 on
Sorry for the repetition, but you simply don´t know if it is a test

Wrong. There is no "test" with sighted "listening". None. Not a test.

Quote from: Jakob1863 on
"blind listening" means listening under the potential influence of a lot of other bias effects but not the "vision effect bias".

Your claim believer. Please list each and it's effect on outcome.
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1540 –
Quote from: krabapple on
So you and jkeny, who, I gather are both from the commercial side of hi-end audio , should just stop this newly fervent, highly convenient,  and wholly farcical advocacy for ultimate experimental rigor.  False negatives are not the problem


Ah...but when, not if, John and Jakob fail to hear Santa, Leprechauns or Organic DACs if undergoing Delusion Blocked Therapy, they now have a couple excuses for negatives.
1) "Negative Preponderance" theory-fact
2) Deliberate "Cheating-Negatives"
3) Deafness inducing "Stress" of DBT
4) Etc, etc
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1541 –
Quote from: Jakob1863 on
...you simply don´t know if it is a test of "hearing or sound" or not as you can´t decide it.
You might assume (or strongly believe) that it is not, but you can´t know...


It is becoming clear that the study of formal rhetoric and reason may not be one of your strengths, Jakob.

What you have just presented is a well known form of "Argument From Authority", which is well known to lead to fallacious conclusions:

The Fallacy Of Argument From Authority

Not only that, but I've explained on this thread several times why audiophile sighted evaluations can't be tests, and quite clearly these explanations were  beyond your understanding.

Since you can't comprehend the explanations that I provided Jakob, repeating them again would be gigantic waste of my time.

I personally think that you really don't have the intellectual tools to follow this discussion. I doubt that like JKeny you understand DBTs well enough to do them, even with a lot of assistance. It looks for all the world that you are shilling for JKeny.

How do you listen to an ABX test?

Reply #1542 –
Quote from: Jakob1863 on
which means that Prob(Ho/Data) is not the same as Prob(Data/Ho)  (Ho means null hypothesis)


I know that very well, and I know (and to a wide extent support) the critique against p-value-based inference, but (I) the procedure does in any case set out to test whether the null is compatible with the data, although the "with" is not commutative, and (II) it is completely irrelevant to what appears to be the main issue, which is how quacks attempt to elevate any claim taken out of the air, to the status of the null.

Relativizing the null is the chief strategy of anti-science these days; sometimes as plainly open as claiming that evolution is "theory" and creationism is "theory", and sometimes obfuscated, like jkeny's unspecified "valid". The reason is of course the bleeding obvious one; asking for evidence ruins the business for quacks and miracle men.

How do you listen to an ABX test?

Reply #1543 –
Quote from: ajinfla on
Quote from: krabapple on
So you and jkeny, who, I gather are both from the commercial side of hi-end audio , should just stop this newly fervent, highly convenient,  and wholly farcical advocacy for ultimate experimental rigor.  False negatives are not the problem


Ah...but when, not if, John and Jakob fail to hear Santa, Leprechauns or Organic DACs if undergoing Delusion Blocked Therapy, they now have a couple excuses for negatives.
1) "Negative Preponderance" theory-fact
2) Deliberate "Cheating-Negatives"
3) Deafness inducing "Stress" of DBT
4) Etc, etc


Why are you being so mean? Chronic peeking, peek-a-boo stress? Failing-to-hear-something-hence-peeking?

How do you listen to an ABX test?

Reply #1544 –
AJ,

You forgot the blind experiment that *could* be used to prove that peeking can increase your hearing acuity:
http://www.hydrogenaud.io/forums/index.php...st&p=897286

How do you listen to an ABX test?

Reply #1545 –
Quote from: Rich B on
Why are you being so mean?


It is just the truth. For example John has posted word-for-word the same lie about DBTs 74, count them, yes,  74 times. How many times was the truth explained to him politely during that same period of time?  Isn't intentionally distorting what other people say, lying about the relevant facts for personal gain, and ignoring sincere attempts at explaining how things work a mean thing to do?

Quote
Chronic peeking, peek-a-boo stress? Failing-to-hear-something-hence-peeking?


2 explanations and a question:

Chronic peeking = Sighted Evaluations

Peek-a-boo stress = Stress due to having to an actual test based on just listening, instead of faking it in a Sighted Evaluation.

Failing-to-hear-something-hence-peeking?  = please explain.

How do you listen to an ABX test?

Reply #1546 –
Quote
Chronic peeking, peek-a-boo stress? Failing-to-hear-something-hence-peeking?

2 explanations and a question:

Chronic peeking = Sighted Evaluations

Peek-a-boo stress = Stress due to having to an actual test based on just listening, instead of faking it in a Sighted Evaluation.p[

Failing-to-hear-something-hence-peeking?  = please explain.


No, it's just a variation of 'peek-a-boo'.

How do you listen to an ABX test?

Reply #1547 –
Quote from: Rich B on
Why are you being so mean?

Would you rather I be well above or below average?

Quote from: Rich B on
Chronic peeking, peek-a-boo stress? Failing-to-hear-something-hence-peeking?

Yep, you got it.
Zero trust in >>Ears<<, afraid of "Just listening", etc, etc., despite all the braying.
Shameful, isn't it Rich? You would think at some point, kids grow up to stop believing that if they and/or their trusted friends "hear" Santa or a Leprechaun by "Just listening", that makes them both physically real.
But alas....
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1548 –
Quote from: Kohlrabi on
Quote from: 2Bdecided on
Some of the information might sit nicely in the wiki. No, I'm not volunteering. ;
Does this thread really bring anything new to the table? I'm truly wondering...
The false negative and false positive stats aren't linked from the FAQ (there's a broken link to a binomial distribution table), or fully explained there.

OT: Several issues discussed during the re-write of fb2k's ABX tool aren't mentioned in the FAQ too.

The HA wiki article is really short, and the Wikipedia ABX article has been messed up (as discussed earlier in this thread).

Cheers,
David.

How do you listen to an ABX test?

Reply #1549 –
Quote from: Kohlrabi on
Quote from: 2Bdecided on
You and krabapple are spot-on as always, but I wonder why you still bother. If the people you're talking to want to learn anything, it's how to avoid the truth. I doubt impartial people seeking enlightenment read 62-page threads. So while it's nice to be right, I think you're well past wasting your time.

Some of the information might sit nicely in the wiki. No, I'm not volunteering.

Cheers,
David.
Does this thread really bring anything new to the table? I'm truly wondering... The original question and its answers were sensical, but it drifted off to pointless debates not shortly afterwards.



My take away from this thread is the lying, ignorance of the facts, lack of actual curiosity, intellectual dishonesty, faulty logic and unflagging devotion to the rotten consequences of those things to their lives that is often found with Placebophiles. I have to admit that my tolerance for those ugly things has been enhanced by years of experience dealing with Amir over on AVS.