Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How do you listen to an ABX test? (Read 357554 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How do you listen to an ABX test?

Reply #1275 –
Quote from: jkeny on
Yes but this is being done prior to the test, not within the actual test itself -there are differences & assumptions implicit in this

No, this is part of the test. Logically, it should be temporally prior to your final preference rating, else you might rate the wrong file.

If this sort of check is optional, then you should do more rating trials or rate several different files, because with just one original-lossy file combination you'd have a 50% chance of guessing right. You could then erroneously score a file lower than "imperceptible" (highest score in BS.1116). If that happens too often (similar to failing ABX trials), you'd have to disqualify yourself from this sort of test.


Quote
And as I've explained many times, it's not JUST about the equipment, etc - it's about testing the test itself to ascertain how good it is, as JJ says!

The home-run test would be testing if you with your equipment can hear audible difference e.g. between two files.

If you check with a low enough bitrate mp3 first (that should be audibly different), you can make sure your test - which includes everything mentioned before - works.
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1276 –
Quote from: jkeny on
Quote from: xnor on
Yes. and as we have explained a million times, if you e.g. test for a preference between original and lossy file, then you could run an ABX first to make sure you're scoring the right file. This is what some people actually do.
Yes but this is being done prior to the test, not within the actual test itself -there are differences & assumptions implicit in this


Putting positive controls into listening tests is old news. My old www.pcabx.com web site from 15 years ago was built on  easy and natural positive controls which are also implemented several times in file sets for ABXing that are hosted at this web site.

The suite of listening tests is created with files that vary the artifact being tested for manifest over a range that varies from blindingly obvious to either the artifact in its natural state or the artifact at a level that is strongly suspected to be well below threshold.  Listeners are instructed to start with the blindingly obvious tests and work their way in logical steps down to the point where their testing results in random guessing.

The means used for controlling the size of the artifact does not require a detailed knowledge of what it is.

A set of files for ABXing that implement this kind of positive control can be downloaded from here: Listening tests related to the audibility of nonlinear distortion


How do you listen to an ABX test?

Reply #1277 –
Now we're using a JJ quote to set up a strawman argument.

Meanwhile, controls set up to validate the discussion's honesty still aren't being correctly exercised.

How do you listen to an ABX test?

Reply #1278 –
Quote from: jkeny on
as JJ says!

Oh great, so now you're channeling JJ? Ok, fine.

John Keny: Yes, I admit it. I get better, more "comprehensive results" with longer term peeking at glaring first note differences DUTs. More comprehensive results than any ABX/blind tests.

JJ: Sir, retract this idiocy.
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1279 –
My whole point is & always was simple:
- I'm looking for something within the test (I suggested controls - I believe "catch trials" is the correct term) which verifies that the actual test just run is a valid test. In other words that the results come from a test run which we can evaluate is sensitive enough to be a valid test.

Or as BS.1116 says:
Quote
A major consideration is the inclusion of appropriate control conditions. Typically, control conditions include the
presentation of unimpaired audio materials, introduced in ways that are unpredictable to the subjects. It is the differences
between judgement of these control stimuli and the potentially impaired ones that allows one to conclude that the grades
are actual assessments of the impairments.


Now, I know that pre-testing & pre-screening are moves towards this goal of ensuring the test is sensitive but it falls short of testing the actual test itself - it seems to me that it just assesses that the listener & equipment are capable & sensitive enough. The actual running of the test is still not being evaluated with these measures. As I said before, these measures will not catch if someone isn't listening (for whatever reason) i.e isn't actually engaging in the "TEST". So the use of catch trials is what I suggested

I hear you saying that it's not possible to incorporate catch trials within ABX testing. I accept your statement but I would suggest that it therefore leaves ABX tests as missing an important element - one is considered an important element (or major consideration as stated above) of such perceptual testing.

Just to note - the BS.1116 is strongly recommending the use of such controls in formal testing to validate the test. It would seem to me that there is an even greater need for such controls in the less formal environment of home tests

How do you listen to an ABX test?

Reply #1280 –
You can repeat this as many times as you want, and we can repeat the answers, but what's the point?

So let's narrow your criticism down to a single point:
Quote from: jkeny on
these measures will not catch if someone isn't listening (for whatever reason) i.e isn't actually engaging in the "TEST"

I think I have answered this point near the beginning of this thread.

We know from previous tests that the test setup enables participants to discern differences.
So we are left with the problem of willingness. But I think we can reasonably make the assumption that audiophiles, that are trying to prove their claims right, are listening during the test. Otherwise it would be like preparing well for an exam to pass it, but then deliberately failing it.
Similarly, scientists and science-minded people would be highly curious in discovering new audible differences.


Quote from: jkeny on
Just to note - the BS.1116 is strongly recommending the use of such controls in formal testing to validate the test. It would seem to me that there is an even greater need for such controls in the less formal environment of home tests

Go back and read my previous replies to your BS.1116 quotes.


edit: fixed spelling
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1281 –
Quote from: xnor on
You can repeat this as many times as you want, and we can repeat the answers, but what's the point?

So let's narrow your criticism down to a single point:
Quote from: jkeny on
these measures will not catch if someone isn't listening (for whatever reason) i.e isn't actually engaging in the "TEST"

I think I have answered this point near the beginning of this thread.

We know from previous tests that the test setup allows participants to be able to discern differences.
Do you mean the pre-screening shows this or do you mean that previous tests show it's possible to discern differences? If you are suggesting that other tests can be used as a validation that this test is there fore sensitive enough, I would have to disagree with you.
If you are suggesting that the pre-screening shows that the test is sensitive enough I would agree that it shows the equipment & listeners are sensitive enough but it still doesn't show if under the test conditions from which the results are gathered, that this sensitivity hadn't fallen off.
Quote
So we are left with the problem of willingness. But I think we can reasonably make the assumption that audiophiles, that are trying to prove their claims right, are listening during the test. Otherwise it would be like preparing well for an exam to pass it, but then deliberately failing it.
Similarly, scientists and science-minded people would be highly curious in discovering new audible differences.
Willingness & motivation are necessary but that doesn't mean that the test conditions itself haven't effected the participant's performance. I suggested catch trials as the most direct way of doing this.


Quote
Quote from: jkeny on
Just to note - the BS.1116 is strongly recommending the use of such controls in formal testing to validate the test. It would seem to me that there is an even greater need for such controls in the less formal environment of home tests

Go back and read my previous replies to your BS.1116 quotes.

I will when I get a chance but I don't remember anything you posted which addressed my point here.

How do you listen to an ABX test?

Reply #1282 –
Quote from: jkeny on
My whole point is & always was simple:

1) No offence taken but I & others have done numerous side by side comparisons of stock unit to modified unit - were they DBT, no - there is absolutely no need, the difference in sound is so glaringly obvious & noticeable from the first couple of notes. No offence but I already have given this proof with links to the reviews. Have you read any? Anecdotal, yes & not worth a damn according to those who have never heard the unit - say this to one of the people who have a unit & they will laugh at your stupidity in requiring DBT.

2) Yes, I get better, more comprehensive results with longer term completely uncontrolled peeking/listening


Yes John, we get your simple point. You will never undergo therapy for the condition. You will come up with endless excuses on the spinning wheel of excuse, as to why you will evade at all costs: "Fatigue", "false positives", Positive controls needed, no test of test, no test of test for test of test, etc, etc. Anything but a DBT.

Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1283 –
Quote from: xnor on
You can repeat this as many times as you want, and we can repeat the answers, but what's the point?
None. Give up. I'm sure there's a suitable quote from Proverbs 26 or even Mark Twain that you're overlooking.

Cheers,
David.

How do you listen to an ABX test?

Reply #1284 –
Quote from: ajinfla on
Quote from: jkeny on
My whole point is & always was simple:

1) No offence taken but I & others have done numerous side by side comparisons of stock unit to modified unit - were they DBT, no - there is absolutely no need, the difference in sound is so glaringly obvious & noticeable from the first couple of notes. No offence but I already have given this proof with links to the reviews. Have you read any? Anecdotal, yes & not worth a damn according to those who have never heard the unit - say this to one of the people who have a unit & they will laugh at your stupidity in requiring DBT.

2) Yes, I get better, more comprehensive results with longer term completely uncontrolled peeking/listening


Yes John, we get your simple point. You will never undergo therapy for the condition. You will come up with endless excuses on the spinning wheel of excuse, as to why you will evade at all costs: "Fatigue", "false positives", Positive controls needed, no test of test, no test of test for test of test, etc, etc. Anything but a DBT.

Actually, I don't think you do as you are unwilling to address it - preferring instead to launch into stream of consciousness ranting.

Let me ask you - if you now insist on proctoring as a necessary condition for acceptance of positive results why do you reject the idea of validating the sensitivity of the test  for null result?

In your reply, just once try to stay clear of personal insult.

How do you listen to an ABX test?

Reply #1285 –
Quote from: 2Bdecided on
Quote from: xnor on
You can repeat this as many times as you want, and we can repeat the answers, but what's the point?
None. Give up. I'm sure there's a suitable quote from Proverbs 26 or even Mark Twain that you're overlooking.

Cheers,
David.

The point is that you have not given an answer to the central issue - instead skirting around it with answers that you may think address the issue. None of your answers address the issue that we have no way of judging the validity of the results of a particular ABX test because the sensitivity of that particular run of the test has not determined.

You talk about pre-screening, pre-testing which qualifies the equipment/listeners as suitable but you do nothing to qualify the actual run of the test itself. Your presumption, it seems to me, is that the test run itself will be exactly as sensitive as the pre-testing run. I can see so many ways that this would not be the case. Tell me, for instance, if the participants know that they are doing a pre-testing run as opposed to an actual run.

How do you listen to an ABX test?

Reply #1286 –
No John, we've talked about where 16-trial ABX is good enough, where it isn't, what the alternatives are, and shown that we use those alternatives too.

Meanwhile, you don't appear to have much experience with ABX testing, don't understand level matching, and don't realise when your questions have been answered.

I know when I'm wasting my time.

Goodbye

Cheers,
David.

How do you listen to an ABX test?

Reply #1287 –
Quote from: jkeny on
Actually, I don't think you do

We do John. You are on record for all to see, as believing blind tests to be "stupidity". You can't comprehend the difference between sound and preference.
Despite repeated attempts by the aforementioned jj to explain this to you in this thread.
You still today believe your daydreams as the best method of determining sound, not preference.
So you desperately seek ways of making your daydreams method better than or at least equal blind testing. Blind tests cause fatigue. False negatives are "preponderant". Absolutely need positive controls inserted for glaring first noted differences or test will mask them. 60+ trials needed. BS-1116 needed. Insert excuse here ___ needed. Etc, etc.
You certainly don't believe BS-1116 will give the most "comprehensive results", but you'll use it as an excuse to avoid any therapy....or test of your BS products.

Quote from: jkeny on
Let me ask you - if you now insist on proctoring as a necessary condition for acceptance of positive results

I insist you do a blind test of your dubious, extraordinary claims about "organic" sounds from DACs. Yes, given your/Amir type history and pecuniary interests, like every AES test, it would have to be proctored.
Loudspeaker manufacturer

How do you listen to an ABX test?

Reply #1288 –
Quote from: 2Bdecided on
No John, we've talked about where 16-trial ABX is good enough, where it isn't, what the alternatives are, and shown that we use those alternatives too.
Well, I think this would be good in some sort of the decision tree style chart showing when 16-trial ABX is suitable, when it's not & what the decisions are that lead to other tests - as knucklehead suggested. Because, to be honest, I can't remember this logic being spelled out in the way you report. But then it's easy to miss some important information among a flurry of posts, most of which are just personal insults & not an attempt to answer questions or address points raised.

Quote
Meanwhile, you don't appear to have much experience with ABX testing, don't understand level matching, and don't realise when your questions have been answered.
I'm open to being corrected about ABX testing. Agreed my level matching figures were off - mea culpa. As to the last point - I feel, it's that the central point hasn't been answered not that I don't realise it.

Quote
I know when I'm wasting my time.

Goodbye

Cheers,
David.
OK

How do you listen to an ABX test?

Reply #1289 –
Quote from: jkeny on
Do you mean the pre-screening shows this or do you mean that previous tests show it's possible to discern differences? If you are suggesting that other tests can be used as a validation that this test is there fore sensitive enough, I would have to disagree with you.

Why?
Previous true positive ABX logs pile up as a mountain of evidence supporting that the method itself and more specifically the implementation works. So at this point there already is little to worry about for a personal test, in which I guess you just want to test if you can hear a difference with your system. But a reality check with a low bitrate mp3 first still wouldn't hurt. Could save you some time if you have bad hearing or a bad system.

Passing such a check would be confirmation that the current conditions and test method (e.g. ABX) are fine.



Quote from: jkeny on
Willingness & motivation are necessary but that doesn't mean that the test conditions itself haven't effected the participant's performance. I suggested catch trials as the most direct way of doing this.

And we have told you that catch trials will negatively impact performance for a couple of reasons. You are basically disrupting the participants and prolonging the test needlessly.
But sure, go ahead and implement your test that requires at least 3 randomly switched DUTs during the 'main' test. Be sure to tell people that A or B can randomly switch to this third DUT, so they cannot concentrate on just comparing the two devices of interest.

Be ready for the attacks from your friends that will throw terms like 'sensory overload' and 'mental distraction'  at you. And once you have the results be sure to ask your favorite omniscient source to tell you how badly your intervention has distorted the very thing you tried to measure in the first place.


Quote from: jkeny on
I will when I get a chance but I don't remember anything you posted which addressed my point here.

No comment.


tl;dr: Read the very beginning of this discussion.
"I hear it when I see it."

How do you listen to an ABX test?

Reply #1290 –
And he still pretends we didn't already do what he's asking?

Of course, and when he's presented with how this forum uses ABX protocol and is reminded of it, he side-steps with this bullshit:
http://www.hydrogenaud.io/forums/index.php...st&p=897299

Wasting our time constructing flow-charts will not change a bloody thing, but hey, knucklehead, you're free to do it if you like.

Honesty control question to jkney (don't bother responding to the stuff above, it wasn't directed at you):
Do you accept that this forum does routinely employ and recommend the types of controls and methods found in BS-1116 (hint: MUSHRA, ABC/HR)?

How do you listen to an ABX test?

Reply #1291 –
@xnor:
It's a wonder how this community ever managed to assist in improving lossy codecs over the last dozen+ years.  You know with all this parading of null results which contain a preponderance* of false-negatives that we do?

*) for giggles, has anyone looked up a definition of preponderance?

How do you listen to an ABX test?

Reply #1292 –
Quote from: jkeny on
Quote from: 2Bdecided on
No John, we've talked about where 16-trial ABX is good enough, where it isn't, what the alternatives are, and shown that we use those alternatives too.
Well, I think this would be good in some sort of the decision tree style chart showing when 16-trial ABX is suitable, when it's not & what the decisions are that lead to other tests - as knucklehead suggested. Because, to be honest, I can't remember this logic being spelled out in the way you report. But then it's easy to miss some important information among a flurry of posts, most of which are just personal insults & not an attempt to answer questions or address points raised.

Quote
Meanwhile, you don't appear to have much experience with ABX testing, don't understand level matching, and don't realise when your questions have been answered.
I'm open to being corrected about ABX testing. Agreed my level matching figures were off - mea culpa. As to the last point - I feel, it's that the central point hasn't been answered not that I don't realise it.

Quote
I know when I'm wasting my time.

Goodbye

Cheers,
David.
OK


I would be really interested to see one that you would come up with. 

How do you listen to an ABX test?

Reply #1293 –
Quote from: jkeny on
[...]
Quote
A major consideration is the inclusion of appropriate control conditions. Typically, control conditions include the
presentation of unimpaired audio materials, introduced in ways that are unpredictable to the subjects. It is the differences
between judgement of these control stimuli and the potentially impaired ones that allows one to conclude that the grades
are actual assessments of the impairments.

[...]
Just to note - the BS.1116 is strongly recommending the use of such controls in formal testing to validate the test. It would seem to me that there is an even greater need for such controls in the less formal environment of home tests

In an ABX test, the "unimpaired" sample is presented as X in an unpredictable way during an a priori unknown number of trials. How about that?

On a different note: I fail to see how the controls could be designed without leaving a backdoor for the usual placebophile criticism.
Let's say I wanted to test the transparency of a 256k MP3 for a specific sample. I have read and heard many audiophiles claim that MP3 can never be transparent (at any bitrate). I myself make no such claim and will supply the null-hypothesis: the MP3 is audibly indistinguishable from the original. The alternative hypothesis is of course, that the MP3 is indeed audibly distinguishable from the original. Experts, please correct me If I'm stating this wrong.

jkeny, please suggest a suitable control which will convince audiophiles that the null-hypothesis has credibly not been rejected. In a sort of role reversal, I will try to discredit your suggestion using oft-heard audiophile excuses.

How do you listen to an ABX test?

Reply #1294 –
Quote from: jkeny on
I'm open to being corrected about ABX testing.


My hypothesis is that you are not open to correction of any of your most significant erroneous ideas about ABX, and the proof is your lack of concession that the errors in your ideas even exist.

Quote
Agreed my level matching figures were off - mea culpa.


It is days later and this is the first I've seen that you've conceded even this trivial point. 

But the point still stands - trivial errors like this happen because of your utter disrespect for ABX such that you won't bother to learn anything at all about what it is really about.

My model  of Jkeny is that the time it takes him to admit his errors depends on his emotional stake in each error which is very small for the level matching issue and much, much higher for the really serious boo-boos.

So on the day when Jkeny concedes anything important related to his multitudinous misapprehensions about ABX, Everest has been washed into the sea by geological action

Quote
I know when I'm wasting my time.


I seriously doubt that!

How do you listen to an ABX test?

Reply #1295 –
Quote from: greynol on
And he still pretends we didn't already do what he's asking?

Of course, and when he's presented with how this forum uses ABX protocol and is reminded of it, he side-steps with this bullshit:
http://www.hydrogenaud.io/forums/index.php...st&p=897299

Wasting our time constructing flow-charts will not change a bloody thing, but hey, knucklehead, you're free to do it if you like.

Honesty control question to jkney (don't bother responding to the stuff above, it wasn't directed at you):
Do you accept that this forum does routinely employ and recommend the types of controls and methods found in BS-1116 (hint: MUSHRA, ABC/HR)?

Yes, I do see these controls being used in HA.

And it may not be possible to implement these same controls in an ABX test - without disrupting it in a negative way - I accept that now.
I initially thought it would be possible but you have convinced me that I'm most likely wrong.

This still leaves me with the dilemma of how to validate the sensitivity of the results from an ABX test without this sort of extra information about the test itself. My thinking is that ABX tests have all sorts of sensitivity issues which go unnoticed due to this lack of self-validating controls & the wide variety of test conditions.

I may have used bad examples or got some things wrong in this thread but I'm still left with this large question mark about ABX results - I just have no way of knowing how sensitive that particular ABX test was & therefore I have no way of judging the validity/meaning of it's results.

How do you listen to an ABX test?

Reply #1296 –
Quote from: jkeny on
If you are suggesting that other tests can be used as a validation that this test is there fore sensitive enough, I would have to disagree with you.


Since this is not what I have been saying for weeks, the above is yet more proof that Jkeny is not capable of understanding the replies he is receiving.


How do you listen to an ABX test?

Reply #1297 –
I see Arny just posted before me so can I highlight my errors & repeat my post

"Yes, I do see these controls being used in HA.

And it may not be possible to implement these same controls in an ABX test - without disrupting it in a negative way - I accept that now.
I initially thought it would be possible but you have convinced me that I'm most likely wrong.

This still leaves me with the dilemma of how to validate the sensitivity of the results from an ABX test without this sort of extra information about the test itself. My thinking is that ABX tests have all sorts of sensitivity issues which go unnoticed due to this lack of self-validating controls & the wide variety of test conditions.

I may have used bad examples or got some things wrong in this thread but I'm still left with this large question mark about ABX results - I just have no way of knowing how sensitive that particular ABX test was & therefore I have no way of judging the validity/meaning of it's results.

How do you listen to an ABX test?

Reply #1298 –
Quote from: jkeny on
I see Arny just posted before me so can I highlight my errors & repeat my post

"Yes, I do see these controls being used in HA.

And it may not be possible to implement these same controls in an ABX test - without disrupting it in a negative way - I accept that now.
I initially thought it would be possible but you have convinced me that I'm most likely wrong.


You still aren't comprhending the basic message that I've been sending for weeks, Keny.

It is possible to implement the kind of controls I'm talking about without disrupting the tests, because I've done it in dozens of tests for over 15 years.

Quote
This still leaves me with the dilemma of how to validate the sensitivity of the results from an ABX test without this sort of extra information about the test itself.


It's a dilemma of your own creating, Keny.

Just goes to show what kind of fragile straws you will clutch for to avoid admitting the truth.

I rest my case!

Quote
My thinking is that ABX tests have all sorts of sensitivity issues which go unnoticed due to this lack of self-validating controls & the wide variety of test conditions.


Right, and now Keny you've regressed all the way back to where you were when you came in the door.

Nobody is surprised!

How do you listen to an ABX test?

Reply #1299 –
Quote from: Arnold B. Krueger on
Quote from: jkeny on
I see Arny just posted before me so can I highlight my errors & repeat my post

"Yes, I do see these controls being used in HA.

And it may not be possible to implement these same controls in an ABX test - without disrupting it in a negative way - I accept that now.
I initially thought it would be possible but you have convinced me that I'm most likely wrong.


You still aren't comprhending the basic message that I've been sending for weeks, Keny.

It is possible to implement the kind of controls I'm talking about without disrupting the tests, because I've done it in dozens of tests for over 15 years.

Sorry but I don't see pre-screening & pre-training as sufficient.