Skip to main content

Topic: ABX vs. ABXY (Read 6088 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
ABX vs. ABXY
I've noticed several double-blind testing tools have (or give the option of) ABXY testing. For those who haven't seen this, it involves randomized duplication of the A and B samples as X and Y. Therefore, the options to the test subject are:

A is X, B is Y    or    A is Y, B is X

Under conventional ABX, the A and B are compared with X, with X being a duplicate of either A or B chosen at random. The options for ABX are therefore:

A is X    or   B is X

I'm struggling to see the logic of using ABXY over ABX. Statistically speaking, I can't see how it has any effect on alpha or power.

Is there a particular methodological/statistical reason why it is used, for example, by the ABX Comparator tool?

  • IgorC
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #1
Not sure about statistics but ABXY is more efficient than ABX during listening test.

If differences are subtle then during ABX test You need to listen A-B-X-A-B-X (or B-A-X-B-A-X) several times because human memory is pretty short.

While I generally need just one pass A-B-X-Y to spot the difference quickly.

P.S. In other words it's critical for me to have a fast access to both X and Y and switch between them quickly to spot difference in a very short amount of time.
(less than 1 second).
  • Last Edit: 30 June, 2017, 11:53:08 AM by IgorC

Re: ABX vs. ABXY
Reply #2
Apologies IgorC if this seems a stupid question (! :) ), I'm just trying to get my head around how other people approach the two tests:

When you click 'A-B-X-Y', you're effectively playing 'A-B-A-B' or 'A-B-B-A' depending on how it's been randomised. If you're identifying subtle differences between the B and the X (as they are sequential in your pattern), then would that same inference not be drawn by playing A-B-X-A or A-B-X-B under regular ABX conditions?

When I do ABX, for example, I  tend to play A, B and then X. If B and X sound similar, I'll immediately play A again to gauge whether the difference with X lies there. If they do not sound similar, I'll still play A again, only this time to confirm that it is similar to X (which was just playing), thus establishing that the difference must exist between B and X.

Interested to hear what you and others think of the two approaches, and how you personally sequence your testing. :)

  • IgorC
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #3
then would that same inference not be drawn by playing A-B-X-A or A-B-X-B under regular ABX conditions?
I think You have answered your own question.

I got 2x time gain. Instead of playing 2 combinations (ABXA and ABXB)  I just play one ABXY which is NOT the same as ABXA and/or ABXB (because of short sound memory).
And that really matters because less time means less fatigue -> more reliable outcome of blind test.

  • Last Edit: 30 June, 2017, 10:22:02 PM by IgorC

  • Kamedo2
  • [*][*][*][*]
Re: ABX vs. ABXY
Reply #4
I strongly support ABXY.

Let's say if
A score is 5.0 - the original.
B score is 4.9 - the encoded.
X score is 5.0 - actually the original.

The ABXABXABXABX sequence is perceived like:
5.0 4.9 5.0 5.0 4.9 5.0 5.0 4.9 5.0 5.0 4.9 5.0

The AXAXAXBXBXBX sequence is perceived like:
5.0 5.0 5.0 5.0 5.0 5.0 4.9 5.0 4.9 5.0 4.9 5.0

The XYXYXY sequence is perceived like:
5.0 4.9 5.0 4.9 5.0 4.9

Then FFT those perceived score sequence. The XYXYXY sequence has the strongest non-dc power, therefore easiest to detect.
(The DC is very hard to detect because, like the IgorC said, human memory is short, and in a very high-rate listening test, the listener will get paranoid, and start to find tiny recording flaws of the original; Now both of them seem encoded.)

Another advantage is that XYXYXY has the greatest Hamming distance between the case X was 5.0 and the case X was 4.9.

The ABXABXABXABX sequence is, in case if the X is A, like
ABAABAABAABA, therefore,
5.0 4.9 5.0 5.0 4.9 5.0 5.0 4.9 5.0 5.0 4.9 5.0
The ABXABXABXABX sequence is, in case if the X is B, like
ABBABBABBABB, therefore,
5.0 4.9 4.9 5.0 4.9 4.9 5.0 4.9 4.9 5.0 4.9 4.9
The Hamming distance is
0.0+0.0+0.1+0.0+0.0+0.1+0.0+0.0+0.1+0.0+0.0+0.1+...

The XYXYXY sequence is, in case if the X is A, like
ABABAB, therefore,
5.0 4.9 5.0 4.9 5.0 4.9
The XYXYXY sequence is, in case if the X is B, like
BABABA, therefore,
4.9 5.0 4.9 5.0 4.9 5.0
The Hamming distance is
0.1+0.1+0.1+0.1+0.1+0.1+...

Thus ABXY is a great tool to detect tiny flaws on the verge of perceptibility.

  • Speedskater
  • [*][*][*]
Re: ABX vs. ABXY
Reply #5
I think that you will loss a lot of resolution.
A skilled listener does:
1]  A : X same or different?
2]  B : X same or different?
It's easier than trying to remember both A & B when listening to X.
Kevin Graf :: aka Speedskater

Re: ABX vs. ABXY
Reply #6
I got 2x time gain. Instead of playing 2 combinations (ABXA and ABXB)  I just play one ABXY which is NOT the same as ABXA and/or ABXB (because of short sound memory).
And that really matters because less time means less fatigue -> more reliable outcome of blind test.

Why would one play ABXA when you can play AXB or AXBX and get essentially the same thing you seem to desire - close comparisons of A and X and B and X.

I see Y as an interfering or worse, irrelevant variable.

  • IgorC
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #7
Why would one play ABXA ...
And how a listener knows  beforehand how sound A and B, huh?

Let's not just concetrate on a pure sequences and let's try to think what happens with a listener during test because we aren't just robots who perceive certain sequences and have perfect memory. It's a huge oversimplification

 
Why would one play ABXA when you can play AXB or AXBX
AXB can be AAB or ABB. So I will waste my concetration  on "AA" part and only then move to "AB" in 50% of cases. Hm. No deal for me. It's a big deal to have an access to X and Y in a short time when differences are subtle.

  • Last Edit: 01 July, 2017, 10:01:12 AM by IgorC

  • IgorC
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #8
I think that you will loss a lot of resolution.
A skilled listener does:
1]  A : X same or different?
2]  B : X same or different?
It's easier than trying to remember both A & B when listening to X.
Simply not true.

Re: ABX vs. ABXY
Reply #9
Then FFT those perceived score sequence. The XYXYXY sequence has the strongest non-dc power, therefore easiest to detect.
(The DC is very hard to detect because, like the IgorC said, human memory is short, and in a very high-rate listening test, the listener will get paranoid, and start to find tiny recording flaws of the original; Now both of them seem encoded.)

Another advantage is that XYXYXY has the greatest Hamming distance between the case X was 5.0 and the case X was 4.9.

The ABXABXABXABX sequence is, in case if the X is A, like
ABAABAABAABA, therefore,
5.0 4.9 5.0 5.0 4.9 5.0 5.0 4.9 5.0 5.0 4.9 5.0

Kamedo, could you please tell me what you mean by FFT and non-DC power? I'm not familiar with the terms in this context

  • Kamedo2
  • [*][*][*][*]
Re: ABX vs. ABXY
Reply #10
I think that you will loss a lot of resolution.
A skilled listener does:
1]  A : X same or different?
2]  B : X same or different?
It's easier than trying to remember both A & B when listening to X.
I can remember both A & B, and answer accordingly in low rate listening tests. However, in high rate listening tests, the difference between A and B is so subtle that human memory can be washed out quickly. I like ABXY. It withstands in noisy channel.

Re: ABX vs. ABXY
Reply #11
Why would one play ABXA ...
And how a listener knows  beforehand how sound A and B, huh?

Let's not just concetrate on a pure sequences and let's try to think what happens with a listener during test because we aren't just robots who perceive certain sequences and have perfect memory. It's a huge oversimplification

 
Why would one play ABXA when you can play AXB or AXBX
AXB can be AAB or ABB.

Right, so if you can hear the difference between A & B the difference is there to hear because in either case you have an A versus B comparison.

Following the same logic, AXBX is the same as AABA if X is A and ABBB if X is B.  You have an A versus B comparison, either way.

Quote
So I will waste my concetration  on "AA" part and only then move to "AB" in 50% of cases.

There Is no such thing as wasting concentration.  When you listen you obtain information that is valuable to the identification process.  Hearing that X sounds the same as A is just as valuable as hearing that it is different. If it the same, then it is not different, and if it is different, then it is not the same.

Quote
Hm. No deal for me. It's a big deal to have an access to X and Y in a short time when differences are subtle.

Whatever makes you feel good!  I suspect that you are so prejudiced in favor the need for the Y variable that lacking it will probably hurt your concentration, even though that is not generally true.

Re: ABX vs. ABXY
Reply #12
Why would one play ABXA ...
And how a listener knows  beforehand how sound A and B, huh?

Ever hear of listener training?


  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: ABX vs. ABXY
Reply #13
Whatever makes you feel good!  I suspect that you are so prejudiced in favor the need for the Y variable
Are you just prejudiced in favor of ABX test?

  • silverprout
  • [*][*][*]
Re: ABX vs. ABXY
Reply #14
I got 2x time gain. Instead of playing 2 combinations (ABXA and ABXB)  I just play one ABXY which is NOT the same as ABXA and/or ABXB (because of short sound memory).
And that really matters because less time means less fatigue -> more reliable outcome of blind test.

Why would one play ABXA when you can play AXB or AXBX and get essentially the same thing you seem to desire - close comparisons of A and X and B and X.

I see Y as an interfering or worse, irrelevant variable.

as our high accuracy memory is short our judgement accuracy time is also very short.
imho in order to manage our moments of very high accuracy efficientely, we should intoduce a sort of ina
ccurate flow of data in order to extract the high accuracy moments.
  • Last Edit: 01 July, 2017, 05:53:25 PM by silverprout

  • Speedskater
  • [*][*][*]
Re: ABX vs. ABXY
Reply #15
I think that you will loss a lot of resolution.
A skilled listener does:
1]  A : X same or different?
2]  B : X same or different?
It's easier than trying to remember both A & B when listening to X.

Simply not true.
Don't ever have a listening contest with a skilled listener. They can notice differences way to small to identify just what the differences are. They go with 'same or different' not sounds like A or B which requires a much larger difference.
Kevin Graf :: aka Speedskater

  • Kamedo2
  • [*][*][*][*]
Re: ABX vs. ABXY
Reply #16
How I use ABXY, after playing A and B.

When the bitrate is very low....
I play X, and answer.
When the bitrate is fairly low, but if I just want to confirm my guess, I play the opposite.
I play X, Y, and answer.
When the bitrate is very high and the artifacts are very subtle, I tend to find the minuscule recording flaw of the original, so the question will be more like "X and Y, which is more damaged?".
I am a human, so I have a confirmation bias. The first guess, be it right or wrong, is more likely to be reinforced on the next play.
To reduce this bias, first, let myself believe that X is better than Y, and play X, Y, X, Y, X, Y, ... 
next, let myself believe that Y is better than X, and play X, Y, X, Y, X, Y, ...
Finally, think to myself, "Which confirmation was stronger?"

That's the way I perform, and pass, the high rate ABXing.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: ABX vs. ABXY
Reply #17
For me, it's really simpler to find subtle differences when I can do XYXYXY... comparison.

  • smok3
  • [*][*][*][*][*]
  • Moderator
Re: ABX vs. ABXY
Reply #18
perhaps it would interesting to try
ABXZ (where Z is random A or B and I'am told which one it is), unless that doesn't break the stats in some way.
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

  • [JAZ]
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #19
The sequence "ABXY" by itself, versus "AXBX" isn't really much different, as Arnold said.

What IgorC and others seem to put emphasis on , is the fact that ABXY let's the tester forget completely about A and B, and focus on locating the difference between X and Y, if there is any which is audible.
Once the difference is found, the next step is seeing if A has the difference, or B has it, and be able to make the A->X, (and so B->Y) or B->X (and so A->Y) connection.

With ABX, you can do exactly the same thing: Let's forget completely about X, and focus on locating the difference between A and B, if there is any which is audible.
Once the difference is found, the next step is seeing if X has the difference, or if X doesn't have it, and be able to make the A->X (and so B-> not X) or B->X (and so A-> not X) connection.

Obviously, the objective is determining the sample under the X, either with ABX or with ABXY, and for that, the difference alone is not enough.
  • Last Edit: 03 July, 2017, 02:11:12 PM by [JAZ]

  • Kamedo2
  • [*][*][*][*]
Re: ABX vs. ABXY
Reply #20
Once the difference is found, the next step is seeing if A has the difference, or B has it,

If my understanding is right, you play AXAXAX and see if A and X have the difference. If it's inconclusive, then you play BXBXBX and see if B and X have the difference.

If the initial guess "A has the difference" is right, A has a higher fidelity than X. If the guess is wrong, then A is same as X. That's not as dramatic as having fidelity opposite to the expectation.

The concept "same" is a tricky one. If you could deduce a difference early in the evaluation, you're lucky. But if you couldn't, it could be either: a) there is indeed no difference. b) there is a difference you missed it by whatever reasons, including but not limited to, forgetting the "knack" by the time elapsed.
That's a devil's proof that consumes the tester's precious time.

In an ABXY testing, if you play X, you can always guarantee that the one you play next will be "different". It can be of higher fidelity or lower fidelity, but never be the same as X.

Of course, you can switch the guess in the ABX test, but why you switch the guess is that the initial guess was inconclusive. The time you spent on pondering whether it's same or not -- it's the devil's proof -- is wasted. Actually, the devil's proof does appear in a ABXY test, when memorizing A and B and the case you don't find any differences, and you think of giving up the test. But that's the consideration you do only once in a single ABXY session, not something you do 5, 8 or 20 times in a single session.

In conclusion, ABXY session is faster to perform because it eliminates the devil's proof.

  • [JAZ]
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #21
@Kamedo2 I don't follow you.  I said that if you are interested in locating the difference, you play A versus B, not A versus X or B versus X. That's the part that I said is the same than playing X vs Y.  There's no need for Y to locate the difference.

I really cannot understand why finding a difference between A vs B is harder than finding a difference between X vs Y.

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: ABX vs. ABXY
Reply #22
I really cannot understand why finding a difference between A vs B is harder than finding a difference between X vs Y.
It isn't, of course. But finding this difference is only the first step.

  • [JAZ]
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #23
I still don't get it... If finding the difference is just as easy with ABX or ABXY,  then, what makes ABXY better? I am still not buying the "I do less tests with ABXY " sentence.

  • IgorC
  • [*][*][*][*][*]
Re: ABX vs. ABXY
Reply #24
I still see that people still ignore the human factor. We  haven't perfect memory.

It's not about sequences  but about timing.

Some people here (including me) claim usefulness of ABXY on a subtle differences. It's important.
Because good level of concentration  is _very short_ actually. And you need a high level of concentration on a subtle differences. You could spot a subtle difference on A-B, at this moment your concentration isn't good enough and now imagine ... You should move to A-X and/or B-X. You are 50% lucky (2 possibilities: A-X or B-X) to get a correct spot. But You get 100% if you move directly to X-Y.

And, again, all this test time your concentration is degrading at very high rate. It's not about logic, mathematics, statistics. Non of them can't describe it. But about an interaction between a test and human.

I think this is the case when common sense doesn't apply. There are scientific theories which go against a common sense but were proven to be correct.