## Re: Maths of ABX tests

#####
Reply #6 –

One of the "issues" (not really for practical purposes) is that significantly ">0.5" does not say how much. If you make 1 000 000 000 000 trials with 500 000 000 000 successes, then you can be confident that you are within 49.9999 and 50.0001 percent. Which is a worry if your life depends on some probability being "1/2" and not "500001 in a million".

The following two statements are not the same:

Given iid trials with a fixed probability >1/2, then there is a number N of trials such that the test should conclude.

Given iid trials with a probability >1/2 and a test with N trials, you should get a conclusion.

Since in the latter case N is given, there is an interval (containing 1/2) of probabilities which is so narrow that N trials is way too little.

But there is a different issue with the OP's framing, and that is to distinguish between ">1/2 vs =1/2" and ">1/2 vs at most 1/2". That is the one-sided vs. two-sided test thing.