Skip to main content

Topic: Calculating the P-Value (Read 7234 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • torok
  • [*][*][*][*]
Calculating the P-Value
Does anyone want to give me the formula? Thanks.

  • phong
  • [*][*][*][*]
Calculating the P-Value
Reply #1
I was looking for the same thing myself...  And I found it.  I don't think I can insert mathML here, and I think code is a much better explanation than stupid mathy symbols, sere's some code I wrote minus some comments.  It could be written to be much faster, but it is plenty fast enough for the number of trials we'd be talking about.  Can you read Python?
Code: [Select]
def factorial(x):
   if x < 0: raise ValueError("Factorials for numbers less than zero are undefined")
   if x <= 1: return 1
   return reduce(operator.mul, xrange(2, x+1))

def binomial(n, k, p=0.5):
   if k > n:
       raise ValueError("Number of occurances (%d) exceeds number of trials (%d)." % (k, n))
   q = 1.0 - p
   return pow(p, k) * pow(q, n-k) * factorial(n) / (factorial(k) * factorial(n-k))

def abxBinomial(trials, correct):
   if correct > trials:
       raise ValueError("Number of correct ABX trials (%d) exceeds total number of trials (%d)." % (correct, trials)
   return reduce(operator.add, map(lambda k: binomial(trials, k), range(correct, trials+1)))


Quick explanation:
factorial() does exactly what its name says.  It computes the factorial of a number.
binomial() uses the binomial distribution function to compute the probability that there would be exactly k occurances of some event of probability p occuring in n trials.
abxBinomial() computes the ABX pval you want.  It takes the sum of the binomial function for correct, correct+1 ... num_trials.

This method is fine for use on a computer/calculator, but if you're working it out by hand there are formulas which approximate the value more quickly (if less accurately).  Also, be aware that it overflows doubles if you take 170+ trials.
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

  • sshd
  • [*][*][*]
Calculating the P-Value
Reply #2
Quote
whatever

I admint I am not an expert, but the so-called P-value probably refers to the Greek letter Rho, which looks pretty much like a P.

Rho stands for probability.

It is quite easy to calculate the value. Say you have a normal 6 edged dice. The probability (Rho) of getting a 6 in the first try is 1/6 ~ 16,7%. The probability (Rho) of getting 6 in the fist try and 2 in the second try is 1/6 * 1/6  ~ 2,7%.

When you are doing simple ABX tests with only two possible results use 1/2 instead of 1/6.

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Calculating the P-Value
Reply #3
The simplest way to do it is to simulate it:

http://ff123.net/abx/abx.html

You don't have to remember, derive, or look up all those messy equations.  Plus, that's the way things happen in real life anyway.

ff123

  • torok
  • [*][*][*][*]
Calculating the P-Value
Reply #4
Quote
Quote
whatever

I admint I am not an expert, but the so-called P-value probably refers to the Greek letter Rho, which looks pretty much like a P.

Rho stands for probability.

It is quite easy to calculate the value. Say you have a normal 6 edged dice. The probability (Rho) of getting a 6 in the first try is 1/6 ~ 16,7%. The probability (Rho) of getting 6 in the fist try and 2 in the second try is 1/6 * 1/6  ~ 2,7%.

When you are doing simple ABX tests with only two possible results use 1/2 instead of 1/6.

I'm pretty sure that's not it. The P-value is finding the odds that something occured by chance alone, without any other external factors.

  • Continuum
  • [*][*][*][*]
Calculating the P-Value
Reply #5
Quote
I'm pretty sure that's not it. The P-value is finding the odds that something occured by chance alone, without any other external factors.

The p-value is the probability that the test is passed under the condition that you were only guessing. It is not the probability that you were guessing.
(Not sure if you meant this anyway.)


And if you don't speak Python (like me):

Code: [Select]
n! := 1*2*3*...*n  (n>0)
0! := 1

                      a!
binomial(a,b) := -----------  (a>=b)
                  b! (a-b)!

          trials
           ---     binomial(trials,k)
p-val :=   \      --------------------
           /              trials
           ---          2
        k=correct
  • Last Edit: 30 October, 2003, 02:23:46 AM by Continuum

  • Pio2001
  • [*][*][*][*][*]
  • Global Moderator
Calculating the P-Value
Reply #6
Quote
The p-value is the probability that the test is passed under the condition that you were only guessing.

Why does the old PCABX program gives a 100% probability for one success out of one test ? It should be 50 %.

  • Continuum
  • [*][*][*][*]
Calculating the P-Value
Reply #7
Quote
Quote
The p-value is the probability that the test is passed under the condition that you were only guessing.

Why does the old PCABX program gives a 100% probability for one success out of one test ? It should be 50 %.

No idea, I never used it. 50% is correct.

I assume the calculations of PCABX are correct for more trials?

  • StoneRoses
  • [*][*]
Calculating the P-Value
Reply #8
Continuum's definition of P-Value for ABX test is correct. (binomial test;  h0: you are guessing [p=0.5], h1: you are not guessing )

And I think PCABX just use wrong formular.

  • schnofler
  • [*][*][*]
  • Developer
Calculating the P-Value
Reply #9
Just for the sake of completeness, here's the method I use in abchr/java:
Code: [Select]
public static double pVal(int n,int correct) {
    double result=0;
    double binom;
    int twos;
    for(int k=correct;k<=n;k++) {
 twos=n;
 binom=1;
 for(int j=0;j<k;j++) {
     binom*=(n-j)/(double)(k-j);
     while(binom>2 && twos>0){binom/=2;--twos;}
 }
 for(int j=0;j<twos;j++){binom/=2;}
 result+=binom;
    }
    return result;
}

Conceptwise, it's exactly the same as the method phong uses, although not as straightforward. The advantage is mainly that it's probably a bit faster and it doesn't overflow during your casual 170+ trials ABX test 
I don't know exactly when it does overflow, but I tested it once for 10000 trials with 5000 correct and it got the right result after about 5 minutes of calculating (well, this is a scientific forum, you know.. every last 0.0001% of certainty in an ABX test is valued  )
When it comes to optimizing programs I'm a total amateur, though (I rarely do much optimizing in my programs), so there are probably much better ways to calculate that.
  • Last Edit: 30 October, 2003, 12:36:19 PM by schnofler

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Calculating the P-Value
Reply #10
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision).  This skirts both the underflow and speed issues.

ff123

  • schnofler
  • [*][*][*]
  • Developer
Calculating the P-Value
Reply #11
Quote
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

Well, that might be true. But as a math student, I take pride in considering this unprofessional. 

  • ff123
  • [*][*][*][*][*]
  • Developer (Donating)
Calculating the P-Value
Reply #12
Quote
Quote
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

Well, that might be true. But as a math student, I take pride in considering this unprofessional. 

Haha!  Since I'm an engineer, I take that as a compliment.

ff123

  • schnofler
  • [*][*][*]
  • Developer
Calculating the P-Value
Reply #13
Quote
Quote
Quote
In my version of abchr, I don't bother calculating something if it's not going to change the final answer by a certain amount (p values < 0.001 are not reported to their full precision). This skirts both the underflow and speed issues.

Well, that might be true. But as a math student, I take pride in considering this unprofessional. 

Haha!  Since I'm an engineer, I take that as a compliment.

ff123

LOL! 
Who said there can be no holy wars in a perfectly scientific discussion?