As I said, the oscillation of the difference between the simulation and the calculation is very suspicious. And I don't see how it could have come from the simulation.
QuoteAs I said, the oscillation of the difference between the simulation and the calculation is very suspicious. And I don't see how it could have come from the simulation.The calculated numbers shown above by continuum are the right values. (It would be extremely unlikely for Maple to be wrong here anyhow).This may be a stupid question. But when you run the simulation multiple times what is the variation? Could it account for the differences?
Any thoughts on the non-even spreading of the alpha error?
QuoteAny thoughts on the non-even spreading of the alpha error?I can't see any problem with it. Maybe there are some subtle effects on the chances of type-2 errors. You could run some simulations to check that out.My thought is that you shouldn't limit the design to evenly spread alpha errors. Arguably, the overall Pr(type-1 error) is all you really need to constitute a valid significance test anyhow.
But going back to the 28-trial case, I still get 0.0491. So there was something wrong with my random numbers, but there must still be something wrong with your calculation (could still be roundoff errors).
>LookPVal(28, array([0.95, 0.95, 0.98, 0.98,0.98]), array([6, 12, 18, 23,28]));> evalf(%); 1704631 -------- 33554432 .05080196261405944824218750000000000000000
at least 6 of 6at least 10 of 12at least 14 of 18at least 17 of 23at least 20 of 28 5,08019626140594E-02
1. No looks allowed for trials 1 through 5
I still don't know why the simulation doesn't agree with the calculation. Let's try a very simple one. Can you calculate the overall alpha for the following lookpoints, maximum 6 trials:2 of 23 of 44 of 6The exact answer is 0.453125My program yields 0.4530 with 10 million simulationsff123
at least 2 of 2at least 3 of 4at least 4 of 60,453125 -> exact
I've been thinking some more about the in-between-look terminations. Since the listener cannot make a decision to continue the test after he terminates it, I think I have calculated things wrong. For example, if the listener gets a look at trial 6, but then stops at trial 8, then all the other looks at trial 12, 18, 23, and 28 should not be counted towards the overall alpha.
I'm having difficulty understanding your point. Are you saying that my in-between values are correct or not?
My thinking came about because I was trying to decide what should be done if the listener performs without knowing progress up to trial 5, and then terminates. What should the program show if he got all 5 correct? It should show an unadjusted alpha of 0.031. In other words, if the listener cannot make a decision to continue or not based on information he has seen, there should be no adjustment.
BTW, I will probably go ahead and show progress for trials 1 through 4. Yes, the listener can perform a Bayesian analysis and decide to stop if he gets all 4 wrong, but since this test is mainly interested in type 1 errors, that should not be a problem.
at least 6 of 6at least 10 of 12at least 14 of 18at least 17 of 23at least 20 of 28 4,91552352905273E-02 at least 2 of 1at least 3 of 2at least 4 of 3at least 5 of 4at least 6 of 5at least 6 of 6at least 10 of 12at least 14 of 18at least 17 of 23at least 20 of 28 4,91552352905273E-02 at least 2 of 1at least 3 of 2at least 4 of 3at least 5 of 4at least 5 of 5at least 6 of 6at least 10 of 12at least 14 of 18at least 17 of 23at least 20 of 28 6,20170868933201E-02
Ok, I'm reviewing your 13/18 case. Using my current table, if the listener is allowed to stop at 19, he does have a 50% chance of randomly getting the next one right and passing the test overall. But what's wrong with that? To get 13 of 18, he had to be pretty close to an overall significance of 0.05 in the first place (sim says about 0.062).
Yes, but nominally, without the option to stop at 19 his chances are far lower, i.e. in a strict 28-trial look point test, they are only about 0.25586. (see above)The other in-between points might have the same problem.
I'm considering things purely from a simulation point of view right now. I.e., what does the simulation say the overall probability of getting 14 of 19 is when he is allowed to look at trials 6, 12 and 18 (and terminate early) and then allowed to stop at trial 19?The simulator says 0.0495 probability of terminating at any of the look points or at trial 19 with an adequate score.
How should one modify the simulation?
Right now the simulation says that the listener always terminates at a look point if the total alpha is 0.05 or less. This is as much to the listener's advantage as possible.
But there is another approach to all this. So far we have investigated the "frequentist" approach. The Bayesian approach could be just as interesting. Say that the listener based his decision on whether or not to continue based on his past performance. What should be his decision at each look point?