New version of statistical analysis tool 2011-02-03 19:54:35 Some time ago I needed an analysis of some test results and tried to use the bootstrap utility we have used for the listening tests. Unfortunately, the results coming out were bogus. I traced it down to an obscure 64-bit compatibility issue, but going through the code some things bothered me. ff123 improved my initial version significantly, but one of the things that was done was to use a normal distribution approximation for test statistics. If you consider the original version of the utility was exactly written to avoid any assumptions about normality, that's a bit sad.So I ended up rewriting the whole thing and fixing all outstanding issues. The new version:Works correctly on 64-bit systemsRemoved all arbitrary limitations of number of samples, codecs, ...p-values are estimated through Monte Carlo resampling instead of normal distribution approximationBlocked and non-blocked analysis fully supportedComparison based on median instead of means supportedPossible to (only) compare all samples against the first one Much slower because it's in Python (v2.5+ required)This is new so it might still contain some bugs. Any feedback appreciated.Download page Last Edit: 2011-08-23 17:30:56 by Garf

New version of statistical analysis tool Reply #1 – 2011-07-29 11:36:01 Quote from: Garf on 2011-02-03 19:54:35p-values are estimated through Monte Carlo resampling instead of normal distribution approximationA quick question: why is it that the usual (binomial) p-values for n trials and k successes are calculated as (in pseudo-TeX notation):\sum_{i = 0}^k \choose{n}{i} p^i q^{n-i}where p is the probability of success in a Bernoulli experiment and q = 1 - p, instead of only:\choose{n}{k} p^i q^{n-i}If the person correctly marked k of those trials are the "correct sample" and there are \choose{n}{k} possibilities given of choosing k from a row of n experiments, why are we summing for other values of k?

New version of statistical analysis tool Reply #2 – 2011-08-23 11:37:52 Because we're interested in the odds that randomly picking will produce a score of k successes or more.