Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New version of statistical analysis tool (Read 5405 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New version of statistical analysis tool

Some time ago I needed an analysis of some test results and tried to use the bootstrap utility we have used for the listening tests. Unfortunately, the results coming out were bogus. I traced it down to an obscure 64-bit compatibility issue, but going through the code some things bothered me. ff123 improved my initial version significantly, but one of the things that was done was to use a normal distribution approximation for test statistics. If you consider the original version of the utility was exactly written to avoid any assumptions about normality, that's a bit sad.

So I ended up rewriting the whole thing and fixing all outstanding issues. The new version:

  • Works correctly on 64-bit systems
  • Removed all arbitrary limitations of number of samples, codecs, ...
  • p-values are estimated through Monte Carlo resampling instead of normal distribution approximation
  • Blocked and non-blocked analysis fully supported
  • Comparison based on median instead of means supported
  • Possible to (only) compare all samples against the first one
  • Much slower because it's in Python (v2.5+ required)


This is new so it might still contain some bugs. Any feedback appreciated.

Download page

New version of statistical analysis tool

Reply #1
  • p-values are estimated through Monte Carlo resampling instead of normal distribution approximation


A quick question: why is it that the usual (binomial) p-values for n trials and k successes are calculated as (in pseudo-TeX notation):

\sum_{i = 0}^k \choose{n}{i} p^i q^{n-i}

where p is the probability of success in a Bernoulli experiment and q = 1 - p, instead of only:

\choose{n}{k} p^i q^{n-i}

If the person correctly marked k of those trials are the "correct sample" and there are \choose{n}{k} possibilities given of choosing k from a row of n experiments, why are we summing for other values of k?

New version of statistical analysis tool

Reply #2
Because we're interested in the odds that randomly picking will produce a score of k successes or more.