Hmm, that's not acceptable for doing actual analysis on though

Post-screening methods can be roughly separated into at least two classes; one is based on inconsistencies compared with the mean result and another relies on the ability of the subject to make correct identifications. The first class is never justifiable. Whenever a subjective listening test is performed with the test method recommended here, the required information for the second class of post-screening is automatically available. A suggested statistical method for doing this is described in Appendix 1.

The methods are primarily used to eliminate subjects who cannot make the appropriate discriminations. The application of a post-screening method may clarify the tendencies in a test result. However, bearing in mind the variability of subjects’ sensitivities to different artefacts, caution should be exercised.

Results after 25000^2 resamples (10+ hours cpu time):

cbr192 is worse than abr224 (0.02208 vs 0.48052)
cbr192 is worse than r3mix (0.04712 vs 0.71088)
cbr192 is worse than dm-xtrm (0.00024 vs 0.01144)
cbr192 is worse than mpc (0.00000 vs 0.00000)
cbr192 is worse than dm-ins (0.00316 vs 0.11308)
cbr192 is worse than cbr256 (0.01032 vs 0.28664)
cbr192 is worse than dm-std (0.00012 vs 0.00588)
abr224 is worse than dm-xtrm (0.07908 vs 0.85528)
abr224 is worse than mpc (0.00080 vs 0.03504)
abr224 is worse than dm-std (0.03364 vs 0.60492)
r3mix is worse than dm-xtrm (0.04024 vs 0.66340)
r3mix is worse than mpc (0.00040 vs 0.01824)
r3mix is worse than dm-std (0.01436 vs 0.36392)
dm-xtrm is worse than mpc (0.04596 vs 0.70132)
mpc is better than dm-ins (0.00808 vs 0.23876)
mpc is better than cbr256 (0.00304 vs 0.10972)
cbr256 is worse than dm-std (0.06392 vs 0.80176)

<Garf> I used raw ratings data, and means.
<Garf> (which is just as well, and even more powerfull)

<Garf> I do not use replacement
<Garf> I changed that after [your] first comment

Originally posted by ff123

I think I have that correct, now.

I agree. 

I'd add medians and stepdown correction, but I'm rather busy with other things like now, so feel free to....

However, the next most extreme p-value counter (dm-std vs. cbr192) is incremented or not only after excluding the value for the most extreme p-value counter 

Don't you mean 'excluding the most extreme p-value'?

and Reference paper (which I don't have): 
Westfall, P.H. (1997) Multiple testing of general contrasts using logical constraints and correlations, Journal of the American Statistical Association, 92, 299-306.

I don't know who's actually reading these posts...

         dm-std   dm-xtrm  dm-ins   cbr256   abr224   r3mix    cbr192

mpc      0.4216   0.2928   0.1263   0.0822   0.0539   0.0342*  0.0016*

dm-std      --    0.8035   0.4672   0.3486   0.2592   0.1871   0.0181*

dm-xtrm     --       --    0.6323   0.4909   0.3788   0.2842   0.0342*

dm-ins      --       --       --    0.8332   0.6877   0.5530   0.1004

cbr256      --       --       --       --    0.8482   0.7019   0.1517

abr224      --       --       --       --       --    0.8482   0.2139

r3mix       --       --       --       --       --       --    0.2928



         dm-std   dm-xtrm  dm-ins   cbr256   abr224   r3mix    cbr192

mpc      0.8494   0.5887   0.1399   0.0569   0.0226*  0.0081*  0.0000*

dm-std      --    0.9999   0.9044   0.7191   0.4983   0.2943   0.0018*

dm-xtrm     --       --    0.9899   0.9257   0.7783   0.5657   0.0081*

dm-ins      --       --       --    0.9999   0.9964   0.9657   0.0871

cbr256      --       --       --       --    0.9999   0.9973   0.2004

abr224      --       --       --       --       --    0.9999   0.3696

r3mix       --       --       --       --       --       --    0.5887

         dm-std   dm-xtrm  dm-ins   cbr256   abr224   r3mix    cbr192

mpc      0.9929   0.9657   0.7902   0.6596   0.5266   0.3970   0.0314*

dm-std      --    1.0000   0.9962   0.9823   0.9506   0.8912   0.2533

dm-xtrm     --       --    0.9997   0.9974   0.9878   0.9621   0.3970

dm-ins      --       --       --    1.0000   0.9999   0.9989   0.7222

cbr256      --       --       --       --    1.0000   0.9999   0.8398

abr224      --       --       --       --       --    1.0000   0.9185

r3mix       --       --       --       --       --       --    0.9657

void qsort_longsamples(longsamples_t *sortedp, int first, int last) {

  int pivot_index, i, j, k;

  long pivot;

  struct {

    long data;   /* data to be sorted */

    int num;     /* numbering of data */

  } temp;

  

  if (first < last) {

    pivot = sortedp->data[first];

    i = first+1;

    j = last;

    while (i <= j) {

      while ((sortedp->data[i] <= pivot ) && (i <= last)) i++;

      while ((sortedp->data[j] > pivot ) && (first < j)) j--;

      if (i < j) {

        temp.data = sortedp->data[i];

        temp.num = sortedp->num[i];

        sortedp->data[i] = sortedp->data[j];

        sortedp->num[i] = sortedp->num[j];

        sortedp->data[j] = temp.data;

        sortedp->num[j] = temp.num;

      }

    }

    temp.data = sortedp->data[j];

    temp.num = sortedp->num[j];

    sortedp->data[j] = sortedp->data[first];

    sortedp->num[j] = sortedp->num[first];

    sortedp->data[first] = temp.data;

    sortedp->num[first] = temp.num;

    qsort_longsamples(sortedp, first, j-1);

    qsort_longsamples(sortedp, j+1, last);

  }    

}