Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Blind report (Read 4168 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Blind report

Having just read the stereophile article regarding amplifier blind testing referred to above, I am astounded by the problems we still have with the assumptions of basic statistics. Are there no proper scientists here? Or do they only do stats by rote, i.e. connect the dot stats, or paint by numbers...

I suck at stats. But so do the people analysing and critiqueing the data and results. Nowhere in these tests have they even looked at that basic premise of the stats applied, the absolute requirement for a normal distribution to even do most statistics. What is a normal distribution? The basic in-alienable variation in performance against any measure which any population WILL have. I mean we have a LD50 score which is the amount of a substance that will kill 50% of a population... that is based on tadaa!!! the normal distribution curve... Why don't you dose too heavily with pesticides when you dose a crop field? because you breed pesticide resistence in your pests... Why? Not because the insects are LUCKY?!?! WTF???

So the mere fact that the stereophile tests have a normal distribution curve  even a skewed one by virtue of their test set-up... means (firstly sample size was sufficient-ish) but more importantly that there are some people who DO hear a difference, and some who DON'T and others who are so hell bent on guessing right that they get it wrong more than "statistically" probable. There is no such thing as a lucky coin in a population - there are only variations in performance. How do we breed better cows? by choosing the performance measure to rate our population and taking those at the top end of the performance measure and breeding with them, hopefully we choose enough measures so that we don't kill other desirable traits - and yes if your focus is too narrow this will happen!!! But if they were mere "lucky coins" there would be absolutely no use in selectively breeding with them.

Moral of the story, stats have an underlying meaning based on it's assumptions, don't lose sight of these... Only in a gippo'd stack of coins do you get that "lucky coin". Everywhere else you get expected performance variations... and of course go flip an audiphile, OK, maybe just AB test your prospective SO... it may lead to companies not just designing for the current average, but possibly for the elite again.

I mean I have no use for ferrari / lambo / Mclaren / GTR / Porsche / Ariel performance - I don't drive well enough, but I might be able to buy it, because I want it and like that even I can drive faster in it and it appeals to certain aspects of my value system... Does that mean that actually it has no better performance than an M3 or AMG or Audi RS? or whatever the average joe can drive "optimally fast in" just after they got their driver's license?

Where are the people that actually GET maths and stats...

Blind report

Reply #1 –
Quote from: Rich B on
Having just read the stereophile article regarding amplifier blind testing referred to above, I am astounded by the problems we still have with the assumptions of basic statistics. Are there no proper scientists here? Or do they only do stats by rote, i.e. connect the dot stats, or paint by numbers...


The statistics situation has been well understood and effective recommendations have been made all along by professionals.  The ABX Development Team included a PhD-level medical researcher, a degreed mathematician, and two degreed engineers, one who had done considerable postgraduate work in statistics and math.  The Stereophile Experimenters went out on their own and tried to break new ground, but failed and were effectively criticized by professionals with good credentials in the field of Math and Statistics.

Quote
I suck at stats. But so do the people analyzing and critiquing the data and results.


Since you admit you suck at stats, how can you credibly critique people who have proper credentials and don't suck at math?

Quote
Nowhere in these tests have they even looked at that basic premise of the stats applied, the absolute requirement for a normal distribution to even do most statistics.


That is a false claim. A novice who sucks at stats who reads a few letters to the editor by people who are trying to correct the mess that Stereophile published does not get enough information to judge.

The results of an ABX test do not follow a normal distribution within the tests and trials themselves. An ABX test is a simple same/different test with two possible outcomes per trial, correct or incorrect. Random guessing in an ABX test is well-modeled by the Binomial Distribution.

Blind report

Reply #2 –
Quote
Since you admit you suck at stats, how can you credibly critique people who have proper credentials and don't suck at math?


My wife has a masters degree in statistics. Since I may suck at it, I doubt she does.