2. Yes, I noticed that the test methodology involves having people send in subjective ratings based upon listening to some sound samples. Basing an opinion on something as enormously subjective as how people interpret sound, especially since they're doing so with non-standardized audio systems.

These surveys have absolutely zero scientific worth and are non-reproducable. If you want to read an article that uses good methodology and testing prodecures, go to http://www.r3mix.net and click on the analysis button.

Response:
[snip]...
Regarding non-standardized audio systems: The concern is that the results could be biased. The listening setups were not standardized, nor was any attempt made to perform a survey of what people were using to listen to the samples. While it might have been interesting to find out if people were using headphones or speakers, or if substandard (i.e., typical multimedia computer speakers) were being used, I think from a practical standpoint, it wouldn't have changed the test procedure or the analysis. How does one account for other things such as background noise in a test? This is as important a factor in being able to hear artifacting as the actual listening equipment. So is listener experience in hearing artifacting.

The most likely effect, in my opinion, of not having controls on the listening setup or on the listening environment, or on the actual listener, is not to add a bias, but rather to add statistical noise or to lessen the sensitivity of the comparison. In other words, I think that untrained listeners or people using cheap multimedia speakers would probably not be able to hear some of the artifacting that others hear, but their relative ratings of codecs would likely not differ radically from the experienced group of people using high quality headphones. Of course that's an unproven conjecture, but I think a reasonable one.

There's another reason why I think that the lack of controls on listener setup, environment, and training is not a fatal flaw: the actual results were statistically significant. I think the most reasonable interpretation of this is that the test was able to discriminate between codec quality in spite of, not because of, the lack of controls.

Are public listening tests constructed in such a way as to abstract the hardware setup?

Should we assume everyone at HA who gets involved with a public listening test has a "quality" setup, i.e. sound card with good DAC's, good speakers/headphones with good amplification, low-noise environment, etc. ?