Quote from: krabapple on 2016-01-03 00:21:48

From M&M's report, it seems they ran 554 trials, apparently(?) in blocks of 10 trials per test (which doesn't divide evenly, I know...so maybe some subjects bailed during some tests)

According to the paper, 2 subjects out of 55 achieved 7/10 correct, 1 subject achieved 8/10 correct. Those were the best scores.

A score of 7/10 has a p=0.172, indicates that in 100 such tests, we 'expect' that  'success' rate to occur ~17 times 'by chance'.

A score of 7/10 should have a probability of nchoosek(10,7) x 0.5^7 x 0.5^(10 - 7) ~ 11.72%, not 17.2%. And this is not the p-value, by the way. For a one-sided hypothesis test, you would add up the probabilities of 7/10, 8/10, etc.

Once we have 55 subjects (each of which went through 10 trials), the distribution of outcomes is approximately normal and you use the normal (or Student's t) to do hypothesis tests.

This Pioneer N-50 they use may have some non neutral behaviour with 48kHz 44.1 response

Quote from: Wombat on 2015-12-31 20:10:44

This Pioneer N-50 they use may have some non neutral behaviour with 48kHz 44.1 response

The measurement you link to is suspect. From other sources, e.g. Hifi News, we can see from their measurements that at 48kHz and with the standard filter enabled the response is flat to 20kHz within + and - 0.04dB.

This is as good as it gets, so no problem there. I haven't found this lately.

First of all, I'm not a huge fan of how poor the research standards of studies submitted to the AES.

Setting aside the issue of sample size and poor performance of ABX protocols for audio testing, we have to look at something more fundamental -- the unfortunate lack of equipment that actually rises to the needs of high-resolution audio in the online community that muddies the issue.

poor performance of ABX protocols for audio testing

the unfortunate lack of equipment that actually rises to the needs of high-resolution audio in the online community that muddies the issue.

the unfortunate lack of equipment that actually rises to the needs of high-resolution audio in the online community

Quote from: GUTB on 2016-03-11 04:06:48

poor performance of ABX protocols for audio testing

When you say this, are you referring to the ABX protocols themselves, which have been proven to be satisfactory with proper testing procedures and adequate sample sizes? Or are you referring to the researchers' ability to adhere to protocol?

Seeing as it is completely impossible to reach 24 bits of dynamic range in the real world, this equipment will never exist.

Apparently GUTB has been drinking the Kool Aid.

I thought it was a well-known fact that ABX produces terrible results due to the nature of audio memory, psychological effects and brain chemistry? The terrible test designs seem really just like icing on the cake.

It's not a "claim", but rather a fact, isn't it? ABX testing has an absolutely abysmal track record -- not just because of poor testing procedures, but mainly because the human auditory system is so unreliable. That is why the only way to reliably identify a subjective aspect of audio quality is in a relaxed environment in which the sound of the material and the equipment is well-understood by and ingrained into the listener; the very first exposure to a change represents the most pronounced perception of a difference, and further listening with A/B comparisons is needed to target and describe small changes. The brain will begin smearing differences, filling in "gaps", and eventually make even large differences diminish. Throw in the stress of an ABX test just poisons the test further, and making participants have to identify specific pieces of gear among several is just begging for a null result.

To sum up, ABX testing is bad because it fails to accurately gauge differences perceived by listeners under normal listening conditions. It's basically worthless.

To address the question of high-resolution audio. Go to the 2L website and download a comparison, they have a bunch of them up. If you can't tell the difference between a 16/44.1 and, say, a DSD-256 version of the same DXD master -- I honestly don't know what to tell you. It's pretty obvious to me, even though my DAC doesn't support DSD decoding and my software has to convert it to 32/192. Well, maybe I DO have an idea -- it could be the difference won't show up on cheap hardware that might be the norm here (I'm listening through a TH900). I'll pull out my HD668B, and maybe use cheaper components in the cahin and see if the differences are noticeable.

It's not a "claim", but rather a fact, isn't it? ABX testing has an absolutely abysmal track record -- not just because of poor testing procedures, but mainly because the human auditory system is so unreliable.

That is why the only way to reliably identify a subjective aspect of audio quality is in a relaxed environment in which the sound of the material and the equipment is well-understood by and ingrained into the listener; the very first exposure to a change represents the most pronounced perception of a difference, and further listening with A/B comparisons is needed to target and describe small changes. The brain will begin smearing differences, filling in "gaps", and eventually make even large differences diminish. Throw in the stress of an ABX test just poisons the test further, and making participants have to identify specific pieces of gear among several is just begging for a null result.

To sum up, ABX testing is bad because it fails to accurately gauge differences perceived by listeners under normal listening conditions. It's basically worthless.

To address the question of high-resolution audio. Go to the 2L website and download a comparison, they have a bunch of them up. If you can't tell the difference between a 16/44.1 and, say, a DSD-256 version of the same DXD master -- I honestly don't know what to tell you.

[...]