To quote Dr. Reiss from the AES press release:
"“Audio purists and industry should welcome these findings,” said Reiss. “Our study finds high-resolution audio has a small but important advantage in its quality of reproduction over standard audio content. Trained listeners could distinguish between the two formats around sixty percent of the time.”

52.3% would qualify as statistically significant but of very limited practical relevance; 60% is usually considered to be of practical relevance.

<snip>
But this is really just the type of hair-splitting argument which you are so fond of.

No, even when reading the entire press release the message remains the same: This is the study the industry and audiophiles was waiting for, and it confirms their position. The reception of the press release I have seen across the internet picks up this message almost invariably. That's not their fault or distortion, that's exactly the gist of Reiss' message. It is not the result of his study, however.

If, as is often the case, one faces the choice whether to assume malice or incompetence as the reason for an act, the saying goes that one should choose incompetence. I don't think, however, that this is always the interpretation that represents the "better will".

I don't think that this distinction matters much in the context we're in here.

The discs used seem to have been examples of discs which audiophiles claimed to be audibly better than CD. The claims haven't been picked out of thin air by M&M.

And do I really have to rebuke your blatant exaggeration?

The problem appears at the very moment when significance is found.

Then, you would wish to know which particular aspect is responsible for the perceived difference. It is also of particular relevance in a meta analysis, because of their inherent sensitivity to "comparing apples with oranges".

Perhaps it surprises you (which I'm not sure of, your surprise may be a purely rethorical device), but I don't find reason for surprise here.

It is actually quite simple: If a study that tests all aspects (wordlength, samplerate) at once doesn't find any significance, none of the individual aspects have been shown to have significance.

It is still very unlikely that - had there really been clearly audible differences between original and CD downsampled version - they would have slipped through.

Besides, how do you check for "hi res ness" if people (claimants) have varying notions of what this means? Which definition should you pick? How would you test? M&M did the sensible thing: They avoided the question by taking material that was being presented to them as being hi res. The fact that some of it wasn't, according to some people's definition, shouldn't have prevented finding audibility at least with some of the material.

The accusation is in the sense unfair, that M&M are being made responsible for something they are not responsible for, namely the vague definition of what constitutes high res.

If the equipment "fault" you criticise should be the small linearity problem of one player that was discernible with one disk, I remind you to consider how and how much that could have compromised the result. I would not even go as far as calling this a fault, owing to its small scale.

Using this as a pretext for dismissing the study is completely out of proportion, IMHO. I much rather have the impression that M&M dutifully rectified the problem once they became aware of it. I trust they would have questioned the results of their own study if they had come to conclude that the fault was of sufficient magnitude to affect the result.

Which M&M did in the case of this player problem, at least as part of the supplementary information published on their website. Had they considered the problem relevant, I believe they would have described it in the paper itself.

I could feign surprise here that you don't object to Reiss' usage of a number of papers without even mentioning their flaws, but knowing you, I won't.

This description is just as selective as you accuse mine to be.

You typically made this look as if you were saying that some material may not offer such a perceivable difference, in other words you have no guarantee that every hi res file is superior to CD quality.

I mentioned this only because you brought up the topic.

I don't say that Reiss' paper is tainted by poor reviewing. It is not the reviewers who are responsible for Reiss' faults, and neither are they responsible for the faults in other papers. So why should I have mentioned it in the context of M&M, if the connection is coincidental rather than causal?

The problem is that such sloppy reviewing reduces the benefit of having a review in the first place. The designation of a paper as being "peer reviewed" may not mean much anymore.

I don't mix it up, I put it into perspective.

I am more critical of his usage of Kunchur's works as support for the suggestion that humans have a monaural temporal timing resolution of 5 µs. He uses language that keeps him neutral regarding these claims, but his presentation ignores all criticism that has been voiced. I don't think that's OK. Uncritical mentioning of dubious references increases their perceived credibility, without adding any argument or evidence in their favor.

Beside the central point that the two formats can be differentiated, further research is recommended (and needed).

That´s why i wrote "no thorough measurment" . I don´t know what other defects they might have not noticed
To not provide thorough measurements before and in between is the flaw.

Up to now i agree that the paragraph mentioning Krumbholz´s article was misleading/wrong, because it does not support Kunchur´s conclusion.

No they (M&M) didn´t issue a press release; instead they used sort of guerilla marketing in forums promoting their publication.

Seems that i´m not alone ..... 

Especially when reading the entire press release, the message is quite different. Beside the central point that the two formats can be differentiated, further research is recommended (and needed).

It would be just fair to adress the same "good will" to Reiss´s work.

I haven´t found any information if any of those "audiophiles claiming outrageous things" were participating in the tests.
Meyer&Moran confirmed the audiophile claims wrt to the sound quality of the "hi res material" (obviously they and most of the supporters of their experiment had no problems with this categorical statement although no controlled test was used) and the methodogical flaws prevent further conclusions.

Which in other words means that the validity of this research approach depends
on the result? Worth to reconsider ....

One should normally wish to know also in the case of a retained null hypothesis, especially if an experiments (like Meyer&Moran apparently did) compares "apples with oranges" too and lacks the advance of an extended data pool.

This conclusion is not warranted; it could be if all effects were balanced and if the sample size were sufficient, but both points were most probably not given.

As nobody noted the specific material used in the trials, how should one know? The number of trials per listeners was obviously to small, so missing a perceivable difference was quite likely.

The vague definition is not their fault, letting the vagueness influence the results.... is.....

Please keep in mind that your consideration is based on pure anecdotical description. That´s why i wrote "no thorough measurment" .
I don´t know what other defects they might have not noticed, beside the problem with low level linearity.

If you have anything comparable wrt papers Reiss used in his meta-analysis, please be specific.

Up to now i agree that the paragraph mentioning Krumbholz´s article was misleading/wrong, because it does not support Kunchur´s conclusion.

First, you haven´t so far presented any flaw in Reiss´s meta-analysis.

Second, you mentioned that the AES (Journal) has problems with the quality of the review process and you did it wrt to Reiss´s article. Maybe you meant it not exclusively; i could assume that the paper from Jackson et al. was also affected.

Kunchur got a lot of unfair criticism.

Afair i criticized his articles because the test results might have based on spectral cues (although below the usual limit) and because his conclusion about the need for higher sampling rates were not warranted. 

As Reiss was "neutral" in his language use, (he used "suggested" instead of "showed" ) that makes clear that research is still going on.

I've done many experiments related to multidimensional perception in controlled listening experiments, and if the difference is often easy to accurately identify.  Based on your former comments, I don't think you have any hands-on experience in this area, nor can you cite any relevant authoritative publications to support your claims.

Beside the central point that the two formats can be differentiated, further research is recommended (and needed).

<snip>

That's a good question. Are you just a believer, or do you have some skin in the Hi-Re$ game?

I'm down-grading my "layman's view" to complete waste of time just to keep this stuffed being talked about, which supports commercial interests."

Speaking of which, not that I am suggesting that this is one of those red-wine-is-good-for-you jobs, but was this work funded? Do people get paid for doing this stuff?

<snip>
And yet rather easy for the eager consumer or Stereophile reviewer to detect, eh? 

Quote

That is the reason why it is mandatory to incorporate positive controls (for other reasons negative controls as well), and any experimenter should think about listener accomodation and even training.

You're really not getting the point .

Be more pedantic why don't you?  OK then, the experiments that used ostensibly 'hi rez' sources and compared them to standard rate audio.

<snip>It's especially notable that the hi rez cheerleading side haven't provided any such experiments, given that they seized on Type II errors as their savior back in the day and will do so again per Reiss.

  But the again, the most vocal of them were also opposed to blind testing , period, for the longest time.  I'm sure now they'll be OK with it now though, since all (be sure to check me on that) of the work he cites used blind protocols of some type. .

(But then *again*, IIRC Reiss did mention the 'cognitive load' line with a straight face.....

-- an other effect that seems to operate only when doing comparison under experimental conditions, rather than when carefully auditioning a new SACD or DVDA or HDtracks download for review --  so you never know. )

Who cares?

M&M took these people at their word.  And *after* that, these people picked up the goalpost and marched downfield, proclaiming that only pure hi rez would suffice for proof, thanks very much

Quote from: ajinfla on 2016-07-17 12:44:36

That's a good question. Are you just a believer, or do you have some skin in the Hi-Re$ game?

In fact, neither.... nor. Was it again a rhetorical question?

If you are not able to check their (means your eager consuments or Stereophile reviewer) detection abilities within their normal listening routine, you have to consider that any test regime will have an impact on the listeners.

The ITU-R BS.1116-x emphasizes for good reasons listener training and the use of positive controls.

And I´m sure you have already noticed similar behaviour within the "non believer camp" ......

Please, reread the ITU-R BS.1116-x again to see what they had to say about accomodation and training.

Jakob2, I've asked multiple times, what positive control(s) must be used? Be specific (or do the dodge/evade/dance routine...which provides the answer).

I believe he's acting deliberately thick here.

He would never ever think of coming up with a test design of his own and put it into practice. He wouldn't be able to reach the bar he's putting up for others, and he would expose himself to the kind of criticism he directs at others.

Quote from: krabapple on 2016-07-17 22:38:03

Quote

(But then *again*, IIRC Reiss did mention the 'cognitive load' line with a straight face.....

To be fair- he just reported one concern from one of the references, did some research and reported that no such effect was shown by the data. Just exactly the correct procedure to handle something like that.

He failed to report the well-known confusion related to which of the extant two very different ABX tests were used (ABX1950 versus ABX1982) , and that comparing ABX tests to 1IFC tests makes no sense because ABX1982 tests can be and often are  performed as 1IFC tests.

To be fair- he just reported one concern from one of the references, did some research and reported that no such effect was shown by the data. Just exactly the correct procedure to handle something like that.

Authors have noted that ABX tests have a high cognitive load [11], which might lead to false negatives (Type II errors).

Quote from: Thad E Ginathom on 2016-07-16 07:24:56

I'm down-grading my "layman's view" to complete waste of time just to keep this stuffed being talked about, which supports commercial interests."

Isn´t that just what you want to believe? (for whatever reason)

Quote

Speaking of which, not that I am suggesting that this is one of those red-wine-is-good-for-you jobs, but was this work funded? Do people get paid for doing this stuff?

That could quite easy led to a thought-terminating cliché. I don´t know if Reiss´s meta-analysis was specially funded, but universities are encouraged to cooperate with the industry (if not hardpressed) and that is of course a two edged sword.
It may give reason for concerns but should not used to dismiss result that one doesn´t like.

Afair most of the technology (not only) in the audio field was/is invented due to strong financial interest. Did somebody say "CD", "MP3" "two channel stereo" ?

<snip>
If one understands how experiments and statistics work, one would not say that a test with 60% correct responses is the same as  "distinguish(-ing) between the two formats around sixty percent of the time.”

The first thing to remember is that in an experiment with 50% correct responses and 2 alternatives indicates 100% random guessing.

In contrast, an experiment with 100% correct responses indicates 0% random guessing.

The thing to take away from examination of these two facts is that there is a non-linear sliding scale of actual percentage of correct identifications ranging from 0% to 100% while distinguishing between the two alternatives from 50%  to 100% of the time.

Listeners who get correct responses only 60% of the time are mostly guessing randomly. Only 1 in 6 responses is anything but random guessing. They are actually giving correct responses only 16% of the time. The rest of the time they are guessing randomly.

If you try to apply these results to the real world, the results might be very depressing. For example If you buy 6 high resolution recordings, based on these statistics, only one of the recordings will sound different or they will all sound different only 1/6 of the time or something like that. I don't foresee anybody selling a lot of recordings based on performance like that.

Quote from: Arnold B. Krueger on 2016-07-18 13:37:50

<snip>
If one understands how experiments and statistics work, one would not say that a test with 60% correct responses is the same as  "distinguish(-ing) between the two formats around sixty percent of the time.”

The first thing to remember is that in an experiment with 50% correct responses and 2 alternatives indicates 100% random guessing.

In contrast, an experiment with 100% correct responses indicates 0% random guessing.

an experiment with 0% correct responses indicates 0% random guessing too.

<snip>
Those of us who do such tests find an interesting thing: there are differences so subtle that we're not sure we can hear them at all in normal sighted listening, but we do a controlled test, and statistically prove that we can hear them.

Army makes a good point. You should try it properly. One instance of having a "night and day" difference melt away, and one of having a "subtle" different statistically proven, is enough to open most people's minds to the interesting nature of human perception.

Not everyone really wants to learn something though.

<snip>
Straw man argument, since any 2AFC experiment with 0% correct responses isn't an experiment at all, it is a botched mess.

Time to diagnose and correct the experiment. 

Thinking that the results of botched messes must  have some higher meaning than human error is just wishful thinking. 
Of course since we're often dealing with placebophiles, and wishful thinking  is their spiritual guide,  GIGO.

In fact, neither.... nor. Was it again a rhetorical question?

<snip>
Yes, Delusion Blocked Testing will certainly impact the believer. Tremendously and often very humorously. Who is questioning this?

Jakob2, I've asked multiple times, what positive control(s) must be used? Be specific (or do the dodge/evade/dance routine...which provides the answer).

Quote from: 2Bdecided on 2016-07-21 00:31:59

<snip>
Those of us who do such tests find an interesting thing: there are differences so subtle that we're not sure we can hear them at all in normal sighted listening, but we do a controlled test, and statistically prove that we can hear them.

Which is correct but does in no way contradict what i have expressed.

Army makes a good point. You should try it properly. One instance of having a "night and day" difference melt away, and one of having a "subtle" different statistically proven, is enough to open most people's minds to the interesting nature of human perception.

To be honest, Arny makes his usual remarks.

In fact i´ve told him already (over at diyaudio) that i´ve started with controlled listening test back in the beginning (or mid) 80s after reading some articles from Dan Shanefield. His arguments were convincing and so it went along.

Being the Arny that he is, he commented that i was "late to the party" as his highness began a couple of years earlier.

<snip>
My usual remarks relate to the science of listening tests,.......

Quote

In fact i´ve told him already (over at diyaudio) that i´ve started with controlled listening test back in the beginning (or mid) 80s after reading some articles from Dan Shanefield. His arguments were convincing and so it went along.

Being the Arny that he is, he commented that i was "late to the party" as his highness began a couple of years earlier.

It's interesting to see the truth bent to fit into a personal agenda.  Fact is that the SMWTMS group was routinely doing DBTs in 1977 which is almost a decade earlier than the "mid 80s".

And still no to any financial interest (or admission of) in "hi res" .

Using eristics will not help in finding the truth, although it helps a lot in self- immunization

A difference which is known to be audible constitutes a positive control.