Reiss - these results imply that, though the effect is perhaps small and difficult to detect, the perceived fidelity of an audio recording and playback chain is affected by operating beyond conventional consumer oriented levels. Furthermore, though the causes are still unknown

He could have instead argued with the ~60% of trained listeners

No they didn´t issue a press release; instead they used sort of guerilla marketing in forums promoting their publication.

To quote Dr. Reiss:

In summary, these results imply that, though the effect is perhaps small and difficult to detect but important advantage, the perceived fidelity of an audio recording and playback chain is affected quality of reproduction over standard audio content by operating beyond conventional consumer oriented levels. Furthermore, though the causes are still unknown, Trained listeners could distinguish between the two formats around sixty percent of the time, this perceived effect advantage can be confirmed with a variety of statistical approaches and it can be greatly improved through training.

Quote from: Jakob1863 on 2016-07-11 09:16:57

If the CD-format is really transparent..

Which of the selected tests were for that question Jakob2 ?

why do you ask for selected tests?

Quote from: ajinfla on 2016-07-11 16:44:11

Quote from: Jakob1863 on 2016-07-11 09:16:57

If the CD-format is really transparent..

Which of the selected tests were for that question Jakob2 ?

It was Dr. Reiss´s starting point, so why do you ask for selected tests?

Quote from: Jakob1863 on 2016-07-12 16:51:19

[...] around sixty percent of the time.”

52.3% would qualify as statistically significant but of very limited practical relevance; 60% is usually considered to be of practical relevance.

This, I assume, mixes up two concepts. Statistical significance means that we have so many data points that we can be confident that these what-looks-like-sixty, is not fifty. (Oh, that was a rough one.)

And if the actual number is indeed 52.3, it does not mean that there is no practical relevance.

<snip> I would call that relevant in practice.

To quote Dr. Reiss:
"Yet many believe that this standard quality audio is sufficient
to capture all perceivable content from live sound.
This question of perception of high resolution audio has
generated heated debate for many years. Although there
have been many studies and formal arguments presented in
relation to this, there has yet to be a rigorous analysis of the literature. "

(Joshua D. Reiss; J. Audio Eng. Soc., Vol. 64, No. 6, 2016 June, page 364)

and
"Dr Reiss explained:  “One motivation for this research was that people in the audio community endlessly discuss whether the use of high resolution formats and equipment really make a difference. Conventional wisdom states that CD quality should be sufficient to capture everything we hear, ...."

(QMUL press release; http://www.qmul.ac.uk/media/news/items/se/178407.html)

To quote Dr. Reiss:
"Yet many believe that this standard quality audio is sufficient
to capture all perceivable content from live sound.
This question of perception of high resolution audio has
generated heated debate for many years. Although there
have been many studies and formal arguments presented in
relation to this, there has yet to be a rigorous analysis of the literature. "

(Joshua D. Reiss; J. Audio Eng. Soc., Vol. 64, No. 6, 2016 June, page 364)

and
"Dr Reiss explained:  “One motivation for this research was that people in the audio community endlessly discuss whether the use of high resolution formats and equipment really make a difference. Conventional wisdom states that CD quality should be sufficient to capture everything we hear, ...."

(QMUL press release; http://www.qmul.ac.uk/media/news/items/se/178407.html)

["Dr Reiss explained:  “One motivation for this research was that people in the audio community endlessly discuss whether the use of high resolution formats and equipment really make a difference. Conventional wisdom states that CD quality should be sufficient to capture everything we hear, ...."

To quote Dr. Reiss:
"Yet many believe that this standard quality audio is sufficient to capture all perceivable content from live sound.
This question of perception of high resolution audio has generated heated debate for many years. Although there
have been many studies and formal arguments presented in relation to this, there has yet to be a rigorous analysis of the literature. "
and
"Dr Reiss explained:  “One motivation for this research was that people in the audio community endlessly discuss whether the use of high resolution formats and equipment really make a difference. Conventional wisdom states that CD quality should be sufficient to capture everything we hear, ...."

I think not at the webpage krabapple linked, where he only expressed that 52,x % is to low to justify the markup. The difference to my question is obvious.

Nevertheless it is what i recommend.

No they didn´t issue a press release; instead they used sort of guerilla marketing in forums promoting their publication.

And they "oversold" (copyright pelmazo) their work right in the published article.
"Unlike the previous investigations, our tests were designed
to reveal any and all possible audible differences
between high-resolution and CD audio.... "
(E. Brad Meyer, David R. Moran; J. Audio Eng. Soc., Vol. 55, No. 9, 2007 September, page 776)

I can´t agree at this point (just from a semantic point of view, provided the statistical analysis holds true).
The AES press release mentioned in the second paragraph the novelty of the analysis approach (which it is afaik in the audio field), the numbers were reported correctly as in the universites press release as well. Of course "training improves dramatically" reads dramatic at first, but as the "dramatic" increase _to_ 60% is reported directly in the same sentence ....

No, they received also unfair criticism, as Reiss does (look just at the comments in this thread), which is sad, especially if it is called "critic based on science" , but a lot of critique was justified.

What you have stated above is a typical "post-hoc-hypothesis" based on the actual results (reminds to the texican sharp shooter fallacy) and it wasn´t what they pretended to do (see the quote above) and, due to the serious flaws, further conclusions are highly questionable.

As stated before, after reading the article i could not believe that it could have passed the review process and that fealing was emphasized after reading the supplementary information on the BAS website. No accurate measurements, detection of a broken player (not mentioned in the JAES afair) without knowing about the number of trials done in broken state, no tracking of the music used, different number of trials for the listeners, no information about the number of trials done on each location, no follow up although subgroup analysis showed some "suspicous" results and so on.

Don´t get me wrong- i have great respect for everybody doing this sort of experiment, because it is a lot of work, but otoh - given the fact that there is a plethora of literature covering DOE and sensory tests - i don´t understand why such simple errors, which could have been easily avoided, were still made.

Reiss actually did, what an author of such a study routinely does, he cites the literature and tries to find out if any criticism is backed up by the data. Reiss analysis did not show any significant impact of the test protocal and so he reported "if methodology has an impact, it is likely overshadowed by other differences between studies"
(Joshua D. Reiss; J. Audio Eng. Soc., Vol. 64, No. 6, 2016 June, page 372)

Is it what Reiss did or is it a strongly biased interpretation ?

Reiss actually did, what an author of such a study routinely does, he cites the literature and tries to find out if any criticism is backed up by the data.

<snip>
All perceivable content from live sound is not the same thing as sonic transparency. You can hear all perceivable content, but if it is colored in any way, the timing is off,, or there are additional signals that were not in the original sound,  transparent reproduction remains elusive.

<snip>
Nevermind the sad fact that the masters used for the "hi-res" versions are often tweaked compared to the masters used for CDs, giving an audible difference that people will attribute to the format. They could use the exact same master for both versions, but often they don't, either due to incompetence or malicious intent.

The only way to do a proper evaluation is to take a hi-res source and downsample it to CD quality yourself, so you can be sure of the provenance. Not very many people have the skills or inclination to do this.

But,if one compares and the "hi-res" version is better he might think the markup is justified.
If doesn´t help that the same quality could have been offerend on a CD, if that doesn´t happen. The customer is only able to choose what is offered.

If the "malicious intent" is indeed true that should be critized but please based on facts sampled during a serious investigation.

But,if one compares and the "hi-res" version is better he might think the markup is justified.
If doesn´t help that the same quality could have been offerend on a CD, if that doesn´t happen. The customer is only able to choose what is offered.

If the "malicious intent" is indeed true that should be critized but please based on facts sampled during a serious investigation.

No they didn´t issue a press release; instead they used sort of guerilla marketing in forums promoting their publication.

If the "malicious intent" is indeed true that should be critized but please based on facts sampled during a serious investigation.

<snip>
Archimago's question is: Is it worth it. You ask the same, and merely play with the actual numbers. If that is the obvious difference, fine.

That's looks more swanky when ripped out of context than in the paper. It is not difficult to figure out what they wanted to say if you read it with a slightly cooler mind.

A lot of criticism relies on selective reading, goalpost shifting, strawman attacks and the like. But that's worth a different topic and shouldn't be repeated here.

Their motivation and aim is quite clear when one reads their whole text instead of cherry picking convenient snippets.

They start with the claims of superiority of SADC and DVD-A over the CD .......

......and set out to check them with controlled blind tests.

Their aim is to cover the whole range of potential effects arising from the technical differences between the media, rather than focusing on a particular aspect like the wordlength. I don't see a problem with this, except perhaps with the exact wording in a few places.

Their method is largely appropriate for what they want to achieve.

Specifically, testing a medium through a direct path vs. the same medium through a restricted path is the most logical and straightforward way to do this, because it removes all factors that could be attributed to different source material. Moreover, it is simple enough to be used by ordinary people who want to know for themselves whether they hear a difference between CD format and higher formats.

And it is worth a reminder that the proper way of fixing the flaws in a test....

....is to run another test with the flaws fixed. In the ten years since the M&M tests were run, no audiophile interest group seems to have countered with anything even remotely appropriate. Isn't that telling something, too? Perhaps how cheap the criticism is in comparison to running a credible and convincing test? Who would be in a better position to do this than those who pretend to know what to listen for, what to listen with, how to test and how to evaluate it? Wouldn't that be much better than such a meta analysis?

Before foaming at the mouth over this, it might be worthwile checking what the potential of these alleged flaws was to corrupt the result. There's hardly a study that couldn't be criticised in a similar way if only put under a similar amount of scrutiny. You know this better than anyone else: I know your talent and resolve for years now at arguing the oxygen out of the air when you don't like the conclusion.

But I agree that the AES review process leaves something to be desired......

It shows in Reiss' paper, too.

Apart from the problems with the content and its interpretation, what caught my eye quite quickly was the inconsistent way of referencing literature. Some of it is in the traditional JAES style using reference numbers, some is using a name and year style as is more typical of textbooks. Surely this would have registered in a review, if I notice it within minutes?

It is especially difficult and laborious if you have to brace and defend your work against even the pettiest and defeatist objections that might appear.

He certainly appears to have done that very selectively. He references and uses quite a number of papers without taking notice of the sometimes very substantial critique and debate they have attracted. For example take the papers by Kunchur.

For some reason, the M&M study is the only exception, and here he shows not only that he is aware of the debate and where it took place, he also picks exclusively the negative points.

If he had done what you say in the usual impartial way, I wouldn't have a bone to pick. But alas, that's not how it turned out. It will be hard for him to shake off suspicions of bias this way.

based on his meta-analysis give some advice to achieve better quality in experiments.

Sorry, pelmazo, you mentioned a specific issue and i addressed that specific issue. Please don´t mix up this specific case with others.

If the HiRes version is indeed produced to higher standards, the markup may be justified. The same business model was already used many years ago with the CD, where you could sometimes buy improved versions for a markup (i.e. for a different mastering). This also shows that this possibility doesn't depend on a different format, improved versions can be provided independently from the format used.

The issue I have is that studios were previously willing to do things like record in DSD or 20+ bits and then sell me a nicely decimated and noise-shaped 16-bit rendition on actual physical media with a printed booklet for $10-15. Now they want to forgo the media and the printing and charge me $20-30 to get a master they haven't deliberately screwed up for whatever reason. Stereo "hi-res" products should arguably cost *less* than CDs did, but when people buy into the "bigger numbers sound better" then the opposite ends up true.

Archimago first asks if the difference is worth the markup; and secondly he asks, if 52.3% accuracy rate in a research setting sound like a valuable proposition to grab "hi-res" .
This i addressed (as said before, the consumer is only able to compare what is offered and to buy or not to buy), asking which "accuracy rate" would justify ......

The same holds true for the press releases for Reiss´s meta-analysis.

I don´t care so much about this topic if the same degree of "good will in interpretation" is applied in each case.

Last time you said they addressed "audiophile claims" but in fact, they didn´t really specify what their target was. At least did Meyer/Moran seperate recording engineers claiming superiority from audiophiles claiming "whatever still to specify" ....

And as usual if no research hypothesis is clearly specified the test regime (and level of control) isn´t as good as it should be.

That you don´t see a problem is a bit surprising, because you expressed strong concerns (even invalidity) wrt to Reiss´s meta-analysis as he did not focus on a "particular aspect like the wordlenght" ........

Notwithstanding that they (Meyer/Moran) combined wordlength and sampling frequency effects, they did not check if any "enhancement" was delivered at all.

It depends on the hypothesis; if it was the vague sort of "audiophile claims" that you mentioned the last time, it must have been a claim like "if a disc is labelled as hi-res it will under all circumstances , by all listeners at all times perceived as better as a downsampled to CD quality version" .

Because they did not check for "hi-res ness", they did not really check for the quality of reproduction, they did not provide positive controls, they did not really track which music was used in the trials and so on.
And they used mostly a 10 trials per listener approach, which is surprising because they should have known at least since Leventhals articles about power, that a small number of trials is accompanied by a large risk of error type II.

I did not criticize their "path choice" but that they have not provided thorough measurements before starting the experiment and checking routinely during the experiment.

Sorry, but first of all flaws must be mentioned

, because bad science is bad science. It does harm the reputation of science if these methodological flaws were belittled.
What you proposed would be a real "cheap get out of jail approach"

Mhm, further research is recommended? Me thinks Dr. Reiss did not just recommend further research, but could, based on his meta-analysis give some advice to achieve better quality in experiments.

What you forgot to mention is, that i did critisize experiments with the same scrutinity although i should like the results (according to your assumptions). And you should have mentioned, that i routinely recommended blind home listening experiments for any listener to learn about his perception and gave a lot of advice to improve the quality of listening experiments.

And you should have noticed, that i (nearly always) did emphasize beside all criticism that Meyer/Moran´s hypothesis ("hi-res" does not offer a perceivable difference or advantage compared to CD quality) might be true.

Something that you forgot to mention provided you liked the published findings (see for example Meyer/Moran, or did i miss it? ) ...

I´m sorry, but the flaws in Meyer/Moran´s experiment (further more if the additional information is considered) are evident just by reading, provided the reviewer has any expience in the DOE. No complicated analysis was needed to realise that.
In the case of Reiss´s analysis a reviewer must imho do a lot of work to find something.

Do we really qualify questions of style as equally important as methodological flaws??

You mean something like "he shouldn´t have used important, but significant instead" or "his press release overemphasized this or that", or he (maybe) didn´t notice that others criticized an older variant of "ABX"?
Isn´t the fact, that we in this thread mainly discuss semantics instead of real errors in the analysis telling?

One poster even mentioned Leventhal´s publication in the JAES because he felt that Dr. Reiss might have given rise to the impression to be the first one to talk about the importance of Type II Errors !?

Sorry, pelmazo, you mentioned a specific issue and i addressed that specific issue. Please don´t mix up this specific case with others.

Shouldn´t you complain instead that he was unfair to Kunchur? Without further discussion he excluded his test results from his meta-analysis.

He explained why Meyer/Moran got detailed remarks and why their result couldn´t be used generally but only in parts of the analysis. Nothing wrong with that and he even find encouraging words (at least in my opinion) in stating:
"However, their experiment
was intended to be close to a typical listening experience
on a home entertainment system, and one could argue
that these same issues may be present in such conditions."

May be you just want him to be biased.
He could have express quite stronger critique, but didn´t and you should be able to precisely argue where his reasoning to not include their results is wrong.

The issue I have is that studios were previously willing to do things like record in DSD or 20+ bits and then sell me a nicely decimated and noise-shaped 16-bit rendition on actual physical media with a printed booklet for $10-15. Now they want to forgo the media and the printing and charge me $20-30 to get a master they haven't deliberately screwed up for whatever reason. Stereo "hi-res" products should arguably cost *less* than CDs did, but when people buy into the "bigger numbers sound better" then the opposite ends up true.