HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: Axon on 2007-09-10 04:11:51

Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Axon on 2007-09-10 04:11:51
"Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback". E. Brad Meyer and David R. Moran. JAES 55(9) September 2007. It's worth noting that members of the BAS wrote the paper and performed in the tests. You'll need an AES membership to access the article, so no link.

Abstract:
Quote
Claims both published and anecdotal are regularly made for audibly superior sound quality for two-channel audio encoded with longer word lengths and/or at higher sampling rates than the 16-bit/44.1-kHz CD standard. The authors report on a series of double-blind tests comparing the analog output of high-resolution players playing high-resolution recordings with the same signal passed through a 16-bit/44.1-kHz "bottleneck." The tests were conducted for over a year using different systems and a variety of subjects. The systems included expensive professional monitors and one high-end system with electrostatic loudspeakers and expensive components and cables. The subjects included professional recording engineers, students in a university recording program, and dedicated audiophiles. The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: PoisonDan on 2007-09-10 07:08:39
Interesting, thanks!
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: hushypushy on 2007-09-10 07:14:23
Very interesting...people are still going to claim they hear a difference anyway, though...
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Kees de Visser on 2007-09-10 12:16:24
Thanks for the info. Any details about the A/D/A loop ?
Discussing a publication based only on the abstract can be quite dangerous. However, complete AES articles are only available to members, or can be bought by non-members for $ 20 each. It is not allowed to share copies of AES documents with others. This makes public discussion rather difficult, although "It is permitted to quote from this Journal with customary credit to the source."
Interesting papers like this make me consider renewing my AES membership to have access to the online JAES and their On-line Electronic Library, despite the restrictions
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: LANjackal on 2007-09-10 13:18:36
Thanks for this
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Bad Monkey on 2007-09-10 13:48:40
The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

Sorry, does this mean they were testing whether this noise from the ADA conversion could be heard or not, or if there was an actual difference in sound quality between the hi-res and lowered-res sound? I.e. maybe there was a detectable difference in SQ but the question asked of the subjects explicitly referred to that noise only?

I'm confused.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: kdo on 2007-09-10 13:58:29
Can smb remind me please if there was any AES article that would prove audibility of lowpass higher than 16 khz in music?  Is there a consensus in the AES circles on this matter?

(The question is not directly related to the paper in the OP. Sorry)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: hushypushy on 2007-09-10 17:15:46
The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

Sorry, does this mean they were testing whether this noise from the ADA conversion could be heard or not, or if there was an actual difference in sound quality between the hi-res and lowered-res sound? I.e. maybe there was a detectable difference in SQ but the question asked of the subjects explicitly referred to that noise only?

I'm confused.


It seems to me like they played hi-rez content downsampled to 16/44. That would be the fairest comparison, the high-rez master and the 16/44 version made directly from it.

And no one could tell the difference, what a huge surprise.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Madman1153 on 2007-09-10 17:57:40
The title as well as the Abstract of this article seem only to refer to noise levels.  No one should be making any inferences to sound quality.  The conclusion of the article is that no one can distinguish a -96dB from a -120dB noise floor except at very high levers,  which everyone in this forum knows already.

Manuel
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Bad Monkey on 2007-09-10 17:59:48
@ hushypushy:
You've rush-read my post and haven't answered my query.

I suspect only the OP can, or anyone else who's read the paper, unless I'm being completely dim.

To clarify, I'm unsure how to interpret the abstract as quoted, because it only talks about detecting the presence of the ADA loop, not general sound quality.

@ madman:
Okay.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: ExUser on 2007-09-10 18:47:18
Did they noise shape the 16-bit audio? Noise shaping, as we know, decreases the perceptual noise floor. This should improve the perceived quality of the 16/44 content.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: AndyH-ha on 2007-09-10 19:34:44
The several comments about lack of inferences on sound quality do not fit either the title or the quote. It says no one can tell whether straight SACD or resample audio is being played. When no one can correctly guess whether it is  SACD or resample to CD spec audio, the only rational conclusion is that there is no audible difference in sound quality.

It says there is an audible difference is noise, but only when the volume is turned up very high. The only thing really unclear is “what noise?” Is it the CD noise floor, which cannot be heard in well mastered recordings except during very low level passages (played very loudly) , or is there some unusual (not to be expected in a normal CD) noise from their 16-bit/44.1-kHz "bottleneck."?
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Axon on 2007-09-10 20:37:29
Sorry, I haven't had a whole lot of time to summarize the paper. I just wanted to be the first one to get the scoop

The paper is rather light on details of the setup. They don't list the speakers they use, or the DAC/ADC they use, or the analog gain stage they use to match levels. That's pretty disconcerting, but I'm unsure exactly how much that matters, given how many people were involved in testing. (They also tested at several different venues.) I'd suspect that nobody would be placated if they were more detailed in their equipment descriptions. They used an ABX CS-5 for switching. They tested the quantization level of 16/44 by ABXing the system with no music playing at +14db above the normal listening volume. When playing real music at that level, they described it as "unpleasantly (often unbearably) loud".

The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.

In summary, I don't think this test will convince any audiophiles about the uselessness of high res. Some people will always come up with a bullshit excuse to believe what they want to believe, and there are a lot of people who have blind faith in SACD/DVD-A. But it could give producers more leeway to release well-mastered stuff on CD instead of more obscure formats. Or it could convince more people to get high res players to listen to better mastered stuff. Either way, listeners win.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: AndyH-ha on 2007-09-11 00:03:55
They tested the quantization level of 16/44 by ABXing the system with no music playing
Was this in addition to ABX tests on the music, or were they really just testing what people thought about the noise? Why would they bother with the noise? It is easy to demonstrate the difference between quantization noise under those conditions (no music, high volume level) and no quantization noise because of good noise shaped dither, but who is going to avoid dithering during mastering anyway? i.e. what is the point of the test? Is there any way it relates to differences between SACD and CD, or were they just exploring a difference from CDs and their setup i.e. they didn't (noise shape) dither the 16/44.1 reduced version?


the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons
This is still not clear as far as these tests go. The quote you provided above seemed to say that when this SACD mastered music was "passed through a 16-bit/44.1-kHz "bottleneck."" it could not be distinguished from the original.

If so, then the tests say there is nothing at all about SACD per se that is better -- except the mastering standards, which are quite irrelevant to the tests. Those differences from CD are arbitrary processing decisions and have nothing per se to do with the different technologies. If the mastering differences were not eliminated from the test, the tests would be as useless as those dual layer disks that are purposefully doctored to convince the `gullible that SACD is inherently better.

I think your statements are pretty clear, but I don't think everyone will take them the same way.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Kees de Visser on 2007-09-11 06:33:40
OK, I finally got the paper (written by the Boston Audio Society). Test setup was a hi-res media player (DVD-A and SACD), connected directly to the ABX switcher. The same player's output was connected to a 16/44.1 HHB (professional) cd-recorder (not listed, but clearly visible on a picture). Analog level matching was done only in the "lo-res" path, which was also fed to the ABX switcher. They did listen to music material:
Quote
Many types of music and voice signals were included in the sources, from classical (choral, chamber, piano, orchestral) to jazz, pop, and rock music. The subjects included men and women of widely varying ages, acuities, and levels of musical and audio experience; many were audio professionals or serious students of the art.
Quote
The test results for the detectability of the 16/44.1 loop on SACD/DVD-A playback were the same as chance: 49.82%. There were 554 trials and 276 correct answers. The sole exceptions were for the condition of no signal and high system gain, when the difference in noise floors of the two technologies, old and new, was readily audible.
The high system gain was about 14 dB. At that level, the noise difference became noticable, but music at the same play-back level was found to be "unpleasantly (often unbearably) loud".

ps:
Quote
...it is a rare playback venue that is quiet enough to reveal the 16-bit noise floor of our A/D/A loop—which has no noise shaping and was therefore less than optimal in this regard.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: greynol on 2007-09-11 06:53:11
Sounds like more than just quantization noise was introduced with their CDDA "bottleneck".  Because of this and beyond the fact that they jacked the volume to an unrealistic level to reveal the differences I'd say the evidence is even more damning that hi-res formats provide no tangible benefit when dealing with 2-channel audio.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2007-09-11 12:26:12
It implies the best and most convenient content available to consumers would be the SACD / DVD-A versions, recorded onto CD-R!

(because of the "well-known mastering reasons" - i.e. commercial CD releases are trashed with too much DRC, SACD/DVD-A sometimes are not)

I like the implication (which I agree with entirely) that you can make a better CD at home (using a good source, and a CD recorder's analogue input) than you can buy in a shop. It's quite sad.

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: user on 2007-09-11 13:53:24
I think, the result of this paper and tests are clear and not surprising due to previous experiences, knowledge and tests.

Maybe you wonder about their "final conclusion", that DVD-A/SACD offer quality benefits compared to corresponding CDs, due to mastering reasons.
Well, simple answer: Though the AES is more or less independent and working scientific,
it is with every science, science needs money from the industry (and vice versa the industry needs sometimes researches by scientists), so the performers of that study have printed this sentence to their paper, to have something compromising sounding towards their industry... (without changing facts, truth etc. !), though Joe Average and the mass-magazines might focus only on this sentence, and change it only a litlte bit to "DVD-A/SACD proven to be superior to CDs !"


Don't laugh, a writer for HiFi-magazines in Germany has done already this in past, he conducted (a very poor conducted) similar test, and though his results, which he published as some kind of diploma work, indicated that CD and high-rez are on par,
but he managed to cite his diploma thesis falsely in magazines he wrote later for, writing to the audiophile masses, "a" diploma work had shown, hi-rez is superior (without mentioning, it was his own crappy diploma work/thesis) lolololol.

In audiophile internet forums, this guy, his diploma thesis, and his magazine articles were torn "in the air", even by audiophile guys !!

At least, this episode has shown to me, that even "audiophiles" (nothing wrong with being audiophile, we all are!) know about the marketing crap written in certain HiFi-magazines, if they are informed eg. by internet. Not everybody buys, what's advertized in those magazines.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: dmckean on 2007-09-12 00:59:59
The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.


Theres really no need to move to an entirely new format though. Just more push for studios to release properly mastered CDs on labels like mofi or whatever.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: krabapple on 2007-09-13 17:27:04
The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.



Sorry, but hi rez is no guarantee of good mastering.  I know for a fact that there are DVD-As out there that have the highly restricted dynamic range of modern CDs (Yes 'Fragile', anyone?).  You're still taking your chances.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Axon on 2007-09-13 20:02:32
Sorry, but hi rez is no guarantee of good mastering.  I know for a fact that there are DVD-As out there that have the highly restricted dynamic range of modern CDs (Yes 'Fragile', anyone?).  You're still taking your chances.

Agreed, but right now, the chances are not so bad. It all depends on how many producers do wind up remastering. Allegedly most of them have been doing it, but that could change in the future.

Moreover, if everybody really believed that the higher res is the ultimate reason of improved quality, then the perceived benefit of improved mastering (which is the real reason for improved quality) could be drastically reduced.

The myth of high res quality directly hurts consumers by increasing the risk of a shoddy mastering product.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Axon on 2007-09-13 20:23:09
So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: krabapple on 2007-09-14 22:56:09
So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.
  • JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.
  • Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.
  • The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.
  • Musical selections are not listed.
  • Listeners are not described in terms of experience in any detail.


Too bad.  I guess we'll have to wait for Atkinson & Co to perform the slam-dunk scientific demonstration that hi-rez really audibly matters 
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: david moran on 2007-09-19 22:15:15
>>    * JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.

>>    * Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.

It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."


>>    * The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.

What could this mean? Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.


>>    * Musical selections are not listed.

Made no difference. This kind of thing is just to try to poke holes in an ironclad coin-flip result. We got a very wide range of material, types, instruments, vocals, genres, etc. Special stuff made for hi-rez demos, etc. etc.


>>    * Listeners are not described in terms of experience in any detail.

Not so --- just not enough detail for some who simply cannot believe the results.


David Moran
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: krabapple on 2007-09-19 23:52:24
>>    * JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.

>>    * Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.

It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."



The genuinely compelling reason for a detailed methods section in any scientific paper, is so the work can at least theoretically be reproduced.  And as you've seen, the subjectivist camp/Audio Asylum inmates will be all over you if you don't document everything scrupulously.



Quote
>>    * The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.

What could this mean?


It suggest you did not publish the Type I and Type II error statistics associated with your tests.  Most scientific papers that involve statistics at least include a Type I error (p value).


Quote
Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.


I'd be happy to buy it, but it appears to be not yet available for purchase.  I'd love to get a copy.

Quote
>>    * Musical selections are not listed.

Made no difference. This kind of thing is just to try to poke holes in an ironclad coin-flip result. We got a very wide range of material, types, instruments, vocals, genres, etc. Special stuff made for hi-rez demos, etc. etc.



>>    * Listeners are not described in terms of experience in any detail.

Not so --- just not enough detail for some who simply cannot believe the results.


Believe it or now, most of us here are very much on your 'side'.  Hydrogenaudio even has advocacy of blind tests built into it Terms of Use. 

It's just regrettable if a paper on this perennial hot topic failed to provide extensive method detail, because every omission will be used against it. And thus a topic that should have been laid to rest years ago, will remain 'hot'.

Perhaps you could set up a 'supplementary materials and methods' webpage to provide those details?
This is common in journals where printed space is at a premium (e.g., Science, Nature).
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Axon on 2007-09-20 00:42:28
Thank you for replying (and hopefully you'll stick around!)

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.
Well, duh! Of course it's nitpicking. That point isn't evidence of anything actually being wrong with the experiment - but it seems to me like it could potentially obscure real issues. Or, at the very least, the information is useful if another blind test would be attempted, in order to meet or exceed the technical specifications of the original test. (I'm not saying that I'd be doing it, but again, nice to know.)

Quote
It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."
Why would you wait for a reason to arise before listing the gear? That isn't a rhetorical question, and I'm not trying to be confrontational. I just don't understand. A lot of the individual ABX tests conducted here have pretty detailed gear descriptions associated with them, so I figure an AES paper would do the same thing...

Quote
What could this mean? Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.
I already emailed you (and Mr. Meyer) 3 days ago, actually. And I posted here when I first saw the complete article in the online JAES.

You're allowed to drag me back to the woodshed and beat me for calling the statistics "shoddy", if you want. Heck, I think I may have the least statistics experience of anybody commenting in this thread. What I meant to say was that the paper violated my expectations of how much detail is supposed to exist in the results/analysis. Based on other blind tests I've read, I would have expected to see clearly defined null hypotheses, the full listener responses, numerical analyses thereof, etc. Maybe some lip service to estimating the proportion of discriminators (which, honestly, really ought to be 1 anyway - but still, it's a subject worthy of debate, and I'd figure it's worth commenting over).

The results as they stand in the paper seemed to breeze through a lot of this very quickly, which surprised me. That doesn't diminish the results any - they are shockingly conclusive. They're just, well, short. And I think many audiophiles are using that as an excuse to ignore them completely. (Granted, though, most of them would probably ignore the paper anyway.)

Quote
Not so --- just not enough detail for some who simply cannot believe the results.
Alas, you're confusing me for somebody who doesn't believe the results.

I really do. They match my personal evaluation of SACD/DVD-A, they match my understanding of audio engineering and psychoacoustics, and the obviously large amount of work that went into the setup and listening, combined with the large number of listeners, makes them extremely compelling. And as the rest of this thread indicates you'd be hard-pressed to find anybody else here who would not agree. Enough of us have done our own personal ABX tests to understand how much effort goes into them, and what kind of results one can draw from them.

But when such a compelling result is established, why be stingy with so much information? You state that the ball's now in the pro-highres camp to show any sort of proof of audibility, but it seems like if anybody actually attempted another test like this, they'd want to know all of these things, in order to try to strengthen the sensitivity of the test. You spend a considerable amount of effort in stating that so many of these factors were carefully eliminated with the best testing conditions, and yet you don't actually say how you do it. "Trust us" is not a valid argument!

Already a lot of audiophiles (not me!) are using the lack of information in this paper to justify some sort of AES cabal that does not actually engage in peer review (http://www.audioasylum.com/forums/prophead/messages/3/36664.html). Like krabapple said - why try and settle the matter for good, when you've thrown them such a juicy bone to chew on at the same time? Unless the paper was meant only for AES readers, not for audiophiles.

As far as I'm personally concerned, all of those issues are really of omission than of actual error. I just want to know the information. Or, I just want to know why it's not there. It's mainly because I'm curious. I'm not expecting to find any flaws in it. And I had the expectation that asking for information like this is an extremely reasonable request.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: dekkersj on 2007-09-20 20:58:43
I contacted the authors and they replied that they used most of the time a Yamaha DVD-S1500 (but not only that particular player) and the cd recorder was a HHB CDR-850.

Although it is a pity that they didn't specify exactly what they have done and so on, it is a strong indication that frequencies above the Nyquist frequency of the cd format are not needed in a strict sense. Also the increased resolution (dynamic range) of sacd or dvd-a is not a must have.

The burdon of proof is shifted I would say.

Regards,
Jacco
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: david moran on 2007-09-20 22:40:49
Okay. Thank you all for your thoughtful responses, and I apologize for misinterpreting or impugning anyone here. (Or indeed anywhere.)

Rightly or wrongly, the chief reason we did not provide details was to preclude, or try to preclude, being nitted to death over the details. Which surely would have happened. The hi-rez players sucked, your amp and preamp suck, your speakers suck, so of course you could not hear anything. Etc. etc. Quite apart from the other venues and the number of listeners and trials.

So, as I say, rightly or wrongly. Y'all will observe that none of the AESJ editors or reviewers felt the lack of specifics was of any import whatsoever. No one mentioned a thing, not even once. We were asked for some more breakout of the results, which we provided.

Nonetheless, since we are receiving almost exactly the same flak as if we had given all the specifics, I am sure at some point, soon, we will post it all somewhere, perhaps on the BAS website.

Thanks for your modulated reading and responding.

David Moran
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: dekkersj on 2007-09-20 22:49:00
Thanks David,

Good to talk to you this way 

I think you did a great job by doing this experiment and if it was really true that there is a big difference between 44k1/16bits and high resolution, this should have been the outcome of your work. If it is that hard to differentiate, one could ask themselves the question what good it is in the first place to make such a big fuss about high resolution formats for consumer applications.

Regards,
Jacco
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: krabapple on 2007-09-22 06:48:56
Okay. Thank you all for your thoughtful responses, and I apologize for misinterpreting or impugning anyone here. (Or indeed anywhere.)

Rightly or wrongly, the chief reason we did not provide details was to preclude, or try to preclude, being nitted to death over the details. Which surely would have happened. The hi-rez players sucked, your amp and preamp suck, your speakers suck, so of course you could not hear anything. Etc. etc. Quite apart from the other venues and the number of listeners and trials.

So, as I say, rightly or wrongly. Y'all will observe that none of the AESJ editors or reviewers felt the lack of specifics was of any import whatsoever. No one mentioned a thing, not even once. We were asked for some more breakout of the results, which we provided.

Nonetheless, since we are receiving almost exactly the same flak as if we had given all the specifics, I am sure at some point, soon, we will post it all somewhere, perhaps on the BAS website.

Thanks for your modulated reading and responding.

David Moran



Good job.  If you post the specific, then it's up to the naysayers to prove they weren't sufficient to the task.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Kees de Visser on 2007-09-22 08:52:02
So, as I say, rightly or wrongly. Y'all will observe that none of the AESJ editors or reviewers felt the lack of specifics was of any import whatsoever. No one mentioned a thing, not even once. We were asked for some more breakout of the results, which we provided.
IMHO it's a pity that the AES doesn't provide any standards or even guidelines regarding subjective listening tests. The AES technical committee on Perception and Subjective Evaluation of Audio Signals (PSEAS (http://www.aes.org/technical/pseas/)) doesn't seem to be of much help in this respect. Anecdotal evidence (http://en.wikipedia.org/wiki/Anecdotal_evidence) is still widely accepted in the pro audio community. A possible reason could be that most engineers don't know how to set up and conduct a proper listening test. An often used excuse is that it involves a huge budget and lots of time and therefore can only be done by large companies and institutions.
AFAIK Perceptual Coding is the only area where subjective evaluation is used methodically. This forum is a good example of that.
I do hope that the AES (or another organisation of equal importance) will provide standards or guidelines and make an effort in promoting them.

Kees de Visser
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: spockep on 2007-09-22 13:37:54
Okay. Thank you all for your thoughtful responses, and I apologize for misinterpreting or impugning anyone here. (Or indeed anywhere.)

Rightly or wrongly, the chief reason we did not provide details was to preclude, or try to preclude, being nitted to death over the details. Which surely would have happened. The hi-rez players sucked, your amp and preamp suck, your speakers suck, so of course you could not hear anything. Etc. etc. Quite apart from the other venues and the number of listeners and trials.

So, as I say, rightly or wrongly. Y'all will observe that none of the AESJ editors or reviewers felt the lack of specifics was of any import whatsoever. No one mentioned a thing, not even once. We were asked for some more breakout of the results, which we provided.

Nonetheless, since we are receiving almost exactly the same flak as if we had given all the specifics, I am sure at some point, soon, we will post it all somewhere, perhaps on the BAS website.

Thanks for your modulated reading and responding.

David Moran


Thanks for your well thought out response and attention to this issue.  Unfortunately the AESJ study would have been nitted to death on internet forums regardless.  It's best to take things written on forums with a grain of salt.  In the end the great majority of people believe what they want to believe.  I hope to see more well designed and performed studies in the future.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Axon on 2007-09-23 04:01:03
What they said.  Thanks for responding.

Clearly if there's a disagreement here, it's only in how much test detail in a paper is considered reasonable. And I can totally understand your position.

For everybody else here, Meyer has posted a number of extremely informative posts on AA (http://db.audioasylum.com/cgi/search.mpl?author=EBradMeyer&user_id=45230&forum=ALL&sortRank=None&sort=date&sortOrder=DESC), at least until the test details are released.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2007-09-24 11:39:23
For everybody else here, Meyer has posted a number of extremely informative posts on AA (http://db.audioasylum.com/cgi/search.mpl?author=EBradMeyer&user_id=45230&forum=ALL&sortRank=None&sort=date&sortOrder=DESC), at least until the test details are released.
I like those responses. It will be interesting to hear which recordings were considered "the best commercial recordings we've ever heard" - and how they could easily sound just as good on CD, but apparently don't (presumably due to mastering, or non-availability).

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: usernaim on 2012-05-28 18:43:26
I'm not an advocate of hi-rez, but I wanted to revisit this test in light of what we now know.

The simplest reason why the test would get a null result if if the hi-rez material were not actually.  And what we know now is that many commercial SACDs and recordings sold on HD Tracks (many dervied from SACD masters) are not high resolution.  Bruce at Puget Sound, who does a lot of the transfers, has said that they have to reject 30% of what is submitted by labels (now that they have been stung, they check every track but that was not always the case).  And many SACDs with spectral content above 24 kHz have merely been upsampled after previously being downsampled to 44 or 48 kHz. 

Looking at the list of sample tracks (http://www.bostonaudiosociety.org/explanation.htm), I know at least Steely Dan Gaucho is not a legit high res transfer.  Has anyone checked the spectrals on these samples? 

So, how many of these tracks aren't what they purport to be?  Who has the disks and can look at spectrals?

Patricia Barber – Nightclub (Mobile Fidelity UDSACD 2004)
Chesky: Various -- An Introduction to SACD (SACD204)
Chesky: Various -- Super Audio Collection & Professional Test Disc (CHDVD 171)
Stephen Hartke: Tituli/Cathedral in the Thrashing Rain; Hilliard Ensemble/Crockett (ECM New Series 1861, cat. no. 476 1155, SACD)
Bach Concertos: Perahia et al; Sony SACD
Mozart Piano Concertos: Perahia, Sony SACD
Kimber Kable: Purity, an Inspirational Collection SACD T Minus 5 Vocal Band, no cat. #
Tony Overwater: Op SACD (Turtle Records TRSA 0008)
McCoy Tyner Illuminati SACD (Telarc 63599)
Pink Floyd, Dark Side of the Moon SACD (Capitol/EMI 82136)
Steely Dan, Gaucho, Geffen SACD
Alan Parsons, I, Robot DVD-A (Chesky CHDD 2003)
BSO, Saint-Saens, Organ Symphony SACD (RCA 82876-61387-2 RE1)
Carlos Heredia, Gypsy Flamenco SACD (Chesky SACD266)
Shakespeare in Song, Phoenix Bach Choir, Bruffy, SACD (Chandos CHSA 5031)
Livingston Taylor, Ink SACD (Chesky SACD253)
The Persuasions, The Persuasions Sing the Beatles, SACD (Chesky SACD244)
Steely Dan, Two Against Nature, DVD-A (24,96) Giant Records 9 24719-9
McCoy Tyner with Stanley Clark and Al Foster, Telarc SACD 3488
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2012-05-29 15:37:07
I'm not an advocate of hi-rez, but I wanted to revisit this test in light of what we now know.

The simplest reason why the test would get a null result if if the hi-rez material were not actually.  And what we know now is that many commercial SACDs and recordings sold on HD Tracks (many dervied from SACD masters) are not high resolution.  Bruce at Puget Sound, who does a lot of the transfers, has said that they have to reject 30% of what is submitted by labels (now that they have been stung, they check every track but that was not always the case).  And many SACDs with spectral content above 24 kHz have merely been upsampled after previously being downsampled to 44 or 48 kHz. 

Looking at the list of sample tracks (http://www.bostonaudiosociety.org/explanation.htm), I know at least Steely Dan Gaucho is not a legit high res transfer.  Has anyone checked the spectrals on these samples? 

So, how many of these tracks aren't what they purport to be?  Who has the disks and can look at spectrals?


A little quickie searching of Wikipedia and Amazon comes up with the following release dates:

Patricia Barber – Nightclub (Mobile Fidelity UDSACD 2004) Original Release Date: September 26, 2000

Chesky: Various -- An Introduction to SACD (SACD204)  needs track by track analysis

Chesky: Various -- Super Audio Collection & Professional Test Disc (CHDVD 171) needs track by track analysis

Stephen Hartke: Tituli/Cathedral in the Thrashing Rain; Hilliard Ensemble/Crockett
(ECM New Series 1861, cat. no. 476 1155, SACD) Audio CD (November 18, 2008)

Bach Concertos: Perahia et al; Sony SACD  Audio CD (March 12, 2002)

Mozart Piano Concertos: Perahia, Sony SACD Audio CD (October 25, 1990)

Kimber Kable: Purity, an Inspirational Collection SACD T Minus 5 Vocal Band, no cat.  no audio CD equivalent found

Tony Overwater: Op SACD (Turtle Records TRSA 0008) Audio CD (March 18, 2008)


McCoy Tyner Illuminati SACD (Telarc 63599) Audio CD (June 22, 2004)


Pink Floyd, Dark Side of the Moon SACD (Capitol/EMI 82136)  Audio CD  (March 1973)

Steely Dan, Gaucho, Geffen SACD Audio CD  (1980)


Alan Parsons, I, Robot DVD-A (Chesky CHDD 2003) Audio CD  (1977)


BSO, Saint-Saens, Organ Symphony SACD (RCA 82876-61387-2 RE1) Audio CD (July 5, 1991)


Carlos Heredia, Gypsy Flamenco SACD (Chesky SACD266) Audio CD (February 15, 1996)


Shakespeare in Song, Phoenix Bach Choir, Bruffy, SACD (Chandos CHSA 5031) Audio CD (September 21, 2004)

Livingston Taylor, Ink SACD (Chesky SACD253) Audio CD (September 23, 1997)

The Persuasions, The Persuasions Sing the Beatles, SACD (Chesky SACD244) Audio CD (February 26, 2002)

Steely Dan, Two Against Nature, DVD-A (24,96) Giant Records 9 24719-9 Audio CD (May 2, 2006)

McCoy Tyner with Stanley Clark and Al Foster, Telarc SACD 3488 Audio CD (January 25, 2000)

I would suggest that anything released prior to 1997 would have been originally tracked, mixed and/or mastered in what we would call now a legacy format, either 15 ips analog tape or 44-48-50 KHz sampled digital. 

This suggests that 7 of the  20 or so total sources are guaranteed to give random guessing in the tests they did.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2012-05-29 17:47:18
Chesky: Various -- Super Audio Collection & Professional Test Disc (CHDVD 171) needs track by track analysis
At least some of this is from 24/96 masters.

Quote
I would suggest that anything released prior to 1997 would have been originally tracked, mixed and/or mastered in what we would call now a legacy format, either 15 ips analog tape or 44-48-50 KHz sampled digital. 

This suggests that 7 of the  20 or so total sources are guaranteed to give random guessing in the tests they did.

Oh come on Arny  - the whole point of Sony inventing DSD was to capture 15ips analogue tape in the best quality possible at the time. That's how it was presented at the AES: the best possible digital archive format for the Columbia archive they'd just bought.

Fans are/were forever wetting themselves about how these older recordings sound so much better on hi-res than CD.

The idea that 15ips analogue recordings don't sound better on hi-res than CD is only mentioned to try to debunk Mayer and Moran. Outside this specific purpose, fans of hi-res seem convinced that everything (http://www.hydrogenaudio.org/forums/index.php?showtopic=94162) sounds better in hi-res than CD.

(but then, I know you know all this  )

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2012-05-29 21:10:01

This suggests that 7 of the  20 or so total sources are guaranteed to give random guessing in the tests they did.


Oh come on Arny  - the whole point of Sony inventing DSD was to capture 15ips analogue tape in the best quality possible at the time. That's how it was presented at the AES: the best possible digital archive format for the Columbia archive they'd just bought.


"The best quality possible" is not the same as passing a DBT.  Objectively speaking, in terms of pure technology, 24/96 and 24/192 are *better* formats for archiving *anything* than mere 16/44. The life's lesson to learn is that "Objectively better" does not always translate into "Sonically better".  I can live with that!

Quote
Fans are/were forever wetting themselves about how these older recordings sound so much better on hi-res than CD.


"Sounding better"  has as its prerequisite "Sounding different"  There are very many things that objectively measure different that totally and utterly fail to sound different. This has been true ever since the HP 330 THD analyzer replaced the 300 (1958). ;-)

Quote
The idea that 15ips analogue recordings don't sound better on hi-res than CD is only mentioned to try to debunk Mayer and Moran. Outside this specific purpose, fans of hi-res seem convinced that everything (http://www.hydrogenaudio.org/forums/index.php?showtopic=94162) sounds better in hi-res than CD.

(but then, I know you know all this  )


Again, they are confused and conflate measuring different with sounding different. ;-)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2012-05-31 10:17:02
Arny, I'm on your side. The point is they claimed (still claim) hi-res copies of old recordings sound better than CDs. But then "excuse" the failed ABX by saying that there's no content in these old recordings beyond what can be stored on CDs.

It's a pointless argument anyway, because there were enough genuine hi-res recordings in that list (at least, recordings with real high frequency content above 22kHz) to allow people to ABX those if they were in any way special / sonically above what CD can achieve.

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2012-05-31 12:45:42
Arny, I'm on your side. The point is they claimed (still claim) hi-res copies of old recordings sound better than CDs.


Not only were they on high rez media, but they were often remastered.  This business about remastering making recordings sound better is probably more a matter of taste than accuracy. Give the same master to 3 different mastering engineers, and you will probably get 3 different sounding recordings.  Given them to someone who has an equalizer in their system and the number of reasonable permutations probably approaches infinity. ;-)

Quote
But then "excuse" the failed ABX by saying that there's no content in these old recordings beyond what can be stored on CDs.


The failed ABX did avoid conflating remastering with hi-rez. Excusing something is subjective. Whether or not downsampling to CD format is audible is a scientific fact.

Quote
It's a pointless argument anyway, because there were enough genuine hi-res recordings in that list (at least, recordings with real high frequency content above 22kHz) to allow people to ABX those if they were in any way special / sonically above what CD can achieve.


I think the most telling fact relates to how apparently all of the high end reviewers failed to detect the clear lack of resolution in the nearly 50% of all of the so-called hi rez recordings that were released. They can't argue that the blind testing procedure got in their way!
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: SoNic67 on 2012-06-01 00:08:47
I am a fan of SACD recordings. Not necesarelly because of the theoretic capability but because the fact that remasterings for this media are less prone to be compressed to coply with "loudness wars". They are recordings dedicated for the niche represented of people that can turn up the volume by hand, don't listen those discs in boomboxes or car subwoofers...

Anyway, I did test some SACD recordings of old masters (on analog domain) and they barelly pass the original CD bandwidth. This is because of poor quality of the originals ('70's, '80's).
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2012-06-01 11:40:18
Anyway, I did test some SACD recordings of old masters (on analog domain) and they barelly pass the original CD bandwidth. This is because of poor quality of the originals ('70's, '80's).
You can't really say "poor quality of the originals". You can say"restricted bandwidth of the originals". But given that the bandwidth is still more than most human ears appear to be able to detect, the quality (in that respect) is more than fine.

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2012-06-01 12:01:21
Anyway, I did test some SACD recordings of old masters (on analog domain) and they barelly pass the original CD bandwidth. This is because of poor quality of the originals ('70's, '80's).


Interesting. In the 70s and 80s it was possible to do recordings with 22-24 Khz bandwidth, sometimes up to 30 Khz. Your typical studio tape machine running at 15 or 30 ips could do that if it was properly aligned (IOW recorders aligned daily, and playack azimuth optimized for each tape played back).

Quote
You can't really say "poor quality of the originals". You can say"restricted bandwidth of the originals". But given that the bandwidth is still more than most human ears appear to be able to detect, the quality (in that respect) is more than fine.


Agreed.

Seems like a lot of people need to do DBTs of their favorite "HD" recordings brick-walled at 16-18 Khz and then upsampled to original format for a clean ABX. ;-) At worst they'd have to work harder than they expected  to hear a difference, and at best they'd be fooled. ;-)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: SoNic67 on 2012-06-02 00:56:59
You can say"restricted bandwidth of the originals". But given that the bandwidth is still more than most human ears appear to be able to detect, the quality (in that respect) is more than fine.
Cheers,
David.

Everyone tells me that I can't hear past 18 kHz. Good... But I can hear the differences between 14kHz and 18kHz. This is how bad it is for some old recordings.
I can hear on the CD "The Police - Reggata de Blanc" things (cymbals) that are definitely not present into any CD or SACD of Pink Floyd (TOSOTM).
Run the audio via Foobar one can see that the spectrum over 16kHz is very reduced as "weight" compared with "The Police" disc. I use the free VST plug in from BlueCat for better resolution .
I have many other examples... good and bad.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Wombat on 2012-06-03 00:11:15
Fans are/were forever wetting themselves about how these older recordings sound so much better on hi-res than CD.

The idea that 15ips analogue recordings don't sound better on hi-res than CD is only mentioned to try to debunk Mayer and Moran. Outside this specific purpose, fans of hi-res seem convinced that

Indeed! This Patricia Barber, Nightclub is well regarded for its really impressive quality.
You will find nearly on every review a remark about the better sounding SACD layer even if there is not anything above 22kHz because the original recording was done on an old digital machine. So if MOFI didn´t mess up the CD-Layer this one should have been a pro HiRes one even without extended HF content. I bet so it goes for other releases listed also.
So i don´t think the reasoning of BW limited sources count at all. It would call all further praising reviews and listening reports about these records pointless, so the complete debate.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2012-06-03 00:27:18
Quote
Everyone tells me that I can't hear past 18 kHz. Good... But I can hear the differences between 14kHz and 18kHz. This is how bad it is for some old recordings.
I can hear on the CD "The Police - Reggata de Blanc" things (cymbals) that are definitely not present into any CD or SACD of Pink Floyd (TOSOTM).
Run the audio via Foobar one can see that the spectrum over 16kHz is very reduced as "weight" compared with "The Police" disc. I use the free VST plug in from BlueCat for better resolution .
I have many other examples... good and bad.


Are you trying to make technical judgements about media formats or the audibility of different frequency renges based on two different songs recorded by two different groups?
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: bernlin2000 on 2012-12-20 09:44:25
"Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback". E. Brad Meyer and David R. Moran. JAES 55(9) September 2007. It's worth noting that members of the BAS wrote the paper and performed in the tests. You'll need an AES membership to access the article, so no link.

Abstract:
Quote
Claims both published and anecdotal are regularly made for audibly superior sound quality for two-channel audio encoded with longer word lengths and/or at higher sampling rates than the 16-bit/44.1-kHz CD standard. The authors report on a series of double-blind tests comparing the analog output of high-resolution players playing high-resolution recordings with the same signal passed through a 16-bit/44.1-kHz "bottleneck." The tests were conducted for over a year using different systems and a variety of subjects. The systems included expensive professional monitors and one high-end system with electrostatic loudspeakers and expensive components and cables. The subjects included professional recording engineers, students in a university recording program, and dedicated audiophiles. The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.



The quote of the century: I'll have to cite this anytime someone suggests 24-bit music is the future...this is one of those unusual areas where technology can do no more improvements (narrowly talking about sound bit rate here)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: bernlin2000 on 2012-12-20 09:50:00
Believe it or now, most of us here are very much on your 'side'.  Hydrogenaudio even has advocacy of blind tests built into it Terms of Use. 

It's just regrettable if a paper on this perennial hot topic failed to provide extensive method detail, because every omission will be used against it. And thus a topic that should have been laid to rest years ago, will remain 'hot'.

Perhaps you could set up a 'supplementary materials and methods' webpage to provide those details?
This is common in journals where printed space is at a premium (e.g., Science, Nature).


Yeah, and religion is still a "hot" item...people will  believe what they want, regardless of the facts. If people haven't already figured out that 24-bit sound is only useful for production, not for playback, than they likely never will. 44.1khz should be easy, though: we can't hear beyond 20khz, guys...
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: DonP on 2012-12-20 13:26:44
One argument I would consider is that if any component in the chain wouldn't pass the higher frequencies of the hirez signal, then THAT would prevent the very golden eared from ABXing the difference.  Getting good dispersion even up to 20 kHz is a notorious weak point for most speakers. 

Playing the music at borderline painful levels may be ok for validating the noise floor, but I'd think triggering the ear's internal AGC (tensing up the ear bones) could reduce high frequency hearing (in the moment; not referring to the permanent damage it does.)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: sawdin on 2013-01-06 18:58:11
If the BAS testing was so flawed, why haven't those who vociferously disagreed with the stated results stage 'better designed/implemented' ABX tests?  If they have, please provide links.  If they haven't, why not; Is it a question of expense and/or logistical concerns?

Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2013-01-07 14:51:03
If the BAS testing was so flawed, why haven't those who vociferously disagreed with the stated results stage 'better designed/implemented' ABX tests?  If they have, please provide links.  If they haven't, why not; Is it a question of expense and/or logistical concerns?


Most critics of Meyers and Moran also seem to have problems with ABX testing all by itself.  After all, it tends to not reinforce their prejudices! ;-)

The first thing that most golden ears do when trying to replicate tests with results they disagree with is *improve* on ABX.  The results are usually pretty funny (in a dark way) once the dust settles. Example that comes to mind: John Atkinson testing power amps for Stereophile with his own DBT procedure.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: drmoran@aol.com on 2013-01-20 00:25:10
If the BAS testing was so flawed, why haven't those who vociferously disagreed with the stated results stage 'better designed/implemented' ABX tests?  If they have, please provide links.  If they haven't, why not; Is it a question of expense and/or logistical concerns?


Most critics of Meyers and Moran also seem to have problems with ABX testing all by itself.  After all, it tends to not reinforce their prejudices! ;-)

The first thing that most golden ears do when trying to replicate tests with results they disagree with is *improve* on ABX.  The results are usually pretty funny (in a dark way) once the dust settles. Example that comes to mind: John Atkinson testing power amps for Stereophile with his own DBT procedure.



+++

Have not been here for a long while, but I hasten to add (in response to assertions elsewhere) that Bob Stuart was not one of the auditioners (that would've been awesome, since it was his AESJ article bogoclaims which prompted the test in the first place; if I won a big lottery I would have all sorts of trip-paid challenges to tweakos). And second, we were not 'misled' by hi-rez manufacturers, at least not directly; we simply invited all listeners to bring their own favorite hi-rez material, which we listed, and much of it was Chesky direct, and not ever remasters of old stuff. Anyone who wants to redo our work should use lots of SACD and beyond, ultrahi-rez stuff, to see if that gets debunked too (whaddaya bet?). Our point was to be ample and capacious enough in all respects to give every chance for listeners to prove they could hear the RBCD bottleneck when switched in blindly. This led to a certain amount of suboptimal statistical hygiene. As a trained statistician and Journal reviewer put it, 'Given that your test was designed to allow participants every opportunity to demonstrate their ability to discriminate between A and B, you were more concerned that you not get a false positive conclusion than that there not be a possibility of reaching a false negative conclusion.'
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: mzil on 2013-01-20 05:01:06
Dr. Moran, I think this test you used is brilliant! Thank you.

I pondered in another thread (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=91248&view=findpost&p=786212) that this same methodology would be an excellent way to battle LP against CD. As you know, all other attempts to AB(X) test the two is pointless because of obvious pops and ticks (audible at least during the quiet sections of even the best LPs) acting as "tells" or "giveaways", as well as the fact that they use different master recordings,  so it wouldn't be an apples to apples test even if there were no pops/ticks to contend with on the vinyl versions. Your test also conquers any synchronization problems since the LP and the A/D/A looped version from the CD recorder's output would always be matched, regardless of any speed drift with the turntable.

I don't know if you follow these things, but there is a huge contingent of young people who claim LPs are superior to CD, "As long as you can hear beyond the occasional pops and ticks" they typically claim. Sales for LPs are also at an all time high over the past 20 years (http://blog.dubspot.com/files/2012/11/music-sales-vinyl.png)!

Any chance we can see you apply this great A/D/A loop test of yours to vinyl records? Without your test there is no way to conclusively show the young folks that their preference for an LP over a CD is simply due to the different master recording that made it, or some LP distortion they "dig". [Oddly, for reasons that escape me, some recent LPs have a larger dynamic range to the CD version (due to the mastering), with some pop music, I'm told. This would also help expose that this fact is a mastering issue, and not inherent to the mediums themselves.]
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2013-01-22 18:02:13
Dr. Moran, I think this test you used is brilliant! Thank you.


Two words: Prior art.

The Meyer Moran test is basically a straight wire bypass test of the ADC/DAC combination they used to create the CD-response bandbass filter. 

Here's an example of its use from over 23 years ago:

http://bostonaudiosociety.org/bas_speaker/abx_testing.htm (http://bostonaudiosociety.org/bas_speaker/abx_testing.htm)

Straight wire bypass testing was old 23 years ago!


Quote
Any chance we can see you apply this great A/D/A loop test of yours to vinyl records? Without your test there is no way to conclusively show the young folks that their preference for an LP over a CD is simply due to the different master recording that made it, or some LP distortion they "dig". [Oddly, for reasons that escape me, some recent LPs have a larger dynamic range to the CD version (due to the mastering), with some pop music, I'm told. This would also help expose that this fact is a mastering issue, and not inherent to the mediums themselves.]


The test in question has been done many times. It is fairly well known that if you burn a CD of a LP transcription it sounds just like the digital file that was originally created by the transcription process.

Yes, many of the things that people say they prefer about LPs is mastering, and it is almost certain that the rest is simply hype.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: mzil on 2013-01-22 21:50:13
Dr. Moran, I think this test you used is brilliant! Thank you.


Two words: Prior art.

The Meyer Moran test is basically a straight wire bypass test of the ADC/DAC combination they used to create the CD-response bandbass filter. 

Here's an example of its use from over 23 years ago:

http://bostonaudiosociety.org/bas_speaker/abx_testing.htm (http://bostonaudiosociety.org/bas_speaker/abx_testing.htm)

Straight wire bypass testing was old 23 years ago!


Rather than linking to earlier discussions by Dr. Moran's co-author of this very AES paper, Brad Meyer [where Moran is also mentioned], I'd think some of the much earlier straight wire bypass tests of Peter Walker or David Hafler's somewhat similar, although not identical "straight wire differential test", discussed decades before the digital era, would have been more appropriate, but since neither I nor Dr. Moran made any claim the concept is original in the first place, I have no idea why you are even bringing up any "prior art" discussion.

Quote
The test in question has been done many times. It is fairly well known that if you burn a CD of a LP transcription it sounds just like...

Although Ive done the test myself decades ago, there has never been a scholarly paper written about it like Moran and Meyer did with this SACD test. That's what I was hoping for,  so I can refer to this future paper when my pro-LP friends give me their "analog is better if you can just listen beyond the pops and ticks " mumbo-jumbo.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: eretsua on 2013-01-23 02:22:40
The difference in sound between vinyl and CDs, I think, comes down to the mastering and not the media used.  So given they would use the same source a double blind AxB comparison may indeed prove there is no perceivable sound quality difference between the two.

However, I do think there is a difference in the real world in the experience people have with a CD vs Vinyl. Putting on a vinyl is a more conscious, deliberate act. It takes more effort to put on a vinyl than it is to do a few clicks on your digital playback device. Even popping in a CD is fast & simple compared to the 'elaborate hassle' you have to go through to put on a vinyl.

So I think that this ritual you go through helps put you in the mood for the material you are going to listen to. And since you've made an effort into putting it on you may also pay more attention to it. Generally having a better experience listening to the material. When just opening a play list it is easy to (mentally) just walk away from it.

But maybe more importantly, vinyl ends and you will have to physically (get up and) flip it. This takes time, it forces a break in the sound. Gives your mind and ears a rest for a moment. I think that is very important. 15 minutes of intense listening is quite fatiguing. Listening to the 50min album in a sitting on cd without a break probably more so. And having 8hour long play lists continuously bombarding at you... bet that forces people to tune out mentally. So these forced breaks probably help the overall listening experience.

All of that adds up and may help explain why people say vinyl sounds better than cd (other than the different mastering). Even if a direct comparison between digital vs analog would not reveal any sonic differences.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Engelsstaub on 2013-01-23 03:28:26
The difference in sound between vinyl and CDs, I think, comes down to the mastering and not the media used.  So given they would use the same source...


I believe that "mastering" usually isn't the best way to describe the differences heard between the audio put to modern CD vs. that sometimes found on modern vinyl. It seems to me that, when they differ, it is probably due to the same master being pushed destructively to a 0 dB "normalization" for the digital release. (There is also the RIAA equalization, but that isn't really relevant, IMO.)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: julf on 2013-01-23 07:33:39
I believe that "mastering" usually isn't the best way to describe the differences


"Amateuring"?
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: markanini on 2013-01-23 08:14:28
I believe that "mastering" usually isn't the best way to describe the differences


"Amateuring"?

Do you think Bob Katz is an amateur?
http://youtu.be/u9Fb3rWNWDA?t=7m58s (http://youtu.be/u9Fb3rWNWDA?t=7m58s)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: julf on 2013-01-23 09:54:46
Do you think Bob Katz is an amateur?
http://youtu.be/u9Fb3rWNWDA?t=7m58s (http://youtu.be/u9Fb3rWNWDA?t=7m58s)


I think any "engineer" that over-compresses and clips recordings is an amateur. I have no idea if Bob Katz is in that category or not.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: eretsua on 2013-01-23 10:27:14
Sigh, regardless of how well or how poorly anyone does their job if it is their main source of income they are a professional.
If their income isn't coming from mastering they are an amateur regardless of their level of skills and talent.

Bob Katz is a driving force behind the K-system metering system. A system that is essentially the tool (in the broadest sense of the word) for preserving dynamics.
He is both a talented professional as well a front runner when it comes to having dynamic music.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: julf on 2013-01-23 11:10:20
Sigh, regardless of how well or how poorly anyone does their job if it is their main source of income they are a professional.


Sure - so replace "amateuring" with a verb derived from some other opposite of "master".
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: markanini on 2013-01-24 05:40:34
That's like calling a prostitute a nun for having a holiday.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2013-01-24 18:45:33
The difference in sound between vinyl and CDs, I think, comes down to the mastering and not the media used.


Vinyl has enough built-in, irreducible audible artifacts that it can't pass a bypass test.  As a rule vinyl mastering is designed to make those audible artifacts as unnoticeable as possible.

1 generation of the best quality analog tape can't pass a bypass test either, but it is a tougher proposition than vinyl.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2013-01-25 10:14:10
No, I think he means use the vinyl as the source, and push it through 44.1/16, to show that 44.1/16 does not remove the "benefits" of vinyl.

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2013-01-25 12:17:37
No, I think he means use the vinyl as the source, and push it through 44.1/16, to show that 44.1/16 does not remove the "benefits" of vinyl.


Of course!

Interesting how you manage to make statements that agree with mine look like they are corrections!


Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: db1989 on 2013-01-25 12:50:38
Of course!

Interesting how you manage to make statements that agree with mine look like they are corrections!
I’m sorry, but I just totally read this in Bane’s voice, hahaha. The first line is what sells it, obviously, but the second one would also fit right in!
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2013-01-25 13:01:25
Interesting how you manage to make statements that agree with mine look like they are corrections!
Good grief 
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: dhromed on 2013-01-25 13:02:43
I’m sorry, but I just totally read this in Bane’s voice, hahaha.


That's what the B. stands for.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: db1989 on 2013-01-25 13:12:27
Oh lawd! Why didn’t I ever suspect before?

Completely unrelated, but I would just like to state for the record that I love strawberries (http://www.youtube.com/watch?v=IkMPZ7WeDck).
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: greynol on 2013-01-25 16:47:44
eretsua is suggesting that a CD using the same mastering as vinyl will sound exactly the same as the vinyl. The implication is the vinyl will faithfully produce every nuance of the master every bit as well as CD.  How this was missed and/or glossed-over by all but Arny is anyone's guess.

Anyhow I dont think it is in any way controversial to say that eretsua was dead-wrong.

Now one might suggest that he meant to say passing vinyl through CDDA is a transparent process, however that is not what he said (twice).  Why Arnold would agree with David with an "Of course!" seems a little strange. Maybe I'm wrong and no one got it right.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: eretsua on 2013-01-25 23:14:29
Greynol is correct. What I meant is that indeed that the audio is presented with the same mastering (targeted for the lowest common denominator) on the two different media.

Very short on time so my apologies if this is a bit real world inaccurate / I'm taking a few shortcuts. Unfortunately, I  don't have time to flesh it out right now so please try to think along in the spirit of what I'm saying rather than the letter of it.

Lets do a test.  I'm guessing that would be a vinyl-targeted-master played back on a cd. there would of course be obvious problems to over come like pop & crackles on the vinyl playback that aren't present in the cd presentation of the material. Yes, I am aware that influences the sound and experience but what I'm getting at lies beyond those obvious real world differences.  So lets for a moment to assume we can solve those / listen around them. Even it is just adding a simulated vinyl playback noises to the cd presentation. I think that if that playback "analog noise" has been solved you'd be hard pressed to hear to the differences between the two. If not, why not?

I totally agree that what I'm suggesting is more of a "which type of mastering style do you prefer?" kind of test rather than an actual analog vs digital test, yes. I am also well aware that in the real world the artifacts of the vinyl playback does have an influence on the reproduction of the material, yes. But I am not convinced that is the determining element in why some people prefer vinyl over cd.

So the point I'm getting at is this: I think it is the kind of mastering as they do for vinyl what the people really like, the people who prefer vinyl over cd, of course. Maybe it is that of the modern vinyl pressing have a greater dynamic range than their cd counterparts. Maybe it is the differences in the depth of the bass or some such. Maybe it is related to something else. But I don't think it is related to the physical medium itself and that if they are not visually exposed to the medium they'd still prefer the vinyl-style mastering over the mastering intended for cd reproduction. And I still stand by my idea that putting on a vinyl is more of an effort and that has a positive influence on the experience of music.







Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: greynol on 2013-01-25 23:42:42
We have a saying around here: you can make your CD sound exactly like your vinyl; good luck going the other way around (or something to that effect).

As was pointed out, vinyl has inherent problems preventing it from delivering a flat frequency response from DC to 20kHz with SNR and dynamic range of 96 dB (and even better when you play tricks) like you can with CDDA.  This goes well beyond clicks and pops and is well documented in our wiki and elsewhere.

Still, people are free to claim two things sound identical on this forum; even if there is evidence to prove them wrong.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: krabapple on 2013-01-26 16:48:36
So the point I'm getting at is this: I think it is the kind of mastering as they do for vinyl what the people really like, the people who prefer vinyl over cd, of course. Maybe it is that of the modern vinyl pressing have a greater dynamic range than their cd counterparts. Maybe it is the differences in the depth of the bass or some such. Maybe it is related to something else. But I don't think it is related to the physical medium itself and that if they are not visually exposed to the medium they'd still prefer the vinyl-style mastering over the mastering intended for cd reproduction. And I still stand by my idea that putting on a vinyl is more of an effort and that has a positive influence on the experience of music.


It's the mastering *and* the 'euphonic' distortion (and the packaging and the nostalgia and the ritual of LP care and playback, but let's stick to sound).

Leaving aside externally-induced noise (dust, scratches, warping):

Vinyl playback -- the technology itself, at its best -- inherently has less channel separation (this can be a 'euphonic' effect for some listeners) .  Vinyl playback -- the technology itself, at its best -- inherently has more noise (I would not think this likely to be 'euphonic', but who knows?  Audiophiles are a strange bunch) .  I'm sure I'm forgetting a few other inherents (Woodinville or Arny can fill them in -- IIRC there's another euphonic effect related to perceived dynamic range). 

These inherents would be true even if you managed to master the vinyl and CD exactly the same (good luck with loud bass signals on that endeavor) and played the LP on the very finest turntable/cart system.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2013-01-28 13:04:41
I'm totally lost.


Anyway, while I'm happy that I can record vinyl to CD-R without introducing any audible change, I also have a few recordings on vinyl and commercial CD that sound the same on both (i.e. no mastering differences and no audible (to me) vinyl flaws while the music is playing). Pop music, obviously. I've yet to hear this magical vinyl set-up that plays wide dynamic range classical music without audible flaws during the quiet passages.

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Nessuno on 2013-01-28 14:25:03
I've yet to hear this magical vinyl set-up that plays wide dynamic range classical music without audible flaws during the quiet passages.

Uh? Regarding classical, wider dynamic range is maybe the only advantage that even the most diehard vinylists acknowledge to digital.

Maybe I've already told it: I have a couple CDs of symphonic music from early '80 with a great red "WARNING" written on the first page of the booklet together with an advice not to turn up too much on quieter passages not to damage speakers on subsequent fortissimi...

Edit: for example this CD (http://www.eclassical.com/orchestras/gothenburg-symphony-orchestra/sibelius-symphony-no2.html).
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: Arnold B. Krueger on 2013-01-28 18:24:16
So the point I'm getting at is this: I think it is the kind of mastering as they do for vinyl what the people really like, the people who prefer vinyl over cd, of course.


More likely, vinyl sells to people for whom it is new and its warts seem like beauty marks, and to people for whom it is old and its warts seems like beauty marks.

Quote
Maybe it is that of the modern vinyl pressing have a greater dynamic range than their cd counterparts.


Not a chance. They made the mistake of releasing test records on "modern vinyl pressings" so that technical types can measure its dynamic range. It's about 30 dB shy of what its CD counterparts can do.

Quote
Maybe it is the differences in the depth of the bass or some such.


The differences in the depth of bass or some such is that vinyl doesn't have greater depth, it has audibly far less depth.

Quote
Maybe it is related to something else.


Yes, naivete and sentimentality.

Quote
But I don't think it is related to the physical medium itself and that if they are not visually exposed to the medium they'd still prefer the vinyl-style mastering over the mastering intended for cd reproduction.


It is only a very tiny minority that prefers vinyl.

Quote
And I still stand by my idea that putting on a vinyl is more of an effort and that has a positive influence on the experience of music.


Good thing that you stand by your ideas... ;-)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: greynol on 2013-01-28 18:54:24
I'm totally lost.

You and I both.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: greynol on 2013-01-28 19:03:01
Quote
Maybe it is that of the modern vinyl pressing have a greater dynamic range than their cd counterparts.

Not a chance. They made the mistake of releasing test records on "modern vinyl pressings" so that technical types can measure its dynamic range. It's about 30 dB shy of what its CD counterparts can do.

Time for my 180.

I think he means "mastering" instead of pressing, implying that usually heavy DRC is often only applied to CD releases of titles.

If you want to join in on a discussion over that subject, here's a recent one:
http://www.hydrogenaudio.org/forums/index....showtopic=98199 (http://www.hydrogenaudio.org/forums/index.php?showtopic=98199)
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: krabapple on 2013-01-29 04:12:03
I'm totally lost.



Why?

True differences between vinyl and CD issues, such as exist, can be *at minimum* attributed to mastering (with all that entails), and to inherent audio issues with the analog technology.  (Unless you believe that digital has inherent audible issues too)

I think you know this already....? 


Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: 2Bdecided on 2013-01-29 09:45:42
I was lost with who suggested we blind tested what. It doesn't matter.

I'm quite clear that both vinyl and CD can sometimes sound the same (to my ears), but CD is sonically transparent to the source (to my ears) while vinyl often isn't (to almost anyone's ears!).

I do have a lot of records, and some decent turntables - but I can't take it too seriously. I don't look for, or expect, perfection. That would be to invite disappointment. I get easily as much fun out of my mp3 player.

Cheers,
David.
Title: Double-blind test of SACD and DVD-A vs. Redbook 16/44 in JAES Septembe
Post by: greynol on 2013-06-15 01:22:09
Due to the two off-topic posts in the last 24 hours, this thread will close.  If you wish to talk specifically about the study send me a PM and I will consider re-opening the discussion.