Could we agree on the following:

Now isn't it senseless for a reviewer to conduct exhaustive blind testing for components, that when properly built (and some audiophile grade gear has unbelievably low distortion) should all sound the same?

• A "properly" built amp has the following properties: Low output impedance, flat FR, any noise and distortion well below the threshold* of human hearing.
  [indent]
• An absolute threshold of hearing can be agreed on, at least in terms of a rather "safe" lower limit. Relating research is already quite old and nothing that would matter in this context has been discovered for a long time. Current and recent research has discovered many interesting properties of human hearing, but it mostly relates to masking and spatial processing. Both aren't of interest when looking at supposedly neutral elements of a playback chain as amps, DACs, cables, CD players, etc., which are already accurate at much smaller time scales than this research is concerned (exclude lossy encoding)

I'm left wondering: where are all these hundreds of amplifiers, DACs etc are that are completely faultless for the whole 120dB below peak output? These discussions assume this equipment is everywhere. That's not my experience.

ABK suggested 100dB as a good number a few pages back, but I'm not convinced. It's not enough if you calibrate to SMPTE RP200, for example. It's close, but I like to leave a margin of error, rather than cut it as fine as possible. For one thing, we have to chain several items together to make music - it won't work if we only make each one "just good enough".

Don't both sides just come down to faith? Objectivists believe they know what to measure, that they can measure it and that they know the effect it has (if any). Subjectivists believe that's not true and there must be other things involved that haven't been identified or measured yet.

Not everything is a codec. Codecs seem to be very different from things like amplifiers and CD players. In the hardware world we have seriously inaudible things like 0.1 dB FR and level shifts, and noise floors that are 30 dB below the noise in the program material. Not the same as ABXing 128 or maybe 320 kbps codecs.

I don't have a lot of experience with testing codecs. I do just about everything including personal listening with pure .wav files and redbook CDs.

Testing real-world hardware, particularly that of the floobydust kind, seems to be an especially different world than testing codecs.

Quote from: Arnold B. Krueger on 2009-04-28 12:15:36

The solution to questionable results is to simply run more trials. Run enough independent trials and the results will converge to something that is very reliable.

Absolutely. I can't recall, does Mr Atkinson or does he not agree on this?

Now a "better" audio magazine could promote the following chain:

| Source | -> | Colorization Device (DSP)| -> | DAC | -> | Amp | -> | Speaker |

Anything excluding the DSP and speakers would be neutral elements and could be just ticked off for being "proper" and else be reviewed for looks and features. The DSP adds noise and/or harmonic distortion just as preferred and could also be replaced by an analog component behind the DAC.

Now would that be a too honest approach? Would people want to read that their acoustical taste is intended signal degradation? Or do they want to read that their choice of a non-neutral amp (e.g. with quite some output impedance) is more musical and probably "closer" to the original performance? Subjective reviewing staff could really relocate all their magic to DSP analysis and how it can bring sound reproduction even closer to human experience.

Quote from: andy o on 2009-04-28 08:09:49

I think Fremer is XXXX. I think he's a XXXX XXXX. The Beach Boys review(s) quoted earlier are clear evidence, not to mention pretty much all we know about the guy.

Er, please try to stay within the boundaries of libel laws, people.

Mr Fremer has made contradictory - but in legal terms, 'fair comment' - statements. If he likes something today and doesn't like it tomorrow, he's 'mercurial', not 'lying'. If he got facts wrong and then corrected them without comment, unless his actions were in breach of code of conduct guidelines by the publication, the publishing house or a professional body or union he is a member, this does not constitute 'dishonesty'.

Quote from: andy o on 2009-04-28 09:44:44

That's exactly what I do with the iPod. Thanks for your comment though, it does give me some peace of mind that I was doing the right thing. I have tried the Shure earphones up to the SE530, but liked better the Westone UM2, which is too efficient for its own good. The attenuators that come with those though are very crappy, at some attenuation level they start to mess with high frequencies.

I share the enthusiasm for the UM2. Never heard anything better in-ear. The sensitivity really is a pain with iPods in very quiet environments. Your bad experiences with in line attenuators might be caused by the fact that the UM2 is two way with a passive crossover. The attenuator's resistors alter the behavior of the crossover.

Sorry, but calling someone dishonest, or even a liar, especially when having reasonable evidence, libel? I didn't know it was such a formal environment here. Perhaps I should adopt Penn & Teller's strategy of not saying any specific words, and just proclaim BS?

But anyway, what kind of small weak mind would even think of suing for libel someone over some internet comments, even if I did commit libel? I'm not writing for the New York Times for crying out loud.

[krueger:] the only way to understand Fremer is to consider the meaning of the word hyperbole. Fremer seems to live in a world of hyperbole where nothing is anything like what it seems....

has Fremer fanned the flames of Fremer-celebrity or possbily Fremer-fear so well that enough high end audio dealers and/or manufacters have been cowed into giving or loaning him most if not all of that equipment?

[krabapple]: There's a certain part of me that sees modern "audiophilism" as some sort of mental disease. Unfortunately, Fremer would probably feel similarly about anyone who subscribes to the notion of audio objectivism: that we are not only wrong but that we're indeed knee-deep in some sort of circle of madness. Thankfully, logic dictates that objective thinking is not only rational but quite comfortably sane, so I can (generally) rest easy

So 100dB down is 3dB SPL.

If some piece of equipment in the chain generates a whining noise at 3kHz, at a level which comes out at 3dB SPL, it meets your -100dB rule, yet has an audible fault.

Quote from: Stereoeditor on 2009-04-27 22:35:03

I argued at the time that 5 trials was too small, for the reasons mentioned, but that was the number decided upon, presumably to be able to get as many listeners through the listening test as was possible during the convention. The goal was to have a large number of listeners each doing a small number of trials, as I understood from the test organizers at the time.

As I think I mentioned before to Michael, I agree wholeheartedly that the results do not support the conclusion that you and Michael were "lucky coins", and it is news to me that you two got 5/5 and 4/5. It's plainly unfortunate that more testing was not done.

At the same time, do you not agree that the overall result of the test - IIRC, that the results overall were fairly indistinguishable from chance (I don't have the paper on me right now) - still has a very important meaning? I am comfortable with the notion that a small fraction of listeners are able to tell a difference in such situations, while most listeners can't. And if you get close to a 50% result, while individual testers such as yourself can still pass a 16- or 32-trial test with flying colors, well..

Moreover, I think that such results can have considerable importance for those who cannot pass such ABX tests, and perhaps can readjust their purchase priorities accordingly. Of course this has to be balanced against the odds of the listener becoming more adept in the future to hearing such details.[

So I'm really tempted to just straddle the fence here. That test did not show that you and Michael could hear the difference, because your results were as to be expected as due to chance. But they certainly did not show that you couldn't hear the difference, either - it is quite plausible - and more testing really should have been done on that matter. The test did strongly suggest that a majority of the testers could not tell a difference, and that in itself is important.

Put another way, the test was certainly "flawed" if it was attempting to show that absolutely nobody could hear a difference, but under a more relaxed criteria (of showing that the proportion of discriminators must be below some low percentage), I don't think it's flawed at all.

If there was an Audiophile union, you could have added Class Action on top of it.

Quote

You do the cause of DBT a huge disservice. John Atkins comes across as perfectly reasonable; you come across as a complete twit (I misspelled that). If I had no prior knowledge, and could only judge this subject from the tone of this discussion, I'd be deleting my HA account and subscribing to Stereophile immediately.

As someone who really does have very little knowledge of the subject, I've got to second this. I mean, I totally understand being tired of making the same civil and informative arguments over and over and over (and over) again, but that is what being an advocate for something entails. If you (for whichever value of 'you' is preferred) would rather score points on the Great Electric Arguing Machine that is the internet (people think I'm joking when I call it this) than actually convince people of anything, well done I suppose.

As it is, I think I hate ABX tests on principle now, and will be making my future audio equipment purchasing decisions based on the number of people it will annoy. I wonder if there's a double-blind test for that?

Quote from: 2Bdecided on 2009-04-28 15:14:53

So 100dB down is 3dB SPL.

If some piece of equipment in the chain generates a whining noise at 3kHz, at a level which comes out at 3dB SPL, it meets your -100dB rule, yet has an audible fault.

I found this very interesting and generated a -100db 3kHz (24 bit, 44.1kHz) test tone in Wavelab. Just to hear it at all I had to turn up the volume about three times as high as my usual listening setting and about twice as high as my recently loudest setting (party).

I think I understand why I am (and Canar is) so flummoxed about this entire topic, going all the way back to when B0RK was the center of attention. You, and also Steven to a certain degree, have turned this thread into what seems like a word-for-word rematch of the exact same argument, made against the exact same people, that has been replayed on Usenet for what seems like a decade - an argument I have absolutely no intention of supporting.

I would guess that your PC is louder than 3dB SPL - you need to try in a properly quiet room.

Over half a century of psychoacoustic testing says that 3kHz @ 3dB SPL is audible for most people with "normal" hearing.

btw, there's no calibration in your system,

I would be very happy if all the crap that originated in the RAO cesspool, stayed there.

I have my own concerns; I sincerely believe that Stereophile's work within its ambit has a pernicious effect on 'my' hobby, and I'm not afraid to act on that.

I'm sure I can be perceived as 'militantly' anti-audiophool, just as I suppose I would be called a 'militant' atheist or a 'militant' scientist by the sorts of people who find Richard Dawkins or Chris Hitchens just too plain *rude* and who fret over whether they are helping or hurting 'the cause'

Quote from: Frumious B on 2009-04-28 01:56:19

If an individual listener is unable to differentiate a lossy file, either under the conditions of an ABX test or in a more relaxed and conventional listening setting, from its lossless counterpart then in what way, precisely, does the lossy file "not offer sufficient audio quality for serious music listening"?

For that person at that time, then yes, the lossy compressed file is presumably good enough. But everyone's threshold is going to be different and perhaps more significantly, their threshold of defects will change with time. For example, when I was younger, I wasn't bothered by scrape flutter in affordable analog tape machines. Just as well, as that's all I could afford at that time. But over the years, I have become much less tolerant of it, presumably because I have learned to identify it, and that is something that can't be unlearned.

Hence my blanket recommendation to which you refer: lossless or uncompressed for "serious" listening, to which I would add archiving. Why not when hard drive capacity is now so cheap. And for portable listening, I personally use AAC at 320kbps and recommend that, even if it might be thought overkill. People are not obliged to follow my advice, of course, and they are free to make their own decision about where to make the trade-off between file size and bit rate. But I think of an email from a Stereophile reader who ripped all his CDs as 128kbps MP3s and disposed of the CDs. He is now dissatisfied with the sound of his music collection but can't do anything about improving it short of repurchasing the CDs.

John Atkinson
Editor, Stereophile

PS: I always recommend to my readers that they keep their CDs, as inconvenient as that might be. They are the backup of last resort.

• The entire notion of using frequency response plots to estimate the performance of lossy encoders is fundamentally flawed, because they only show one source of distortion while being ignorant of many far more important ones, like pre-echo and stereo imaging. They are almost useless for such evaluations. Relying on plots to tune coders - especially at lower bitrates - is a direct cause of the crappy MP3 sound that JA comments on, and yet he makes the exact same mistake that so many others have made in the past! In a magazine that prides itself on the primacy of subjective evaluations! HA as a forum is profoundly against the use of objective measurements (like frequency response) to evaluate coders, as is jj IIRC, who is about the best expert opinion one is likely to find on the matter...
  
• Too little information is provided on the Fraunhofer codec that JA uses. Is it fastenc or l3enc? Which version? CBR or VBR? Which other settings? All these things matter tremendously. As Sebastian's latest test results pointed out, 128kbps can vary quite tremendously in quality - and at least one Fraunhoder codec is considered among the worst codecs out there.
  

You may be thinking of Michael Gindi who was parodied in Sam Tellig's 1980s columns in Stereophile and then got a gig with The Absolute Sound and maybe Ultimate Audio and had an MBL fixation--that guy was definitely a psychiatrist.

Reading Terms Of Service #8 specifically refers to ABX (ie subjective) tests as the objective support:

"8. All members that put forth a statement concerning subjective sound quality, must -- to the best of their ability -- provide objective support for their claims. Acceptable means of support are double blind listening tests (ABX or ABC/HR) demonstrating that the member can discern a difference perceptually, together with a test sample to allow others to reproduce their findings. Graphs, non-blind listening tests, waveform difference comparisons, and so on, are not acceptable means of providing support."

As this method is the #1 acceptable form of support, it explains why A.krueger was expected by some to carry the flag in this debate.
Reading A.Krueger's viewing angle though, & the posts provoking him to do so, it becomes clear to me, that this is not the case.

A.Krueger explains that the ABX tests he conducted /designed, were designed/intended to be used in comparing Hardware, & closely specd Hardware:

Not everything is a codec. Codecs seem to be very different from things like amplifiers and CD players. In the hardware world we have seriously inaudible things like 0.1 dB FR and level shifts, and noise floors that are 30 dB below the noise in the program material. Not the same as ABXing 128 or maybe 320 kbps codecs.

I don't have a lot of experience with testing codecs. I do just about everything including personal listening with pure .wav files and redbook CDs.

Testing real-world hardware, particularly that of the floobydust kind, seems to be an especially different world than testing codecs.

Reading the above, If I am not mistaken, I can only gather that it is not A.Krueger, that is promoting the use of ABX test methods as applied here, He is not even claiming they are essential or even vaild in this case.