Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: ABX testing vs. perceptual states (Read 23247 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

ABX testing vs. perceptual states

Reply #50
No. Not when we're evaluating recording equipment and what we're looking at is the cumulative effect that the recording chain and process has on the resulting recording. The recording is not the source, it is the product.

No. In an ABX test of recordings you would compare recording A with recording B. If a listener cannot find a difference between A and B the recordings would appear to be transparent with regard to each other for that listener.

You are incapable of arguing anything without using logical fallacies it seems (I've seen at least four now), so I'm done with this. Keep living in your fairytale world.


Well, what we've been talking about mostly is comparisons of gear. Comparing a recording made with the gear isn't the same as comparing the gear because there are other factors that go into the recording.

ABX testing vs. perceptual states

Reply #51
There are circumstances where no good scientific test is possible, but that's a defect, not a recommendation, particularly when the differences get subtle and bias and confabulation become far more likely.


Ah, and that is exactly one of the points I'm trying to make.

p.s.  I can't resist adding this beautiful example by Feynman of the power of bias.  The only thing I would add is that it is now and will ALWAYS be a problem, even in the sciences, because sometimes there are very powerful desires behind a bias, and scientists are by no means immune:
Quote
We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease.



Great quote! Feynman is one of my heroes.

ABX testing vs. perceptual states

Reply #52
It's not that ABX testing, used correctly, is invalid. The key phrase is "used correctly". Some people appear to believe that ABX is some kind of panacea - that if it can't be proved by ABX it doesn't exist and that is simply not true.


Point us to one.

If you can't see it, you can't see it. Sorry.

ABX testing vs. perceptual states

Reply #53
If I may address all, please, folks, realize that the creative side and the technical side are both important in this industry, and it woudl be remarkably nice, not to say unusual and amazing, if we could get along with each other once in a while.

John, ABX testing really does work for determining if a given subject can hear a difference, when it's done right.  Nobody can prove a negative, not a scientist or anyone else, but there is valid statistical inference regarding populations from repeated test results, being negative or positive. There is never any absolute, but a probability suggesting that the age of the universe is involved is certainly definitive.

Techie guys, you know, the artistic side tries to lay down what sounds good. Euphony is their stock and trade. There's more than a little bit to be gained in the knowledge they have, as well, rather than simply telling them that their tape deck distorts and isn't as accurate. It's NOT as accurate, but it might just sound better, since rather obviously, what we capture in the studio or anywhere else is about, oh, what, 1/10000th of the analytic information in the soundfield around the listener's head.


So, how about some friendly dialog?

Hydrogen audio may not be the best place, I suppose, you know, you CAN accurately measure a given person's preference, but with an ABC/hr test, so strictly speaking, it wouldn't be ABX, it would be another test.  So, I suppose purely testing preference would violate TOS 8.

But given the ranting I get from audiophiles (check out the talk announcement I put up on stereophile board), I understand why TOS 8 exists. Some arguments just aren't worth having.

So, dunno an answer, but fighting accomplishes nothing.

Exactly. There is a lot we don't know - we need to observe and study.

What bugs me are people who assume that we have everything we need to understand everything (which is patently absurd).

What also bugs me are people who attempt to misuse a tool like ABX to try to "prove" a negative, which is also patently absurd.

What's so weird about that?

What bugs me about Ethan is that he commits both of these errors but claims to be standing up for "science".........

ABX testing vs. perceptual states

Reply #54
4) Now about perceptual state of the brain, while fascinating, I'm not sure it invalidates the abx test. But maybe it can be useful to study how a trained human being can perceive things that others don't.

malice


I'm not really saying that it invalidates the ABX test at all. What I'm suggesting is that it might not be testable by ABX. The very situation of the ABX test might "Heisenberg" it, as it were.

ABX testing vs. perceptual states

Reply #55
Point us to one.
If you can't see it, you can't see it. Sorry.
Unless you can point us to one, I'm going to insist that you cease arguing your point. krabapple is making a very important distinction, one that I touched on earlier as well. So I insist, as a moderator: point us to one or this argument becomes completely pointless.

You're trying the most intellectually insincere of evasive maneuvers here.

Whether your example is accepted as having a point is irrelevant. State your example or shut up.

ABX testing vs. perceptual states

Reply #56
What bugs me are people who assume that we have everything we need to understand everything (which is patently absurd).

What also bugs me are people who attempt to misuse a tool like ABX to try to "prove" a negative, which is also patently absurd.
You are dragging this off-topic and back towards the AES thread. I can't comment about the former, but if you bring up genuine cases of the latter, your position is unassailable around here. However, you seem to want to keep levelling personal attacks at Ethan without presenting one iota of hard fact or even any crappy, bogus examples to lend any substance whatsoever to your argument.

If you are unwilling to provide any substance, your posts are not welcome here.

ABX testing vs. perceptual states

Reply #57
So John is what you're saying is that the same basic kinds of performance and orthodox electronic theory that make a digital convertor good for playback does not make it good for recording?


What I'm saying is that a higher degree of precision is desirable for recording equipment; consumer level digital gear imparts characteristics that are, for lack of a better term, uneuphonious. Actually, ALL pcm digital equipment has some degree of undesirable sonic character, but it is more pronounced in the lower quality stuff, yes. That's not to say that just because something is expensive it's automatically good, of course.

Joh, you are picking the wrong pliers. Try the pliers that orthodontists use to adjust braces. Your mistake here is that you are comparing a brute force tool to a fine adjustment tool, which invlidiates your comparison of dentisty to jewelry making.

I wasn't comparing dentistry to jewelry making. I was comparing a coarse tool of a given category to a fine one. I actually considered using a plumber's pump pliers as the coarse example, but they're usually not considered the proper tool for the job (compared to a pipe wrench), so to avoid ambiguity on that count I used tooth pulling pliers instead.

The comparison was a gross pair of pliers to a fine pair of pliers. The Soundblaster may be fine for everyday consumer playback of music. It does not, however, have the precision and detail of, for example, a Lavry.

ABX testing vs. perceptual states

Reply #58
Recordists are invulnerable to bias?

Not so much! ;-)


I don't believe recordists are invulnerable to bias (as I don't share many of John's view, mind you)

I do believe that recordists (and I mean professional experienced recordists) have one major advantage as matter as bias is concerned. Good engineer mixers (especially mixers) know that bias is the number one concern in their field. Mixers do know that the mind is a very tricky thing that will make their mixes sound good while they are "not so good yet". Same applies to ME and  I think it's Bob Katz that said once : "you will only have one chance to hear a mix for the first time". I would be very curious to learn more about a serious research about this topic of bias and how it affect our judgement and how we integrate the fact that we KNOW we are vumlnerable.


It turns out that Katz and Massenberg have granted crediblity to blind listening tests in public statements. I know of no hand-waving from them about fear and loathing and performance anxiety.

Quote
Quote
Your first problem John is that Ethan and I are recordists. I know that Ethan records well because I've heard his work. The recordings I make are copyrighed by a number of someone else's so I can't freely disseminate my work like Ethan does. And, they are not made using professional artists in professional grade venues. So I don't have that to wave around. I can fearlessly predict that other than an outright miracle of God, I'll never be credited with anything to do with an award-winning recording! However, I probably have in the past 10 years made more recordings by accident than most professional recordists have made on purpose.


I do not know about your credentials in matter of recordings, and I certainly won't evoke what I know about Ethan's work.


I don't know about yours Malice. I asked you to provide your credentials privately several days back and you are a no-show.

As far as Ethan's work goes, it is a jillion times better than what is available to me of yours, because something  that is even just halfways-decent always beats absolutely nothing.

Quote
You did notice I was not good enough in the Electrical engineering side.


To say the very least!  This raises further issues about crediblity credibility because you did give your statements as if you thought you were some kind of authority in the matter. "I don't know" is a fair answer that gets respect. So is silence.

Quote
I might be wrong, and I hope you'll forgive me about this, but I've seen comments from you that tells me you would be able to handle a very long argument about mixing with me or mixerman.


That  statement is  trash talk because you have provided no credentials, despite you making an issue of the matter and failing to provide them, even privately.  The womb has proven itself to be far from being a paragon of free speech.


Hey Arnold, I've repeatedly asked you for credentials and you haven't shown me squat. Checking online all I find is a raft of forum posts. I've told you more about my experience than you have told me about yours. About the only thing you've said is your claim to have "invented" ABX, which I take to mean your particular variant of standard double blind AB testing, which is something that I was involved in with a group of associates in the early to mid '70s. Oh, and that you favor cheap junk gear over quality gear.

ABX testing vs. perceptual states

Reply #59
So John is what you're saying is that the same basic kinds of performance and orthodox electronic theory that make a digital convertor good for playback does not make it good for recording?


What I'm saying is that a higher degree of precision is desirable for recording equipment; consumer level digital gear imparts characteristics that are, for lack of a better term, uneuphonious. Actually, ALL pcm digital equipment has some degree of undesirable sonic character, but it is more pronounced in the lower quality stuff, yes. That's not to say that just because something is expensive it's automatically good, of course.



That would appear to be a fairly massive TOS8 infraction. Actually several of them.

At this point, a massive TOS8 infraction might even be strategic for you. :-(

ABX testing vs. perceptual states

Reply #60
I don't believe recordists are invulnerable to bias (as I don't share many of John's view, mind you)


I don't recall having ever said that recordists or anybody else are invulnerable to bias. Nor can I think of any reason why I should.

 

ABX testing vs. perceptual states

Reply #61
Hey Arnold, I've repeatedly asked you for credentials and you haven't shown me squat.


Back at you, John.

Quote
Checking online all I find is a raft of forum posts. I've told you more about my experience than you have told me about yours.


I don't think that's true at all.  I don't think I mentioned that I have a BS in Engineering, for whatever that is worth this far down the road.

Quote
About the only thing you've said is your claim to have "invented" ABX, which I take to mean your particular variant of standard double blind AB testing, which is something that I was involved in with a group of associates in the early to mid '70s.


There is a JAES paper that dates and validates my claim. 

Got game?

Quote
Oh, and that you favor cheap junk gear over quality gear.


So John, in your book Fluke, HP and Audio Precison is cheap junk?

Interesting.




ABX testing vs. perceptual states

Reply #62
Maybe I've missed the point, but I don't see why the discussion has to be rancorous.  It's certainly possible that abx testing or the other tests that are well-accepted here do not cover all forms of listening perception or even interfere with those other forms of perception.  But what is the way of proving the existence of that other form of perception?  What is the alternative testing to be used? 

It's as if one were identifying a forgery or a wine knockoff from the originals.  Someone may do it through meticulous concentration on brushstrokes or acidity.  Someone else react to the whole, and say the painting or wine lacks depth, warmth, richness etc.  Fine.  There's no problem either way, whatever perceptive capacity they exercise, if they can consistently identify the painting or wine without cheating. 

All the tests used in this forum are designed to rule out guessing and cheating (and the worst kind is the unknowing or unintentional one of bias).  I don't see any particular theory required by any of the tests, except for commonplace observations about the prevalence of bias, mechanisms to avoid or neutralize that bias, and the use of well-established laws of statistics.   

So why any rancor?  There's no attachment to a theory or not much of one, and no attachment even to a particular test, if there's another test that can rule out bias and guessing.


I see no reason for rancor at all.

I'm just calling into question the applicability of certain tests for determining certain things about certain phenomena and pointing out that some things may not in fact actually be disprovable in spite of claims that they are, although they are also not easily provable, either, despite a huge amount of anecdotal evidence, which nobody seems to be in any great hurry to put to the test.

I don't see any reason to get nasty about any of that.

OTOH, when certain people instigate that sort of behavior they should expect to be repaid in the same coin.

ABX testing vs. perceptual states

Reply #63
Incidently, I wonder why I cannot find an hi-fi magazine or a pro-audio magazine totally unbiased because of  their sponsors.
There's at least one on-line hi-fi "magazine" which is subscription only, and carries no adverts. It claims to be unbiassed. (If I could remember what it was called, I'd say - I'm being forgetful, not coy!)
Cheers,
David.


That would be "The Audio Critic" magazine.
But the publisher Peter Aczel has semi-retired and only made 6 entries in 2009.
Only 4 entries in 2008.

http://theaudiocritic.com/plog/index.php?blogId=1
Kevin Graf :: aka Speedskater

ABX testing vs. perceptual states

Reply #64
It's certainly possible that abx testing or the other tests that are well-accepted here do not cover all forms of listening perception or even interfere with those other forms of perception.  But what is the way of proving the existence of that other form of perception?  What is the alternative testing to be used?

Indeed. It is possible that ABX tests do not cover all forms of listening perception. Russell's Teapot is worth considering here, even if it was initially proposed in a different context:

Quote
If I were to suggest that between the Earth and Mars there is a china teapot revolving about the sun in an elliptical orbit, nobody would be able to disprove my assertion provided I were careful to add that the teapot is too small to be revealed even by our most powerful telescopes. But if I were to go on to say that, since my assertion cannot be disproved, it is an intolerable presumption on the part of human reason to doubt it, I should rightly be thought to be talking nonsense.

If not ABX then what? What would these tests look like? ABX-detractors do not go so far as to identify alternate procedures that contain adequate safeguards against the critical errors caused through placebo and sighted-testing. They do not do so for one simple reason: They lack the nuanced understanding of statistics and science required to create such a novel methodology.

Double-blind tests are one of the gold standards of scientific testing. Where they cannot be undertaken, researchers frequently complain that they wish they could be.


I agree entirely. Except about the teapot. It's actually a wine glass.

But seriously, I don't think I've ever said that properly applied double blind testing is not valid. Nor have I stated unequivocally that the phenomena we're discussing are untestable by ABX or other double blind tests. All I've said is that the absence of a result in any given test group does not constitute proof that the phenomena don't exist. There are, as you noted, some things that just can't be tested in this way, which is unfortunate.

In this case the use of statistics is of dubious validity, because the question is can differences be detected by anybody, specifically any individual who is a trained professional recording engineer, not can a difference be detected by an average member of the population.

ABX testing vs. perceptual states

Reply #65
...But what is the way of proving the existence of that other form of perception?  What is the alternative testing to be used?...

You can't prove if it doesn't exists.
I don't know why some people here don't understand that brain hears what comes through the ears and than imagine or fabricates extra information according to ones biases and mental health. The ABX test eliminates that extra imagined "information". There are people out there that hear voices that nobody else can (schizophrenia). Does that mean they have a better hearing?!

It doesn't matter how euphonic your imagination is. It is irrelevant to anybody else because it is inside your head only. It doesn't exist. That's why it would not survive a double blind test.

There is no such a thing as somebody that have a better discernment in a test that is not double blinded. The double blinded test gives you always a better discernment. Otherwise your "taste" and inclination to like more certain brands over others, added to, lets say, low confidence level, desire to blend in and other psychological deformations will drive you to choose what is more complex and more expensive instead of what actually makes sense.

But you can't prove it doesn't exist just because you can't see it with your available tools. Until a few years ago extrasolar planets were considered a mere hypothesis - some astronomers thought them likely, many believed they didn't exist. Now, with improved technology, we KNOW many do exist; they are in fact common.

Just because you can't see it doesn't mean it isn't there.

The double blind test can prove the difference is there. What it can't prove is that it isn't. If you don't understand that you simply don't understand double blind testing.

ABX testing vs. perceptual states

Reply #66
John Eppstein. Just see it pragmatic: ABX is just a protocol to adress one specific issue: listener bias - not more & not less. Listener bias, or generally any subjective bias in perception is a well researched and very strong phenomenon. I have experienced it myself and very embarrassingly. I would have sworn once that my expensive Parasound DAC sounded way better than my Terratec sound card. Its capacitors alone were probably weighting the equivalent of 10 Terratec sound cards. A difference like night and day. I was so sure, I would have bet my car on it. My description would have been something like "very analog sounding with a very precise and well defined lower end".

A friend was also interested to purchase an expensive, external DAC himself and brought over his audio workstation, already containing a high end Lynx converter. We recorded the output of the Parasound, then a loopback of the Lynx, and later also the Terratec's output. The result was shocking and eye-opening: No matter which of our favorite records we threw at all three, both of us were unable to tell which was which in a double blind test. And trust me, I made sure that none of the contenders got a unfair advantage: I insisted, for example, on 192 kHz recording and and both the Parasound and the Lynx were used for playback... In the end my friend was happy, that he already owned all the DAC he needed, and I am very sincere since that day to not trust my ears anymore, as long my eyes are also involved. And see, the effect of sighted testing was so strong and ABX is so good at eliminating it, I don't see any reason to question its benefit for some obscure undefined MRI hypothesis.

ABX testing vs. perceptual states

Reply #67
The second argument is about the euphonic qualities that we are debating for several days.
It refers to how some people, like myself, feel that small amount of distortion can increase  apparent depth and dimension, and therefore, sounds "better".

And that would be an impossible can of worms to debate here because it's impossible to know if that "better perception" is caused by the increase of the harmonic distortion components that increase the overall level or by a real "improvement" in depht and dimension caused by the said distortion.

So yes: very difficult to prove or disprove this but that is definitely worth considering when you compare a good tape recorder and a converter or an analog EQ with big Iron transformers in it, and a plug-in EQ.

malice


And if it is the addition of the distortion and not a level change that increases apparent depth and dimension, is it because the distortion is in effect, replacing some component of the sound that was lost earlier in the recording process, say when the mic converts the 3D sound field to an electrical signal? Which therefore make it sound "more real"? Do certain distortion "signatures" do this better than others?

ABX testing vs. perceptual states

Reply #68
Ok, I did make some research about that 0.1 db increase, and I'm afraid I cannot yet post that post from Bob Katz as it is part of a private forum that forbids me to quote exactly or refeer about the sources.


People can say whatever they want. I can say that I've heard 0.01 dB level differences but oh darn, I've lost track of the actual details. ;-)

Back in the 80s there was a guy named Peter Moncrieff who claimed to hear differences measured in millibels. Strangley enough nobody was ever able to duplicate his work in a bias controlled listening test. ;-)

Quote
But I can elaborate on the very interesting things that has been said and that shows how difficult it is to evaluate anything analog in tests like ABX. Not saying impossible, but you have to take in account what you are testing.

The first argument was about the drift of some analog devices shows, even with 20-turn trim posts, calibration can change, and 0.1 dB is not out of question. If you add a very slight of load, you can end up with another 0.1 dB and you're off the charts of proper abx testing.


I've been level-matching all sorts of audio equipment, even relatively unstable equipment such as analog tape and vinyl playback, since the late 1970s. Analog tape and vinyl can definately drift or wobble during playback by up to 1  dB. Normal signal-handling equipment such as amplifeirs, preamps, equalizers and the like have no problem staying within 0.1 dB.

Very shortly after we started doing ABX, I developed a precision level-matching device that was composed of  a pair of 10-turn potentiometers. I put crank handles on the pot shafts, and it was called "The Fishing Reels". It was basically overkill because we ended up adjusting it with big moves. I've had no problems setting levels within 0.1 dB using  RN50 single turn linear taper potentiometers.

Quote
The second argument is about the euphonic qualities that we are debating for several days.
It refeers to how some people, like myself, feel that small amount of distortion can increase  apparent depht and dimention, and therefore, sounds "better".


As soon as there is an actual audible difference, then how it perceived needs to be evaluated on a individual basis. In some cases ditortion can be added to compensate for an undesirable distortion that crept in from someplace else. What a given kind of distortion sounds like is usually very dependent on the actual music you listen to through it.

Quote
And that would be an impossible can of worms to debate here because it's impossible to know if that "better perception" is caused by the increase of the harmonic distoprtion components that increase the overal level or by a real "improvement" in depht and dimension caused by the said distortion.


Here we see yet another example of an unfortunate choice of words that may lead to false beliefs.

There is no such thing as harmonic distortion as a property of equipment.

Harmonic distortion is an artificial theoretical concept that is used to characterize or measure nonlinear distortion.

All nonlinear distortion creates both harmonics and intermodulation products. However, low pass filtering can prevent harmonics from ever being measured. 

Nobody can ever increase harmonic distortion without also increasing intermodulation distortion. In fact it is possible for equipment to be nonlinear and produce no measurable harmonic distortion, but it will still produce measurable IM. 

For example,  back in the days when resistive networks were used to build DACs there could be signficiant nonlineariities in the resistive network but the reconstruction filter would strip off the harmonics for test signals above about 6.5 KHz. This would create an apparent exception to the usual rule of nonlinearity in electronics increasing at the highest frrequencies. 

Intermodulation distortion produces non-harmonic or aharmonic distoriton as it acts on real-world music, which is more likely as being perceived as being undesirable because it is not musical.


Sounds like the same kind of anecdotal evidence that you won't accept from anybody else. Why should we accept it from you?

And no such thing as harmonic distortion as a property of equipment? Are you serious?

ABX testing vs. perceptual states

Reply #69
I think the show me your weiner or else I won't accept your authority posts could be removed from the thread. Actual arguments and references to data instead of credentials would make much more sense for the rest of us.

ABX testing vs. perceptual states

Reply #70
Hey Arnold, I've repeatedly asked you for credentials and you haven't shown me squat.


Back at you, John.

Quote
Checking online all I find is a raft of forum posts. I've told you more about my experience than you have told me about yours.


I don't think that's true at all.  I don't think I mentioned that I have a BS in Engineering, for whatever that is worth this far down the road.

Quote
About the only thing you've said is your claim to have "invented" ABX, which I take to mean your particular variant of standard double blind AB testing, which is something that I was involved in with a group of associates in the early to mid '70s.


There is a JAES paper that dates and validates my claim. 

Got game?

Quote
Oh, and that you favor cheap junk gear over quality gear.


So John, in your book Fluke, HP and Audio Precison is cheap junk?

Interesting.


Well you told me that you don't use your Audio Precision test set because your cheap computer soundcard and a freeware software program is better. I find that assertion very difficult to take seriously. You also told me that you prefer a Behringer power amp to a Threshold SA/4e and didn't appear to understand the importance of correctly biasing the amp or of replacing the aging filter caps - or at least verifying that they were in good condition, which is highly unlikely in the case of 15 year old electrolytics. So I'm just taking you at your word. BTW, if you don't want that "inferior" Audio Precision I'll gladly pay the shipping to take it off your hands.

ABX testing vs. perceptual states

Reply #71
John, is that post really without rancor? If you don't want it, don't do it.

Quote
is it because the distortion is in effect, replacing some component of the sound that was lost earlier in the recording process, say when the mic converts the 3D sound field to an electrical signal?

Is that a serious question, or a diversionary tactic? We can only speak of post microphone. As of yet, a microphone gives a poor representation of the soundfield around the listeners head. I think Woodinville put its efficiency at 1/1000000000th or something of that magnitude!
Quote from:  link=msg=697080 date=0
I think the show me your weiner or else I won't accept your authority posts could be removed from the thread. Actual arguments and references to data instead of credentials would make much more sense for the rest of us.

Agreed.

ABX testing vs. perceptual states

Reply #72
But seriously, I don't think I've ever said that properly applied double blind testing is not valid. Nor have I stated unequivocally that the phenomena we're discussing are untestable by ABX or other double blind tests. All I've said is that the absence of a result in any given test group does not constitute proof that the phenomena don't exist. There are, as you noted, some things that just can't be tested in this way, which is unfortunate.
As this is the case, until you care to make further claims, I am closing this thread, as there is nothing more to say except continue on with petty bickering and trolling.

You are welcome to make testable claims in another thread.