Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Biophysics, Limitations of Shannon and Issues with ABX Testing (Read 54646 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #25
Now is the hearing upper limint of human hearing completely independent of waveform, or is it not?

It appears that humanity, so far, have been unable to show audible differences between the CD format and the DVD-A/SACD formant, a prominent difference between the two is that the latter allows extension far beyond the 20kHz or so of CD.

One would think that the variation of waveshape found in such studies would be sufficient, although the amount of energy beyond 20kHz tends to be low (as is the case with most live music).

-k

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #26
It appears that humanity, so far, have been unable to show audible differences between the CD format and the DVD-A/SACD formant, a prominent difference between the two is that the latter allows extension far beyond the 20kHz or so of CD.


.... indicating that the CD format is “more than good enough”, but not what would be “precisely good enough”.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #27
It appears that humanity, so far, have been unable to show audible differences between the CD format and the DVD-A/SACD formant, a prominent difference between the two is that the latter allows extension far beyond the 20kHz or so of CD.


.... indicating that the CD format is “more than good enough”, but not what would be “precisely good enough”.

Indicating that the decades of research into human auditory perception is right, a lowpass filter at 15 kHz or 25 kHz or whatever (depending on age, gender, genes, ...) seems to be transparent. Any claims that humans can somehow take advantage of ultra-sound in certain non-linear, waveform-dependent ways are less believable as a result.

At my age, 15 kHz tends to be sufficient.

-k

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #28
Now is the hearing upper limint of human hearing completely independent of waveform, or is it not?


Uh, of course it's not "independent of waveform" since the domain the ear responds in is 4th order resonance, give or take, and that is going to map directly to a fourier basis.

So you need to use a fourier basis, not "any waveform".
-----
J. D. (jj) Johnston

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #29
ly.  In principle, one could play a pristine recording, and that same recording to which very low levels of a very irritating form of audio distortion had been added, while monitoring a specific physiological attribute.  With a sufficiently large N, and with the right attribute, one might find significant differences in physiological response to the pristine and non-pristine versions, even if the subjects subsequently failed to distinguish them based on ABX testing.


And away we go.... 

http://en.wikipedia.org/wiki/Hypersonic_effect

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #30
If it is found that subjects indeed fail to reliably distinguish colors when presented sequentially that they can reliably distinguish when presented simultaneously, this will suggest that sequential ABX testing may not be a good method for assessing our perceptual limits (i.e., for determining transparency) --  and not just for vision, but possibly for hearing as well.

I'm a little late to this discussion but the partial quote above seems to contain a fundamental flaw, my highlighting.

I would suggest that "sequential" ABXing and "simultaneous" ABXing of colours would show a large discrepancy because most people have very poor colour memory. I spent a large portion of my working life in colour-related industry and only ever came across a single person whose colour-memory was uncannily good. That aside, it surely is a flawed idea that hearing must be also be so afflicted, unless there is evidence to suggest it. Is there any evidence either way?

In addition, it is patently obvious that "simultaneous" ABXing is possible with coloured tiles, whereas with audio samples it is inherently somewhere between difficult and impossible. The fact that a person's ear could discriminate between samples if they were simultaneous is irrelevant since that isn't feasible. What matters is if they can "hold" the audio memory or not.

Let me suggest a more appropriate analogy to colour. Let's say you decide to re-paint your living room walls. You pay a visit to a paint supplier and select your desired colour from a chart. A machine then dispenses the required mixture and off you go and paint your walls. Knowing the technology involved, I can almost guarantee the walls will not match the chart exactly. Will you notice this? Highly unlikely, assuming there was no grave error in the process. Would you be able to ABX the chart and your walls? Highly likely. Does this make the wall-painting process "non-transparent", in the audio-ABX sense? Not a bit.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #31
...sequential ABX testing is hard, because it requires accurate memory of both A and B when judging X...


A NON blind test also requires accurate memory of both A and B. So your argument is irrelevant.


Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #33
The fact that a person's ear could discriminate between samples if they were simultaneous is irrelevant since that isn't feasible.


And what level-roving experiments show is that you want clean, clickless near-instantaneous switching, no more, no less.

As to the title, the Shannon theorem is right, and ABX testing is as sensitive as anything else when it's done right.

I've seen many miserable excuses for why that isn't, but I am frankly tired of them, it's the same old mistakes over and over again.
-----
J. D. (jj) Johnston

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #34
The fact that a person's ear could discriminate between samples if they were simultaneous is irrelevant since that isn't feasible.


And what level-roving experiments show is that you want clean, clickless near-instantaneous switching, no more, no less.

As to the title, the Shannon theorem is right, and ABX testing is as sensitive as anything else when it's done right.

I've seen many miserable excuses for why that isn't, but I am frankly tired of them, it's the same old mistakes over and over again.

I didn't word that well - a better wording would have been: The fact that a person's ear might be able to discriminate between samples if they were simultaneous is irrelevant since that isn't feasible.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #35
Quote
And what level-roving experiments show is that you want clean, clickless near-instantaneous switching, no more, no less

In my experience, based on my perception, lightning fast transitions are indeed critically important to being able to audibly discern subtle differences between samples [such as level or minor tonal balance variations] however I've always assumed that the reason we want to avoid transitional clicks between A, B, and X is because they potentially may act as a "tell", a giveaway as to the samples' identity, not because hearing the minor clicks themselves diminishes our sensitivity to picking up on small differences. Correct?

What I'm getting at is that consistent, minor ticks/clicks which are first verified, pre-test, to be audibly identical each time one switches between samples [so it can't act as a "tell"] is perhaps not ideal, however it is far less destructive to our sensitivity, in my mind, than another solution I've sometimes seen used to overcome minor clicks in some tests, where a quick volume ramp is introduced at each transition to obscure any such clicks. The problem being the time delay introduced by that volume up ramp, even if it is only one second in duration, reduces our sensitivity to hearing minor changes, significantly. [Also, hearing the volume ramp itself, or a beep, or pink noise, or any alternative sounds to the targets, for that matter, may simply throw off our concentration and focus to the task at hand.]

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #36
however I've always assumed that the reason we want to avoid transitional clicks between A, B, and X is because they potentially may act as a "tell", a giveaway as to the samples' identity, not because hearing the minor clicks themselves diminishes our sensitivity to picking up on small differences. Correct?


No, actually, if you'll think briefly about the loudness of a click, in addition to being a tell, it ALSO mucks up short-term loudness memory, so it reduces sensitivity AND provides a leak in the DBT protocol. It's a twofer.
-----
J. D. (jj) Johnston

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #37
How can something that serves to make distinguishing two samples more difficult also serve as a tell?

It doesn't.

ABX testing, as I know it, hands the power of switching to the testee. A click will only be a tell if it occurs for one sample and not the other for both the unknown X and the corresponding known (A or B, but not both). That a click occurs as the result of switching is meaningless in alerting the testee to the switch since the testee initiated it.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #38
Two mono samples, A and B.

sequential: Listen to A then B.

simultaneous: Listen to A in one ear and B in the other.

Some differences will be far easier to hear in the simultaneous presentation merely due to the binaural masking level difference (BMLD).

I can't think of another way of doing simultaneous presentation that isn't a nonsense, and that one is slightly a nonsense because mostly what you're measuring is BMLD, not simultaneous vs sequential.

Cheers,
David.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #39
In light of the limits of serial audio testing raised by the OP, I'm pondering two possible conclusions:

"We've reached the limits of differences the subjects can hear"

vs.

"We have reached the limits of resolution for using human memory as the measuring instrument in an AB test".

Though the OP's point is generating some discussion, it seems to be sidestepping his basic criticism (which seems logical, particularly when considering current knowledge about perception).

Thoughts? 

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #40
I see a distinction without a difference.  Subjective interpretation requires memory.  If memory is suspect then so are the impressions born from it.

This was already addressed, was it not?

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #41
I see a distinction without a difference.  Subjective interpretation requires memory.  If memory is suspect then so are the impressions born from it.

This was already addressed, was it not?


I guess not in the way I was hoping or expecting from the dbt experts.  (Great forum, by the way.  I'm not trolling, just trying to learn and generate some discussion.  Hope this horse ain't dead.) 

DBT fails to address the "unreliable memory" problem in any way, as it relies on such for it's data input.  The tests indicate that gross differences are clearly audible.  As those differences shrink, more and more fail the test.  When you reach the useful resolution of a memory based AB test, most people fail.  But then which conclusion posted above should be drawn from the results?  It seems an important distinction, as the validity of the method is assumed in the first while the second one explicitly acknowledges it's (possible, likely?) limits.






Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #42
So you are exploring the possibility that the reason no difference is reported is not because we cannot hear the difference, but because we cannot remember it for a few seconds.

If we have forgotten it before we have even drawn another breath, how do we subsequently know that we ever heard it?

Cheers,
David.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #43
So you are exploring the possibility that the reason no difference is reported is not because we cannot hear the difference, but because we cannot remember it for a few seconds.


Second clause mainly.  I don't want you or anyone to read my posts as any sort of apologia to the audiophile crowd.  I'm more curious if the tools being used to support claims are properly calibrated, so to speak.  The serial listening criticism seems to have teeth, even if it's off in the academic weeds for all practical purposes.

If we have forgotten it before we have even drawn another breath, how do we subsequently know that we ever heard it?


Now you're getting a bit too far into the weeds.  I did not want an epistemic discussion of what we know, I simply wanted to explore what seems like a legit criticism of dbt from those in the know.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #44
Now you're getting a bit too far into the weeds.  I did not want an epistemic discussion of what we know, I simply wanted to explore what seems like a legit criticism of dbt from those in the know.


Generally when ABXing, you can switch rapidly (even instantly) between samples, so if you're going to make this argument you would be diving into the weeds and arguing that an instant is too long.

I think mainly this is just an argument against setting up tests poorly such that subjects cannot easily switch between samples.  With proper setup, this should not be an issue.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #45
People who look for reasons to doubt DBTs generally believe they can hear an audible difference sighted but cannot hear an audible difference blind. They are trying to discover some difference in the testing methodology (other than sighted-vs-blind) to justify their belief that they really hear a real audible difference. They want this other difference in testing methodology to be the reason the believed audible difference vanishes in blind testing, rather than accepting that it vanishes simply due to the blinding.

In both kinds of tests, people need to rely on their memory. If there is any difference, it is that they need to rely on memory less so in some DBT designs (as saratoga says), not more. Hence reliance on memory is not this "other" testing methodology difference that they seek.

In short, I think you're barking up the wrong tree.

Cheers,
David.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #46
People who look for reasons to doubt DBTs generally believe they can hear an audible difference sighted but cannot hear an audible difference blind. They are trying to discover some difference in the testing methodology (other than sighted-vs-blind) to justify their belief that they really hear a real audible difference. They want this other difference in testing methodology to be the reason the believed audible difference vanishes in blind testing, rather than accepting that it vanishes simply due to the blinding.

In both kinds of tests, people need to rely on their memory. If there is any difference, it is that they need to rely on memory less so in some DBT designs (as saratoga says), not more. Hence reliance on memory is not this "other" testing methodology difference that they seek.

In short, I think you're barking up the wrong tree.

Cheers,
David.


I'm not one who claims to hear differences, nor do I really care much about those that do.  What I'm trying to discuss is not people suffering delusions, but the validity and reliability of a test based on human sensory impressions/conscious awareness/memory as the source of data. 

I guess I like my data to be more raw and empirical.  Brains are really amazing biological devices, but they make for piss poor lab equipment.

 

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #47
How can something that serves to make distinguishing two samples more difficult also serve as a tell?

When the click noise has not been verified, pre-test, to be uniform in its sound quality (and/or timing) when transitioning to either A or B; then it can act as a tell.

Arnold B. Krueger mentions this problem can exist in hardware ABX testing and personally being able to reliably distinguish the sound of the QSC ABX switcher transitions, at least in the absence of music playing, here. He also mentions it occasionally can be an issue with PCABX (at least when using certain computers) in that post.

A slightly different but equally problematic issue is when the click sound doesn't immediately identify the signal itself, however is does indicate to the listener if the transition was to the same signal vs. the alternate signal [For example, the artifact click sound switching from A to X, where X = A, vs. the sound of A to X where instead X = B].

Kees de Visser, who started that thread, mentions a difference in the click sounds (and gives some examples for us to listen to) in software ABX testing, in this post. His belief at the time, if I understood correctly, was that it was random in nature [a good thing] however his response to a question here seemed to suggest otherwise to me.

The unrelated reason why we ideally don't want clicks at all, is because the presence of any other obtrusive sound, during the transition period, be it a blast from a bull horn, an annoying loud beep, or even just a loud click/thump can muck up our short term memory for things like loudness (or I would assume, also tonality). This I would assume is rather self evident. Sorry if I wasn't more clear about the "tell" part.

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #48
Except comparing two mono signals simultaneously, one fed to each ear via headphones. A niche pass-time, but possible.
This generally won't work because human body is rarely perfectly symmetrical (similar for headphones). Left and right ear may deteriorate (be damaged) at differing rates etc.
For example, when I activate the channel swap plugin in Foobar2000, it's not just side-swap for me. It's likely because I recently damaged my right ear a bit in a gym (we were dropping the weight carelessly while dead-lifting).

Biophysics, Limitations of Shannon and Issues with ABX Testing

Reply #49
I guess I like my data to be more raw and empirical.  Brains are really amazing biological devices, but they make for piss poor lab equipment.


We do not want raw data or data that simply demonstrates that the subject has received a stimulus from the outside world. There are things that we perceive, but that we aren't conscious about that perception (for example, when paying attention to something complex, we might not be conscious of small changes around us), and there are actions we do which we don't control them directly (like heart beating or breathing, although we can partially stop the second one).

As such, we rely on what the subject is able to tell that he has perceived. Since the question is made just after the perception has been received, the idea of forgeting it seems absurd (in a sane individual), and, at much, we can wonder how accurate the communication of what he has perceived is. That is not much different than calibrating a device. (except there are more variables).