Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Controlled testing (Read 23597 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Controlled testing

Reply #25
Correct me if I'm wrong but I read an article by Sean Olive where (if I'm understanding what he says) he says that before 1994 there were no published scientific studies supporting DBT.


http://www.aes.org/e-lib/browse.cfm?elib=3839

High-Resolution Subjective Testing Using a Double-Blind Comparator

(JAES paper - peer reviewed, etc.)

"A system for the practical implementation of double-blind audibility tests is described. The controller is a self-contained unit, designed to provide setup and operational convenience while giving the user maximum sensitivity to detect differences. Standards for response matching and other controls are suggested as well as statistical methods of evaluating data. Test results to date are summarized."

The nearly identical AES conference paper was given the previous year in 1981.

I am informed that if one follows the Journal of the Acoustical Society of America, one would find papers describing earlier controlled testing relating to hearing. BTW the hearing folks use something they call an ABX test which is a little different from the one described above.

Controlled testing

Reply #26
Quote
That's how Science works - nothing about it is totally consistent. ;-)


So you accept that sighted listening (excluding speakers) may or may not result in a null difference under controlled testing?


Of course. Sighted listening can have random results due to one or more of the many uncontrolled influences. Doesn't happen very often because the strongest influences usually produce consistent results.

Quote
Quote
The operative word appears to be give, which ordinarily implies altruism on the part of the provider and an exploitative situation with the person making the demands.


Perhaps you did not take what I said in the spirit that I intended. What I meant was, it would be great to have a compiled list of CDP .. or amp or cable tests to serve as ammunition in these arguments - to have convincing evidence.


Such lists exist. A little use of google turns them up.

Quote
I've heard on a number of forums where proponents of DBT will say something to the effect of "well, show me a single positive DBT of a CDP" or something like that. Or show me a positive result of speaker cable ..or amplifiers ... etc. If at least a few positive tests exist (excluding speakers here) for everything else then what does that mean?


I know how to do a positive ABX test involving a CD player. In fact one is described in this article: Masters, Ian G 'Do All CD Players Sound the Same?' Stereo review, Jan 1986, pg 50-57. I was part of the listening panel and I reviewed the test setup and approved of it.  The player had  measurable faults that were obvious enough to make an audible  difference with very critical program material.

Controlled testing

Reply #27
If you want the most sensitive results, you have to have some kind of fast switching available. Some authorities also want a transient-free switch which involves modulating the signal. Putting the actual switching under the control of the listener also maximizes sensitivity.


If what you are testing is someone's claim that he can tell a difference in the sound of 2 components auditioned on different days in different buildings, then you don't need fast switching.




Controlled testing

Reply #28
[What I meant was, it would be great to have a compiled list of CDP .. or amp or cable tests to serve as ammunition in these arguments - to have convincing evidence


Have you already seen this and this?

Controlled testing

Reply #29
If you want the most sensitive results, you have to have some kind of fast switching available. Some authorities also want a transient-free switch which involves modulating the signal. Putting the actual switching under the control of the listener also maximizes sensitivity.


If what you are testing is someone's claim that he can tell a difference in the sound of 2 components auditioned on different days in different buildings, then you don't need fast switching.


The above ignores a clearly stated condition: "If you want the most sensitive results".  If you audition on different days in different buildings then you don't want the most sensitive results.

Controlled testing

Reply #30
Agreed, although that doesn't help to address the placebophile argument that X is less fatiguing than Y or is more beneficial to the mental, psychological and/or physical health of the listener (yes, still confined to the realm of listening to audio).

I guess it doesn't matter since this generally boils down to metaphysical beliefs, even if the individual espousing them insists otherwise.

Controlled testing

Reply #31
Quote
Have you already seen this and this?


I saw on this site many of the DBTs are in French .. some links don't work and all the tests on the AES or Stereo Review can't be accessed. I see one CDP DBT with a failed link ... and the rest I see are in French. One failed, one even was a success ...

Controlled testing

Reply #32
If the DBT results were published at the AES does that make them peer-reviewed?

Controlled testing

Reply #33
If the DBT results were published at the AES does that make them peer-reviewed?


Everything published in the JAES as a technical paper is AFAIK still heartily peer-reviewed.

Almost anything can be said in a conference paper and they usually get published separately from the Journal.

There are a few JAES articles that some sophisticated and influential members would like to peer review err, retroactively.  Many of them relate to exceptional claims of the audibility of Transient Intermodulation or Slew Induced distortion.

Doing your own DBTs isn't exceptionally difficult nor does it take a lot of rare resources if you are willing to be pragmatic.

Controlled testing

Reply #34
^To add to that I'd like to say that having done it myself (a layman without any schooling in these disciplines), after I made the wager with my my golden-eared audiophile friend, I felt vindicated in my beliefs and now when asked, "But have you ever conducted such carefully controlled tests yourself?" I can proudly respond in the affirmative. 

It is quite empowering Yahzi. Give it a try!


 

Controlled testing

Reply #35
Quote
It is quite empowering Yahzi. Give it a try!


I want to! But first I need to carefully study the necessary steps involved.    Without an ABX comparator I would be at a disadvantage .. I would imagine. Although I meant what I said, that I would love to meet some of you guys in the near future and perhaps learn more about the testing in person and even be tested myself.

Controlled testing

Reply #36
Quote
It is quite empowering Yahzi. Give it a try!


I want to! But first I need to carefully study the necessary steps involved.    Without an ABX comparator I would be at a disadvantage .. I would imagine. Although I meant what I said, that I would love to meet some of you guys in the near future and perhaps learn more about the testing in person and even be tested myself.


Just about everybody can have an ABX Comparator on their computer - they exist as free downloads for just about every kind of computer that runs Java.

Many things of interest including just about everything related to lossy compression can be tested this way.

While it takes a step of faith in the quality of the converters that can be attached to computers, most issues that relate to hardware can also be recast as files for a software ABX  Comparator.

Here is an archive of files for use with a software ABX Comparagtor:

http://www.ethanwiner.com/aes/

Controlled testing

Reply #37
Arny, I noticed on occasion you bring up time synchronisation. Can you please explain what that entails and the importance of doing it? How does one time synch and if you don't do this, would it invalidate the blind or double blind test results?

Controlled testing

Reply #38
It's not that complicated. When you switch rapidly between two audio streams, if they are not exactly synchronized then you will hear a little "hiccup" from either a few samples being repeated, or a few being skipped, also probably a slight click as you switch. This tells you if X is matched to A or B, even if A and B are identical except for the time mismatch.

Controlled testing

Reply #39
Okay, but if the streams are not synched ... that will invalidate the test results? I understand quick switching ... but I've seen many tests that are not time synched, so I'm just asking here whether that would be enough to throw the results in the air.

Controlled testing

Reply #40
Okay, but if the streams are not synched ... that will invalidate the test results? I understand quick switching ... but I've seen many tests that are not time synched, so I'm just asking here whether that would be enough to throw the results in the air.


If you don't synchronize the streams it is possible to correctly identify them, even if they are otherwise identical.

The test isn't really blind.

Controlled testing

Reply #41
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?

Controlled testing

Reply #42
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?


Obvious anti DBT bias noted. In fact listening tests of any kind only work if you have experienced listeners. In general sighted evaluations don't work because of the unresolved problems with listener bias.

Controlled testing

Reply #43
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?


With 'testers', do you mean the person(s) who listen, or the person(s) who administer the test and collect the assessments from the former?

If you 'lure' someone inexperienced with listening into a listening test, then they might not spot differences they would if they were trained, but that is the inexperience with the 'T' part, not the 'DB' part.

If you 'lure' someone inexperienced (... dare I say clueless?) into setting up a DBT, that person may of course invalidate the test by making errors an experienced tester would avoid. Those errors could be related to double-blind framework (for example, not everyone knows what that 'double' is about), or unrelated (e.g. not matching volume, and making the wrong interpretations).

Controlled testing

Reply #44
So double blind testing only really works if you are experienced in it, i.e. having trained yourself to do it, knowing what to listen for. So if you lure inexperienced DBT 'testers' into the test you could invalidate the results that way. Surely?


If a person already claims to hear a difference between A and B,  a blind test  of that person is an excellent way to test them on that claim.

If you are a researcher and want to determine if humans could hear a difference between A and B, then training is part of the experimental protocol.

Surely you  are being purposely obtuse?

Controlled testing

Reply #45
I was just being curious and sometimes curiosity gets the better of me.  I've been told on several occasions that human auditory memory is very short. Are there any studies I can look at that can confirm this? Thanks.

Controlled testing

Reply #46
I was just being curious and sometimes curiosity gets the better of me.  I've been told on several occasions that human auditory memory is very short. Are there any studies I can look at that can confirm this? Thanks.


The topic is discussed pretty thoroughly with extensive footnotes in This Is Your Brain On Music by  Daniel Levitin

http://www.amazon.com/This-Your-Brain-Musi...n/dp/0452288525

The short answer is if the goal is to remember small subtle details, probably a second or less.  Obviously, gross details such as the name of the song or whether we liked the performance can be recalled for months or years.

I built an ABX Comparator with an adjustable delay while switching. In general my ability to distinguish small differences becomes degraded when this delay is much more than  a second.

Controlled testing

Reply #47
Thanks Arnold. Are there any other books or studies you would recommend? I've already ordered "This Is Your Brain On Music" but I would like to know if there are any papers, or research you could recommend I read in addition. Thanks again!


Controlled testing

Reply #49
Yeah I saw the thread but I see a dozen different books. How do I know that there is anything of value specifically concerning auditory memory in there? I have no idea. That's why I'm asking so I can be pointed in the right direction. I can't order all these books you know.