Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How do you listen to an ABX test? (Read 342658 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

How do you listen to an ABX test?

Reply #50 –
Quote from: SoundAndMotion on
Quote from: Arnold B. Krueger on
Quote from: SoundAndMotion on
"Too high" means wrong.

This isn't about right or wrong.

Well, you are very experienced with ABX testing. If I were to tell you I did a test with the old AIX AVS files (with unmatched levels), and I said "I know the levels are different, but I took that into account before responding". Would you tell me I did it "wrong" or would you be more gentle.


You've changed the context of the discussion pretty dramatically for no apparent reason.

Quote from: SoundAndMotion on
Quote from: Arnold B. Krueger on
Quote
You may have an unusual ability to estimate and therefore abstract the SPL, but most people can't.

Decades of training were required. Not everybody wants to do that, and not everybody has the opportunity.

Sounds like you have "golden ears". No problem with TOS #8 though, I'd guess, because you make no claim of quality. I'm just happy that you would agree that Peter Aczel's lie #10 is no lie. He says: "The Golden Ears want you to believe that their hearing is so keen, so exquisite, that they can hear tiny nuances of reproduced sound too elusive for the rest of us." I don't know about "Golden Ears"(capitalized) , but normal variability, plus as you point out, training, do make some people more sensitive than the rest of us.
;-) Don't get angry. I'm playing with you a little. I know you use  "Golden Ears" as a derogatory term, and I just want to underscore your example of yourself, as someone who hears certain characteristics better than most. :-)


Let's look at what I actually wrote:

Quote from: Arnold B. Krueger on
For a few tenths of a dB I do need to hear the samples very close together. For several dB, I can walk into a room stone cold and guess the SPL value with a reasonable tolerance.


I guess that you are so completely unfamiliar with audio that you don't know that claiming the audiblity of differences on the order of "Several dB" is consistent with current scientific knowledge about hearing and therefore is outside the area of concern of TOS #8.

Letsee, "several dB" could be 6 dB. Here is an ABX test log for two files whose only difference is that their  level is different by 6 dB:

foo_abx 2.0 beta 4 report
foobar2000 v1.3.5
2015-03-27 08:53:02

File A: tuttabella_org- 6 dB.flac
SHA1: 737de5dccbd485ca54653eb5183b4270cd496048
File B: tuttabella_org.flac
SHA1: b1875cd08a96b24c300f7a2e4907d995b857ff99

Output:
DS : Primary Sound Driver

08:53:02 : Test started.
08:53:22 : 01/01
08:53:31 : 02/02
08:53:44 : 03/03
08:53:50 : 04/04
08:53:55 : 05/05
08:54:02 : 06/06
08:54:28 : 07/07
08:54:36 : 08/08
08:54:45 : 09/09
08:54:54 : 10/10
08:54:58 : 11/11
08:55:04 : 12/12
08:55:12 : 13/13
08:55:19 : 14/14
08:55:23 : 15/15
08:55:54 : 16/16
08:55:54 : Test finished.

----------
Total: 16/16
Probability that you were guessing: 0.0%

-- signature --
7fdf7be356ef6720a807ad04a13708b2f1ab579b

Compliance with TOS 8 completed, FWIW.

BTW I performed this test by simply running the X's. I never compared any X to A or B, and only listened to A and B once at the beginning to confirm that they had the expected level difference.

How do you listen to an ABX test?

Reply #51 –
Quote from: Arnold B. Krueger on
Quote from: SoundAndMotion on
Quote from: Arnold B. Krueger on
Quote from: SoundAndMotion on
"Too high" means wrong.

This isn't about right or wrong.

Well, you are very experienced with ABX testing. If I were to tell you I did a test with the old AIX AVS files (with unmatched levels), and I said "I know the levels are different, but I took that into account before responding". Would you tell me I did it "wrong" or would you be more gentle.


You've changed the context pretty dramatically for no apparent reason than debating trade points. That in my book is trolling. Have a nice day!

I'd guess that you're not going to respond to me, but I'll say this: I have no intention of being a troll. I'm quite accustomed to challenging what people say AND being challenged, ala Socratic learning. Sorry if I have offended you... not the intent. Challenge yes; offend no.
You have a nice day, too! :-)

How do you listen to an ABX test?

Reply #52 –
Quote from: SoundAndMotion on
I'd guess that you're not going to respond to me, but I'll say this: I have no intention of being a troll.



I did respond by demonstrating a DBT that came as close to supporting my claims that I could think of.

One of the symptoms of trolling is denying the validity or even the existence of support for a point no matter how well supported, whether with logic or actual empirical evidence.

How do you listen to an ABX test?

Reply #53 –
Quote from: Arnold B. Krueger on
Quote from: SoundAndMotion on
I'd guess that you're not going to respond to me, but I'll say this: I have no intention of being a troll.



I did respond by demonstrating a DBT that came as close to supporting my claims that I could think of.

One of the symptoms of trolling is denying the validity or even the existence of support for a point no matter how well supported, whether with logic or actual empirical evidence.

What point did I deny validity? If you mean my stating that testing auditory differential threholds requires a short (<1s) interstimulus gap... I stand by that and if you want I'll provide refs this weekend. If I misunderstood or had an offensive tone or I wasn't clear... sorry, all that happens more often than I'd like.

How do you listen to an ABX test?

Reply #54 –
Quote from: SoundAndMotion on
Quote from: Arnold B. Krueger on
Quote from: SoundAndMotion on
I'd guess that you're not going to respond to me, but I'll say this: I have no intention of being a troll.



I did respond by demonstrating a DBT that came as close to supporting my claims that I could think of.

One of the symptoms of trolling is denying the validity or even the existence of support for a point no matter how well supported, whether with logic or actual empirical evidence.

What point did I deny validity?


I guess you don't read your own posts. You called it non responsive.

How do you listen to an ABX test?

Reply #55 –
Quote from: SoundAndMotion on
Certainly someone doing research as their job (e.g. Olive) would need to put in the time/expense/effort, which his job would give him. And Olive has done lots of interesting and IMO important work. Are you suggesting blind protocols are needed for personal headphone decisions? I'm familiar with their "virtual headphone" method, where they inverse filter Senn HD 518s, and play models of other headphones through them. Do you know of other methods they may have used?



It's kind of  a badly-formed question.  What is 'needed' depends on what level of knowledge/certainty you seek, and what claims you hope to make.  Most people don't really think this through.  So they audition two headphone sets hanging on the wall at their local Best Buy, decide 'this one sounds better' , think 'therefore it is better', and that's it. 

And *unless* they come on a forum like this , and claim, I tried X and Y and X is the better headphone, no one is going to care or challenge their 'method'.


Quote
OT question: why is double blind always stated, when for example, fb2k isn't double? When I use "blind" at work, it's always assumed that no cues from any source (including people) are provided, other than the controlled stimulus, be it just the subject alone (technically single, I guess), or additionally the experimenter (then double) and sometimes the person doing analysis (triple)? Just curious.



ABX is 'effectively' double blind since the app administering the test isn't human  ;>

How do you listen to an ABX test?

Reply #56 –
Quote from: Arnold B. Krueger on
I guess you don't read your own posts. You called it non responsive.

OK, I'm lost. I have read my posts. I searched the thread for "non responsive", even "responsive". I don't understand you. We're talking past each... oh well...
Have a nice day! :-)

How do you listen to an ABX test?

Reply #57 –
Quote from: SoundAndMotion on
OT question: why is double blind always stated, when for example, fb2k isn't double?


Say what?

How is FB2K ABX not a DBT?

Quote
When I use "blind" at work, it's always assumed that no cues from any source (including people) are provided, other than the controlled stimulus, be it just the subject alone (technically single, I guess), or additionally the experimenter (then double) and sometimes the person doing analysis (triple)? Just curious.


What non-audible cues do you get about the unknowns that are presented by FB2K?

Quote
Also, do you know anything about fb2k on a VM on a mac, or a similar program for a mac?


Ummm, google it?

I have no Mac and no experience with these but they claim to be DBT test coordinators like ABX on FB2K.

ABX Tester for the Mac

Lacinato ABX/Shootout-er blind testing audio software for cross platforms including the Mac

Past HA discusisons about cross-platform DBT software

How do you listen to an ABX test?

Reply #58 –
Quote from: krabapple on
ABX is 'effectively' double blind since the app administering the test isn't human  ;>

I would argue that exactly because the app is not human, "double" doesn't apply. I'd say "blind", but that's just me, and I'm burned out on arguing (especially from my iPad), so...
DBT yeah!
;-) Have a good weekend.

How do you listen to an ABX test?

Reply #59 –
Quote from: Arnold B. Krueger on
Quote from: SoundAndMotion on
OT question: why is double blind always stated, when for example, fb2k isn't double?


Say what?

How is FB2K ABX not a DBT?

Quote
When I use "blind" at work, it's always assumed that no cues from any source (including people) are provided, other than the controlled stimulus, be it just the subject alone (technically single, I guess), or additionally the experimenter (then double) and sometimes the person doing analysis (triple)? Just curious.


What non-audible cues do you get about the unknowns that are presented by FB2K?

"Double" in DBT refers to both the subject and experimenter not knowing what X is. The experimenter can involuntarily give non-audible cues to the subject. If I'm alone with fb2k, there's no both to doubly blind. The way my colleagues and I use "blind" and "double blind" at work, fb2k is not DBT. Semantics. Not worth pursuing.
Thanks for the mac tip. I know how to google, but wanted recommendations, not a list. In post 38(?), I mentioned I have ABXTester for mac; it's not so full featured as fb2k. I got a very useful PM for fb2k on mac. Thanks though.
 

How do you listen to an ABX test?

Reply #60 –
Quote from: SoundAndMotion on
Quote from: krabapple on
ABX is 'effectively' double blind since the app administering the test isn't human  ;>

I would argue that exactly because the app is not human, "double" doesn't apply.


I think you misunderstand what blinding means.  Take a look at the wiki:

http://en.wikipedia.org/wiki/Blind_experim...le-blind_trials

This is a double blinded test because it is shielded from bias by both parties in the test. 

Quote from: SoundAndMotion on
Semantics. Not worth pursuing.


This is not a semantic argument, and it is worth pursuing.  Specifically, understanding why double blinded tests like ABX are used is important.

How do you listen to an ABX test?

Reply #61 –
Quote from: saratoga on
Quote from: SoundAndMotion on
Quote from: krabapple on
ABX is 'effectively' double blind since the app administering the test isn't human  ;>

I would argue that exactly because the app is not human, "double" doesn't apply.


I think you misunderstand what blinding means.  Take a look at the wiki:

http://en.wikipedia.org/wiki/Blind_experim...le-blind_trials

This is a double blinded test because it is shielded from bias by both parties in the test. 

Quote from: SoundAndMotion on
Semantics. Not worth pursuing.


This is not a semantic argument, and it is worth pursuing.  Specifically, understanding why double blinded tests like ABX are used is important.

Thank you. Thank you. Thank you.
That is exactly how I would define "blind" "to blind" "single" "double" and "triple"!!!! Where I bold your quote, who are the 2(both) parties? Keep in mind, when it's you alone with the computer, there is no researcher.
From the article:
Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments, since software may not cause the type of direct bias between researcher and subject.Development of surveys presented to subjects through computers shows that bias can easily be built into the process. Voting systems are also examples where bias can easily be constructed into an apparently simple machine based system. In analogy to the human researcher described above, the part of the software that provides interaction with the human is presented to the subject as the blinded researcher, while the part of the software that defines the key is the third party. An example is the ABX test, where the human subject has to identify an unknown stimulus X as being either A or B.

How do you listen to an ABX test?

Reply #62 –
Quote from: SoundAndMotion on
"Double" in DBT refers to both the subject and experimenter not knowing what X is. The experimenter can involuntarily give non-audible cues to the subject. If I'm alone with fb2k, there's no both to doubly blind.



Now I get our communication problem. What double talk!  The word experimenter was unnecessarily added and it  creates a problem. If I wanted to spend the rest of my life with useless arguments about semantics...

Many common definitions don't make this trivial mistake. For example: Here is what Wikipedia has to say:

"A blind or blinded experiment is an experiment in which information about the test that might lead to bias in the results is concealed from the tester, the subject, or both until after the test."

Another way to look at it is that if you are alone with FB2K, the experimenter's role is filled by the software. The fact that it is not a living breathing homo sapiens is irrelevant to most people. Another case where the implementation details are irrelevant and performance is all-important.

The problem of hardware and software stand-ins for the experimenter that give reliable cues to the identity of the unknown of their own making has been around for quite some time.  My original ABX box of 1977 was pretty noisy when it changed state, but the noise followed no discernable pattern. It did not compromise the blindness of the test. QSC's ca. 1990s ABX Comparator could be aced with no other equipment attached. That's a serious non-semantic problem.

How do you listen to an ABX test?

Reply #63 –
Quote from: SoundAndMotion on
From the article:
Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments, since software may not cause the type of direct bias between researcher and subject.Development of surveys presented to subjects through computers shows that bias can easily be built into the process. Voting systems are also examples where bias can easily be constructed into an apparently simple machine based system. In analogy to the human researcher described above, the part of the software that provides interaction with the human is presented to the subject as the blinded researcher, while the part of the software that defines the key is the third party. An example is the ABX test, where the human subject has to identify an unknown stimulus X as being either A or B.


I'm not even sure how to respond to such a bizarre post.  Are you trying to argue that ABX is not double blind?  If so, I'd suggest that you are REALLY confused on what blinding means.

How do you listen to an ABX test?

Reply #64 –
Quote from: saratoga on
Quote from: SoundAndMotion on
From the article:
Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments, since software may not cause the type of direct bias between researcher and subject.Development of surveys presented to subjects through computers shows that bias can easily be built into the process. Voting systems are also examples where bias can easily be constructed into an apparently simple machine based system. In analogy to the human researcher described above, the part of the software that provides interaction with the human is presented to the subject as the blinded researcher, while the part of the software that defines the key is the third party. An example is the ABX test, where the human subject has to identify an unknown stimulus X as being either A or B.


I'm not even sure how to respond to such a bizarre post.  Are you trying to argue that ABX is not double blind?  If so, I'd suggest that you are REALLY confused on what blinding means.



It's called misdirection. Whether the test is computer-controlled or not is given great importance by mentioning it first, but in fact the implementation of the test is irrelevant. How the test works is the most important thing.

The problem of cuing the test subject with the identity of the unknown has at least been around since Clever Hans, the talking horse, No computers required!

How do you listen to an ABX test?

Reply #65 –
Quote from: saratoga on
Quote from: SoundAndMotion on
From the article:
Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments, since software may not cause the type of direct bias between researcher and subject.Development of surveys presented to subjects through computers shows that bias can easily be built into the process. Voting systems are also examples where bias can easily be constructed into an apparently simple machine based system. In analogy to the human researcher described above, the part of the software that provides interaction with the human is presented to the subject as the blinded researcher, while the part of the software that defines the key is the third party. An example is the ABX test, where the human subject has to identify an unknown stimulus X as being either A or B.


I'm not even sure how to respond to such a bizarre post.  Are you trying to argue that ABX is not double blind?  If so, I'd suggest that you are REALLY confused on what blinding means.

Wow, this is exhausting! Read the article. Read what I say in my posts. I never argue all ABXs aren't DBTs.
ABX is a type of experiment. A and B can be files, cables, amps, etc. If you are using files, you can use fb2k!!! If you have a human experimenter in the room conducting the experiment, you can do a DBT. If you alone use fb2k as your method of doing an ABX without an experimenter, YES I'M SAYING IT'S NOT A DBT. THAT DOES NOT MEAN ALL ABX's aren't, just fb2k used alone. And the Wikipedia article agrees!!
All dogs are animals; not all animals are dogs, but some are. Not all DBTs are ABXs but some are and not all ABXs are DBTs but some are.

How do you listen to an ABX test?

Reply #66 –
Quote from: SoundAndMotion on
Quote from: saratoga on
Quote from: SoundAndMotion on
From the article:
Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments, since software may not cause the type of direct bias between researcher and subject.Development of surveys presented to subjects through computers shows that bias can easily be built into the process. Voting systems are also examples where bias can easily be constructed into an apparently simple machine based system. In analogy to the human researcher described above, the part of the software that provides interaction with the human is presented to the subject as the blinded researcher, while the part of the software that defines the key is the third party. An example is the ABX test, where the human subject has to identify an unknown stimulus X as being either A or B.


I'm not even sure how to respond to such a bizarre post.  Are you trying to argue that ABX is not double blind?  If so, I'd suggest that you are REALLY confused on what blinding means.

Wow, this is exhausting! Read the article. Read what I say in my posts. I never argue all ABXs aren't DBTs.


You don't have to argue that all ABX's aren't DBTs.  You can create that impression with sentences like this one:

"Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments"

I just explained to you how that is commonly interpreted as saying what you now say you don't want to say, and you didn't respond to the post so I take it that you dismissed it.  This post looks also looks like you are dismissing the other post.

Giving you the benefit of the doubt, I think you may have wanted to say:

Poorly implemented experiments are sometimes  erroneously referred to as double-blind experiments.

That is not too interesting because it is a truism.

Quote
ABX is a type of experiment. A and B can be files, cables, amps, etc. If you are using files,


or pudding, potato chips, soda pop, or beer. They can be pornographic pictures.  They can be anything that can be perceived and practically manageable.

Quote
you can use fb2k!!!


Thanks for that!

Quote
If you have a human experimenter in the room conducting the experiment, you can do a DBT.


Just put him out of sight and keep him quiet.

Quote
If you alone use fb2k as your method of doing an ABX without an experimenter, YES I'M SAYING IT'S NOT A DBT.


That seems to be incorrect to the point of being bizarre.

Quote
THAT DOES NOT MEAN ALL ABX's aren't, just fb2k used alone. And the Wikipedia article agrees!!


Which Wikipedia article?

Please explain.

How do you listen to an ABX test?

Reply #67 –
FB2k ABX is double blind. The true identities of the UUTs are not conveyed directly to the test subject, the listener, through any of their their own senses nor indirectly (and possibly inadvertently) through the robotic test administrator.

How do you listen to an ABX test?

Reply #68 –
Quote from: mzil on
FB2k ABX is double blind. The true identity of the UUTs are not conveyed directly to the test subject, the listener, through their own senses nor indirectly (and possibly inadvertently) through the robotic test administrator.


However a FB2K or similar test can unblind itself.

Let's say that you are comparing two files whose CPU load for decoding varies quite a bit from each other, and one turns on the CPU fan or makes it run louder.

This is a potential problem with laptops, many of which need to run their fans all the time and throttle the CPU and fan pretty closely to save energy and minimize size and weight.

Failing weird stuff like that, the general rule is that a FB2K test is utterly double blind.

How do you listen to an ABX test?

Reply #69 –
Quote from: SoundAndMotion on
Quote from: saratoga on
Quote from: SoundAndMotion on
From the article:
Computer-controlled experiments are sometimes also erroneously referred to as double-blind experiments, since software may not cause the type of direct bias between researcher and subject.Development of surveys presented to subjects through computers shows that bias can easily be built into the process. Voting systems are also examples where bias can easily be constructed into an apparently simple machine based system. In analogy to the human researcher described above, the part of the software that provides interaction with the human is presented to the subject as the blinded researcher, while the part of the software that defines the key is the third party. An example is the ABX test, where the human subject has to identify an unknown stimulus X as being either A or B.


I'm not even sure how to respond to such a bizarre post.  Are you trying to argue that ABX is not double blind?  If so, I'd suggest that you are REALLY confused on what blinding means.

Wow, this is exhausting! Read the article. Read what I say in my posts. I never argue all ABXs aren't DBTs.
ABX is a type of experiment. A and B can be files, cables, amps, etc. If you are using files, you can use fb2k!!! If you have a human experimenter in the room conducting the experiment, you can do a DBT. If you alone use fb2k as your method of doing an ABX without an experimenter, YES I'M SAYING IT'S NOT A DBT. THAT DOES NOT MEAN ALL ABX's aren't, just fb2k used alone. And the Wikipedia article agrees!!


You have absolutely no idea what you are talking about and should immediately stop arguing, read very carefully how an ABX test works, and then come back here once you understand what you are attempting to talk about. 

How do you listen to an ABX test?

Reply #70 –
Very entertaining read. Lets see if some pinhead like me now learns what Socratic learning is about.
Is troll-adiposity coming from feederism?
With 24bit music you can listen to silence much louder!

How do you listen to an ABX test?

Reply #71 –
If FB2K ABX is single blind, then show me the experimenter who knows the correct answer and is inadvertently passing that information to the subject through subtle means.

How do you listen to an ABX test?

Reply #72 –
Regarding ABX testing on Mac OS X, I've done mine with foobar2k and foo_abx 2.0 running under wine (http://winebottler.kronenberg.org) with no issues.

How do you listen to an ABX test?

Reply #73 –
Quote from: Arnold B. Krueger on
However a FB2K or similar test can unblind itself.


It would be interesting if some clever person discovered they could use their harddrive activity LED's flash pattern to ID X and Y! Since foobar seems to load both A and B into its own, private memory area, prior to testing, this would seem rather unlikely to me though. [Still, a patch of black tape over that light might be in order for formal testing, just to be dead sure.]

How do you listen to an ABX test?

Reply #74 –
My head is spinning. I fell down a rabbit hole and the chess pieces have arisen to tell me where to go. There's a hookah smoking caterpillar, the white knight is talking backwards and logic and proportion have fallen sloppy dead....
I've got to get out and here's my plan.
I contend that there exist 3 groups.
One group has been reading audio forums for a long time and has seen "DBT" perhaps hundreds of times. Whether each member is working with the correct definition, I can't say. But whatever their definition is, they are confident they have been reading and writing it correctly. If a wild-eyed-weirdo, whose opinion they don't respect, tells them their definition is wrong, obviously they reject that. But the cool part is, that their confirmation bias influences their ability to read a Wikipedia article without smashing the round peg words into their square hole existing definition. No chance for change here.
The second group are our non-scientist friends, family, coworkers, who have never read an audio forum, and have never seen "DBT", except maybe in the newspaper article. I contend that if you ask them to read the Wikipedia article, or the section on blind testing in any scientific methods book, and somehow bribe them to read all my posts in this thread, that they will find no errors, deceptions, misdirections or misleading arguments in what I have said about DBTs. I would really be interested in the result, if this actually happened.
The third group is practicing, publishing scientists. If they do blind testing, they won't need to read anything. If they don't regularly do blind tests, they may need to brush up with an experimental design text. And then the same as group 2: I contend that they will find no errors, deceptions, misdirections or misleading arguments in what I have said about DBTs.
I say groups, because I contend a consensus, not one individual will help me stay out of the rabbit hole. Some of the last few posts have been surreal.
I'm not ignoring any posts, I have just been travelling all day and will now spend some time with my family. Mr Krueger, I'll respond tomorrow. Sorry.
Saratoga, your lecturing me in post 70 above has me ROTFLMAO.
FYI, the Wikipedia article is linked in Saratoga's post number 61 above.