This is why when I designed my blind amp testing of my audiophile, expert listener friend....

, I allowed him to decide almost EVERY single aspect of the test:

My guy received no correct answer feedback, mid-test, but never asked for any either. I would have allowed it had he asked, but I think he would have been shooting himself in the foot by doing so, since his results were barely different than random chance, and seeing that, mid test, might have potentially upset him.

Maybe testees could have done better than ~60% had they used Proper protocol.

A certain amount of stress is beneficial.

Quote from: 2Bdecided on 2014-11-12 21:27:50

Back on topic: for those who missed it, a test of filter audibility has already been attempted right here on HA...
http://www.hydrogenaud.io/forums/index.php?showtopic=68524

Any new takers?

Cheers,
David.

foo_abx 1.3.4 report
foobar2000 v1.3.5
2014/11/13 08:40:05

File A: C:\Users\Amir\Music\HA Forum Tests\limehouse\limehouse_maximum_phase_100.wav
File B: C:\Users\Amir\Music\HA Forum Tests\limehouse\limehouse_reference.wav

08:40:05 : Test started.
08:40:37 : 00/01  100.0%
08:42:19 : 00/02  100.0%
08:43:22 : 00/03  100.0%
08:44:21 : 01/04  93.8%  <--- Difference found.
08:45:14 : 02/05  81.3%
08:45:21 : 03/06  65.6%
08:45:34 : 04/07  50.0%
08:45:43 : 05/08  36.3%
08:45:52 : 06/09  25.4%
08:46:00 : 07/10  17.2%
08:46:10 : 08/11  11.3%
08:46:20 : 09/12  7.3%
08:46:29 : 10/13  4.6%
08:46:39 : 11/14  2.9%
08:46:51 : 12/15  1.8%
08:47:00 : 13/16  1.1%
08:47:10 : 14/17  0.6%
08:47:18 : 15/18  0.4%
08:47:26 : 16/19  0.2%
08:47:34 : 17/20  0.1%
08:47:42 : 18/21  0.1%
08:47:49 : 19/22  0.0%
08:47:55 : Test finished.

----------
Total: 19/22 (0.0%)

And here is the same using the new Foobar ABX plug-in with signatures:

2014-11-13 09:16:06

File A: limehouse_maximum_phase_100.wav
SHA1: 722dc26db8d4ce666dc03875b2c8d4570d22b521
File B: limehouse_reference.wav
SHA1: e8ad96830d23cad4bba5bf822ce875ae452b9e7c

Output:
DS : Primary Sound Driver

09:16:06 : Test started.
09:16:48 : 01/01
09:16:56 : 02/02
09:17:04 : 03/03
09:17:14 : 04/04
09:17:21 : 05/05
09:17:29 : 06/06
09:17:38 : 07/07
09:17:45 : 08/08
09:17:52 : 09/09
09:18:02 : 10/10
09:18:08 : 11/11
09:18:14 : 12/12
09:18:20 : 13/13
09:18:28 : 14/14
09:18:36 : 15/15
09:18:42 : 16/16
09:18:42 : Test finished.

----------
Total: 16/16
Probability that you were guessing: 0.0%



-- signature --
5b42b06c414b6ba77a3998695bf119a2d57663c0

Instead of hypotheticals let's look at real results of a real sample:

File A: C:\Users\Amir\Music\AIX AVS Test files\Mosaic_A2.wav

I was able to positively identify a revealing segment and complete the test successfully.

Without that feedback I could not determine that and stay with that segment.  Doing this in trial mode does not work because once you think you have found the difference, you have to go and run the test again and by then you may forget what you had heard.  The newer foobar abx plug-in makes that near impossible anyway because there is no help with re-selection of the precise segment.

Another variation is second guessing yourself which is a serious, serious problem.  You identify a difference and you listen and get a bunch of trials right.  Without feedback you may wonder, "what if I am getting this wrong?"  That is all that is needed to change the perception you had of the difference.  Placebo works both ways.  It can easily erase differences or make them sound different.  Without feedback you would then get a bunch of trials wrong.  With feedback you would know that you got off track and get back on and see confirmation of that in correct answer after correct answer.

Per above, it is only a "good idea" if you want to force more negative outcomes.

Quote from: amirm on 2014-12-01 19:23:37

Instead of hypotheticals let's look at real results of a real sample:

Yeah, a real sample with a couple milliseconds delay in one of the files ... good job showing off false positive ABX logs, amirm.

Maybe testees could have done better than ~60% had they used Proper protocol.

Nope.  This is David's test: http://www.hydrogenaud.io/forums/index.php?showtopic=68524

foo_abx 2.0 beta 6 report
foobar2000 v1.3.5
2014-12-01 23:05:37

File A: Mosaic_A2.wav
File B: Mosaic_B2.wav

Output:
DS : Primärer Soundtreiber

23:05:37 : Test started.
23:06:22 : 00/01
23:06:33 : 01/02
23:06:39 : 02/03
23:06:50 : 03/04
23:06:57 : 04/05
23:07:04 : 05/06
23:07:09 : 06/07
23:07:14 : 07/08
23:07:19 : 08/09
23:07:29 : 09/10
23:07:40 : 10/11
23:07:45 : 11/12
23:07:50 : 12/13
23:07:55 : 13/14
23:08:11 : 14/15
23:08:17 : 15/16
23:08:17 : Test finished.

 ---------- 
Total: 15/16
Probability that you were guessing: 0.0%

 -- signature -- 
03091b9caae2455d289845ccd68536215f94b3fd

Clearly Stuart is right in the stress that triple stimulus creates when differences are small.

Quote from: Chu Gai on 2014-12-01 19:36:20

A certain amount of stress is beneficial.

  Taking a test by definition is somewhat "stressful" however we generally want to minimize any additional stress as best we can, otherwise any failure to hear distinctions can be dismissed as being due to the "added stress".

This is why when I designed my blind amp testing of my audiophile, expert listener friend [to settle a small bet with 2:1 odds in his favor] I allowed him to decide almost EVERY single aspect of the test: He picked to the who, what ,where, when, why, and how of the entire test. My only provisions were that the number of trials he selected must show statistical significance to my satisfaction [we ended up agreeing on 16 trials total, >12 correct to win] and since he was a part time recording engineer he wasn't allowed to use his own, private recordings and had to limit himself to any commercially released CDs or SACDs of his choosing. [Of course the amps weren't allowed to be driven beyond their safe operational range, at any time, and were level matched.]

Forms of possible added stress I successfully avoided, which this Stuart et al. paper's listeners might have theoretically complained about, include:

- test listeners weren't in control of what music was selected

- test listeners weren't allowed to practice with the music and gear for an indefinite period of time, of their choosing, before the test

- test listeners weren't in control of the test transition points/segments

- test listeners weren't in control of what switching methodology was used

- test listeners, I assume, were told when and where to show up for testing, whereas my guy picked both

- test listeners didn't select the room

- test listeners didn't select the speakers used and other gear.

My guy received no correct answer feedback, mid-test, but never asked for any either. I think I would have allowed it had he asked prior to the start, but I think he would have been shooting himself in the foot by doing so [at least in retrospect], since his results were barely different than random chance in the end, and seeing that he was only guessing correctly about 50% of the time, mid test, might have potentially upset him and put him in a bad mood.

Quote from: amirm on 2014-12-01 21:27:14

Nope.  This is David's test: http://www.hydrogenaud.io/forums/index.php?showtopic=68524

Nope. Look at your own post: here.

I clearly commented on that one, did I not? Or do you now copy-paste posts and re-post with different logs? I wouldn't be surprised..

Who knows what topic you are on.

But congratulation for posting your first ABX test in this thread.  It was not so bad, was it?  Now you can speak from experience.

Now do David's test with the old and new plug-in so that when you can't follow mzil's cheat, you can produce the same outcome with the same level of stress.  I have provided my results already.

BTW, how are you and mzil creating log files that don't have the hash of the input files?

Consider this excerpt from Malcolm Gladwell's book, Blink.

http://nemaloknig.info/read-147015/?page=24

While it uses law enforcement and sports to make its point, I believe it's equally applicable to other endeavors. Sharpening one's concentration to the task at hand while minimizing distractions seems beneficial.

If peak playback level at 16 bit resolution is in fact 120dB SPL, you )(*&(*& well should be able to hear the lack of dither in a piano decay.
You should also be able to hear the difference between TPDF and RPDF dither, too, via noise modulation.

On the other hand who in the name of Monty Python would listen at that level? :horrors:

For that matter, in a decent listening room, someone ought to be able to hear the (*&*(&(*&( dither as well.

Which leads to the question: What in the heck are these people doing? Trying to blow up their equipment, or destroy their hearing, or what?

Quote from: amirm on 2014-12-01 19:23:37

Clearly Stuart is right in the stress that triple stimulus creates when differences are small.

No one is making you listen to B in an ABX test.

No this is not a life-threatening situation although one read of xnor's replies you think it is.

3. Frustration. This is what we are calling "stress" for the most part.

Clearly Stuart is right in the stress that triple stimulus creates when differences are small.

It was also possible that listeners would tire during testing sessions; listeners were
able to proceed at their own pace, with the only restriction being that breaks could not be taken in the
middle of a block.
The training and test blocks were completed in under four hours by all listeners (not including breaks).

How often do you think someone has taken the ABX test as AX?

Quote from: Chu Gai on 2014-12-01 22:23:51

Consider this excerpt from Malcolm Gladwell's book, Blink.

http://nemaloknig.info/read-147015/?page=24

While it uses law enforcement and sports to make its point, I believe it's equally applicable to other endeavors. Sharpening one's concentration to the task at hand while minimizing distractions seems beneficial.

  You are drawing an analogy from this: "This is how the human body reacts to extreme stress, and it makes sense. Our mind, faced with a life-threatening situation, drastically limits the range and amount of information that we have to deal with" to what we are doing? 

No this is not a life-threatening situation although one read of xnor's replies you think it is.

The feelings that are involved are:

1. Fear.  Fear of losing.  These tests are not created in a "safe" situation.  One outcome can lead to your ridicule by the most obnoxious people on the planet. 

2. Suspicion.  That the other side has cooked the test to make a fool of you.  This one is deadly and I have fallen victim to it many times.  I am taking the test, and I know I have heard the difference.  But then think, "what if he is tricking me?"  Fear kicks in and I vote the opposite of what I heard, only to find out my first instinct was correct.

3. Frustration.  This is what we are calling "stress" for the most part.  We are having to remember one complex set of notes and freeze it in our mind, and then perform an aural image match.  When differences get small this is not easy to do at all.  Even when testing big differences such as Harman's double blind speaker tests, I found myself in a tough spot trying to remember what the last speaker sounded like. 

We are asking a human to be a machine with precision.  This is never easy.

4. Anticipation.  This is the point I made in my prior post with new version of ABX.  As the trials counted up, you could just hear my heart rate increase.  What if I get to the end and I have to repeat this stupid test again? 

A little story about my younger years.  I was always good with math.  Not perfect but one step below.  I remember one time getting 4 or 5 math problems to solve as a homework assignment.  I sat there and figured them right out.  The next day I come to the class and was amazed that I was the only one to have solved all of them!  The excitement did not last long.  The final exam comes and what is on the test?  A couple of the same problem.  Momentary excitement turned into sheer panic as I could not solve either one!  I tried everything.  I wrote pages and pages of stuff and at the end could not remember how to solve the problem.  Barely got a passing grade.

I did not become dumb between the home work assignment and final exam.  Clearly something about being tested caused me to all of a sudden get frustrated, worried about running out of time, etc.

This is why we need to do everything in our power to make the test simpler for people to take.  Expert listeners and revealing content does that.

Maybe testees could have done better than ~60% had they been given the option to audition A and/or B when making a selection for X.

I'm sure others do it too

Difference so obvious, an ABX would be practically effortless.

But it's not.  It's not in Stuart's test either.

I'm sure it would still have garnered those AES awards.

The fact is, Amir, that Stuart and Co. offered their critique of ABX with no citations of evidence from others' work.

I found that quite peculiar, especially in an Introduction section of a research paper.

It's just, like, their opinion, man.

cognitive *load*, indeed