................
Especially those with strong pecuniary interests.

cheers,

AJ

I think by his [jkeny] logic, if some student were to fill out a true/false test in school without reading the questions and simply checking all the "true" boxes, this would prove that true/false tests "don't work".

You talk about my pecuniary interests.

So you think that my posting on here has something to do with my €500 DACs (my most expensive product).

I guess your $8,500 speakers are 17 times more likely to be a pecuniary motivation for you to post here & try to discredit anything other than speakers

Guessing "true" on every question in a true/false test, without even reading the question, is the same as guessing B on every trial of an ABX test, without really listening to X. In both instances it is the fastest way to complete a test when the only concern is simply to determine how fast a test can mechanically be complete due to the delay in being force to give a response and then click a "move on to the next trial" button.  I've completed such tests in this forum for exactly that purpose, not caring what the score was since my only purpose was to test for how quickly a test can be completed due to the physical demands of moving one's cursor from place to place and clicking when appropriate.

. Similarly for many other situations where the test isn't actually taken i.e. there is no serious attempt at listening involved. Yet all these tests are treated as valid & lumped into the null "evidence" pile.

The point is that there is no way of knowing from the published test results that they are simply just random guesses completed by a deaf monkey. Similarly for many other situations where the test isn't actually taken i.e. there is no serious attempt at listening involved. Yet all these tests are treated as valid & lumped into the null "evidence" pile.

Quote from: jkeny on 2015-04-02 19:57:51

You talk about my pecuniary interests.

Yep, you're "In the biz". The biz that ABX is very bad for.

Quote

So you think that my posting on here has something to do with my €500 DACs (my most expensive product).

Yes and the "organic" way they "sound"....free of ABX. You said so yourself.

Quote

I guess your $8,500 speakers are 17 times more likely to be a pecuniary motivation for you to post here & try to discredit anything other than speakers

Actually I just sold a $20k set...without resorting to nonsensical claims about magic power supplies, interleaving, gold alchemy and smell of flowers, etc.
Or pounding the drums of doubt/rejecting blind/ABX testing.

cheers,

AJ

Quote

. Similarly for many other situations where the test isn't actually taken i.e. there is no serious attempt at listening involved. Yet all these tests are treated as valid & lumped into the null "evidence" pile.

Nope. You don't get it. Not hearing a difference, on that day, with that person, with that system, with that song, doesn't prove anything. You can't prove a negative. But I'm not going to waste my time trying to explaining this concept to you. Bye.

Quote from: jkeny on 2015-04-02 20:37:11

The point is that there is no way of knowing from the published test results that they are simply just random guesses completed by a deaf monkey. Similarly for many other situations where the test isn't actually taken i.e. there is no serious attempt at listening involved. Yet all these tests are treated as valid & lumped into the null "evidence" pile.

Yes, exactly. And that is how it should be. The null result is the default assumption anyway, so it doesn't matter how many results are being piled in there. You see, it all makes perfect sense! 

There is no alternative anyway. Or do you have a way to look into people's heads to determine whether they were deliberately cheating or otherwise unfit?

I have no problem with anybody doing personal blind tests - it's truth for them.

When it comes to presenting such "evidence" for public dissemination as "evidence" for a particular stance then more stringent criteria & examination are required.

I never posted that here - you pulled it off my website & posted it here.

Much like your statement "Our products reflect the philosophy that loudspeakers should strive to sound like the real thing." Who are you trying to fool? Maybe yourself? We all know that audio playback is not about trying to sound like the "real thing"- that's as silly as it gets. Nobody was ever fooled that listening to playback is like a live event. Audio playback is about creating an illusion - an illusion that appeals to our auditory perception as somewhat realistic. There is no system that comes anyway near being mistaken for a live musical event.

Well done! I'm sure we would all be interested in the blind test results of your $20,000 speakers Vs your $1,600 speakers

I don;t see any posted on your website.

So what does the extra $18,400 deliver?

So what does the extra $18,400 deliver?

Quote from: jkeny on 2015-04-02 19:57:51

So what does the extra $18,400 deliver?

Some pretty large, pretty loud, pretty speakers. 

cheers,

AJ

Quote from: jkeny on 2015-04-02 20:52:22

So what does the extra $18,400 deliver?

Oh yes, deeper bass, higher output, more adjustability, fully active (10ch amplification), larger soundstage (indirect drivers), 6 cabinets, in home setup by me.

@jkeny: Let me make it easier for you to understand with a hypothetical;

You claim that your DAC sounds better than a much less expensive DAC. I know of no reason why it should so my assumption is that it does not.

You take an ABX test and fail - no change, I still assume that they sound the same.

Ten million more people try to ABX a difference, and they also fail - no change, I still assume that they sound the same.

It turns out that all of them were guessing randomly, or that they were all monkeys - still no change, I still assume that they sound the same.

One person passes an ABX test convincingly - finally a change - my original assumption was incorrect, and there is an audible difference (although this does not necessarily prove that yours sounds "better").

So you would make the assumption that all of those failed ABX tests were because the subjects didn't know what to listen for - fair enough.

I have no problem with that. But that still has zero effect on my initial assumption that there is no audible difference.

The difference is that I have no ax to grind - I would be just as happy to have my assumption proven wrong as not. It is, however, in your best interest to prove a difference, so you are the one that will be pursuing a positive ABX, not me.

Jkeny, roughly how many foobar ABX tests have you taken? Just that one you posted on AVS?

As I thought it would seem. Just one.

I think by his [jkeny] logic, if some student were to fill out a true/false test in school without reading the questions and simply checking all the "true" boxes, this would prove that true/false tests "don't work" and that the entire test methodology should be scrapped.

3. The p values given in the table linked above are valid only if the two following conditions are fulfilled :

-The listener must not know his results before the end of the test, except if the number of trials is decided before the test.
...otherwise, the listener would just have to look at his score after every answer, and decide to stop the test when, by chance, the p value goes low enough for him.

-The test is run for the first time. And if it is not the case, all previous results must be summed up in order to get the result.
Otherwise, one would just have to repeat the serial of trials as much times as needed for getting, by chance, a p value small enough.
Corollary : only give answers of which you are absolutely certain ! If you have the slightest doubt, don't answer anything. Take your time. Make pauses. You can stop the test and go on another day, but never try to guess by "intuition". If you make some mistakes, you will never have the occasion to do the test again, because anyone will be able to accuse you of making numbers tell what you want, by "starting again until it works".

Of course you can train yourself as much times as you whish, provided that you firmly decide beforehand that it will be a training session. If you get 50/50 during a training and then can't reproduce this result, too bad for you. the results of the training sessions must be thrown away whatever they are, and the results of the real test must be kept whatever they are.

Once again, if you take all the time needed, be it one week of efforts for only one answer, in order to get a positive result at the first attempt, your success will be mathematically unquestionable ! Only your hifi setup, or your blind test conditions may be disputed. If, on the other hand, you run again a test that once failed, because since then, your hifi setup was improved, or there was too much noise the first time, you can be sure that there will be someone, relying on statistic laws, to come and question your result. You will have done all this work in vain.

4. The test must be reproducible.
Anyone can post fake results. For example if someone sells thingies that improve the sound, like oil for CD jewel cases of cable sheath, he can very well pretend to have passed a double blind ABX test with p < 0.00001, so as to make people talk about his products.
If someone passes the test, others must check if this is possible, by passing the test in their turn.

Quote from: jkeny on 2015-04-02 19:14:42

No, results presuppose that a "test" was actually taken. In the ABX null results posted here by mzil, where he just randomly guessed - do you consider that he "took a test" & delivered "results"?

It doesn't matter. Noone except him could tell the difference. That's why your whole idea of an invalid null result is bunk.

And let me add this: The fact that it doesn't matter is an important quality of such a test. It speaks for the ABX test method, and is a major factor in its usefulness.

Quote

If I sat a monkey down in front of the keyboard would I most likely get the same null results? Would you count the monkey's results among the null results?

I am not supposed to judge the monkey. If he produces a null result, I count it as a null result. If he produces a result that deviates significantly from chance, I count it accordingly. If I did anything else I'd be rigging the test.

Quote

Would you consider the results produced by a deaf person as a valid test? What about someone who demonstrated a hearing impairment in the audible area being tested? What about someone who has demonstrated that they are pre-biased to not hearing any differences? What about someone who is so tired that they aren't focussed? What about playback equipment that is unsuitable for revealing differences? Do I need to go on?

No, you needn't go on. I would count all of them as valid results. Doing anything else would put my own judgement above their results. I would effectively override their test results, thereby making the test invalid. Any test is invalid if the test administrator is allowed to override the test results of selected participants. Isn't that abundantly clear?

Now, it is true that a hearing test conducted with monkeys or deaf people might be regarded as pointless. That is an unfortunate consequence of a poor test design. If you aren't interested in the hearing abilities of monkeys or deaf people, you should exclude them from the test before the start. Once they are in, they are in - this doesn't make the test invalid. It may merely make it useless, depending on what the question was. This is so by design, I have to emphasize it again! It is not a fault or deficiency of ABX, quite the opposite, it is a major factor of its usefulness.

Quote

Yes, & the important word here is "result" - you have to decide what is a result & what should be eliminated from "results". In your statements you are including everything into results even the deaf monkeys "results"

Yes, that is quite deliberately so. You are trying to ridicule my position, but you miss that it isn't ridiculous at all. It is the opposite: Your position puts the designer/administrator of the test into a position where he can manipulate the criteria for result acceptance to his liking, potentially even after the fact. That's what I would call invalid!

And you wonder why other people don't trust you as a test designer/administrator? Amusing!

Yep, I get it. Treating all null results as valid & piling them into a block of evidence designed to create an edifice of damning evidence - this is what your tactic is all about.

No, it doesn't make sense unless you ignore all null results & not use them as a body of evidence. Yes, it all makes per sense if you have a particular position you want to advance & want to use this test which is obviously rife with experimenter's bias

Well designed tests use pre-screening, pre-training & internal controls to eliminate as many issues as possible - this will eliminate some listeners/playback systems from the test. Internal controls are used in well-designed tests to catch problems within the test. For instance, let's say this ABX test was testing high-res Vs RB - so we have two files that are A & B - randomly, in some trials, the software could introduce a difference of 0.5dB or whatever (is considered an agreed audible difference) in X. The listener, if he is doing the test correctly should be able to identify this difference. This would verify that the listener is actually listening on every trial & isn't just randomly guessing or isn't too tired & lost focus. The software can report these trials as controls separately to the other trials. The expected result is known for these controls & if the listener doesn't get correct results for these controls then his other results should be discarded

Wow, I'm glad that you laid all this out in black & white for all to see - it really proves the experimenter bias that underlies the thinking by many DBT supporters on thsi forum.