HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: mirrorsawlljk on 2005-10-19 03:03:12

Title: What's the problem with double-blind testing?
Post by: mirrorsawlljk on 2005-10-19 03:03:12
I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable.  I'm afraid I don't understand his angle of attack.  I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test. 

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing.  Is there an audio magazine that isn't filled with this kind of thinking?
Title: What's the problem with double-blind testing?
Post by: bubka on 2005-10-19 03:07:50
some people can actually detect specific codecs by their sounds, which defeats the purpose, but there is no problem, its as scientific as you can get.

Does this person also believe the earth was created in 6 days too?
Title: What's the problem with double-blind testing?
Post by: TheQat on 2005-10-19 03:27:27
Quote
I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable.  I'm afraid I don't understand his angle of attack.  I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test. 

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing.  Is there an audio magazine that isn't filled with this kind of thinking?
[a href="index.php?act=findpost&pid=335553"][{POST_SNAPBACK}][/a]


Keep in mind that if Stereophile's audience realizes that double-blinds are totally valid and acknowledges the importance of that realization, all of Stereophile's ad revenue will evaporate as companies that make massively over-priced cable (I bet you can find several full-page ads for cable in whatever issue you happen to be reading) and other hocus-pocus products go out of business like medieval humans dying of the black plague.

Therefore it is in the best economic interest of Stereophile and all its writers to encourage the entirely wrong-headed thinking that double-blind testing is unreliable, and as apparently unscrupulous people they do so and will continue to do so.

Edit: Hello goon
Edit2: I too would like to know if there is an audio magazine that is not in love with products of dubious worth.
Title: What's the problem with double-blind testing?
Post by: Tahnru on 2005-10-19 04:29:31
The closest thing I have seen to a legitimate criticism fielded when discussing double blind tests was a discussion about how far the results were interpreted. Of course, this didn't address the validity of the testing method at all, but only the misinterpretation or overinterpretation of the results. And even that doesn't seem to be a problem very often.
Title: What's the problem with double-blind testing?
Post by: singaiya on 2005-10-19 04:31:13
Quote
Quote
I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable.  I'm afraid I don't understand his angle of attack.  I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test. 

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing.  Is there an audio magazine that isn't filled with this kind of thinking?
[a href="index.php?act=findpost&pid=335553"][{POST_SNAPBACK}][/a]


Keep in mind that if Stereophile's audience realizes that double-blinds are totally valid and acknowledges the importance of that realization, all of Stereophile's ad revenue will evaporate as companies that make massively over-priced cable (I bet you can find several full-page ads for cable in whatever issue you happen to be reading) and other hocus-pocus products go out of business like medieval humans dying of the black plague.

Therefore it is in the best economic interest of Stereophile and all its writers to encourage the entirely wrong-headed thinking that double-blind testing is unreliable, and as apparently unscrupulous people they do so and will continue to do so.

Edit: Hello goon
Edit2: I too would like to know if there is an audio magazine that is not in love with products of dubious worth.
[a href="index.php?act=findpost&pid=335556"][{POST_SNAPBACK}][/a]



That's the answer. Stereophile as a magazine is not unique in its reliance on advertising revenue. Since all magazines rely on advertising, and advertisers in this scenario rely on selling things like cables that cannot be double blind tested positively, there is unlikely to be any "high end" audio mags willing to promote double blind testing as a valid procedure for evaluation. The mags and the advertisers have a sort of symbiotic relationship; they need each other. DBTs are bad for business all round.

Maybe there are some more modern computer/digital audio oriented magazines out there that have a different segment of advertisers who are not threatened by DBT, but I don't know of any.
Title: What's the problem with double-blind testing?
Post by: Axon on 2005-10-19 07:06:37
The Audio Critic (http://www.theaudiocritic.com) is notably pro-DBT.
Title: What's the problem with double-blind testing?
Post by: onthejazz on 2005-10-19 10:03:46
Interesting publication. I like it, too bad its not in full production anymore.  Thanks much for the link. Anybody have any more quality sources?
Title: What's the problem with double-blind testing?
Post by: Donunus on 2005-10-19 11:00:39
The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-10-19 11:54:00
The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.
Title: What's the problem with double-blind testing?
Post by: Donunus on 2005-10-19 12:06:41
Quote
The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.
[a href="index.php?act=findpost&pid=335627"][{POST_SNAPBACK}][/a]

What do you mean by long term abx? how do I do this? Play entire songs? actually can you give me a link that explains placebo. thx
Title: What's the problem with double-blind testing?
Post by: PoisonDan on 2005-10-19 12:18:26
Quote
The Audio Critic (http://www.theaudiocritic.com) is notably pro-DBT.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=335581")

Thanks. I really like their sample article [a href="http://www.theaudiocritic.com/downloads/article_1.pdf]"The Ten Biggest Lies in Audio"[/url].
Title: What's the problem with double-blind testing?
Post by: kjoonlee on 2005-10-19 18:31:58
Quote
actually can you give me a link that explains placebo. thx[a href="index.php?act=findpost&pid=335628"][{POST_SNAPBACK}][/a]

The word placebo, when used in audio circles, is a little different from the medical term, where people can actually benefit a little from dummy treatments.

According to ff123, it's closer to expectation bias, where a person's expectation can influence his perception, if I understand correctly.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-10-19 19:33:50
Well, I think expectation effects in listening tests actually make people perceive the sound as truly different, so they cause an equivalent effect to the placebo effect in medicine. I mean, both them cause a real effect: patients think that they are having a medicine, and that cures them. Listeners think that their equipment improves or deteriorates the sound, and that makes them truly perceive these effects.
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-10-19 20:06:56
Quote
I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable.  I'm afraid I don't understand his angle of attack.  I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test. 

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing.  Is there an audio magazine that isn't filled with this kind of thinking?
[a href="index.php?act=findpost&pid=335553"][{POST_SNAPBACK}][/a]


By all means read it..it's always good for a laugh.  I have the current issue at home and am waiting for a free hour or two to enjoy it.

I've interacted with editor John Atkinson online and very briefly in person -- he's mastered the art of balancing himself on a rehtorical tightrope between scientific evaluation of claims (he has a background in physics) and audiofoolery.  This manifests itself in his magazine as the schizophrenic devotion to engineering jargon and inclusion of bench measurements -- sometimes extremely detailed -- accompanying sighted 'reviews' that are literally whimsical.  Atkinson knows who his subscriber base is and I doubt he's about to alienate them by embracing valid scientific methods for subjective comparison...ever.  The purely 'subjective' experience will ALWAYS trump ANY measurement or scientific method, to that crowd.

But Stereophile is a model of restraint compared to The Absolute Sound, which is absolutely off the deep end, and has been forever.

Peter Aczel's 'The Audio Critic' (now online only) is the closest thing to a purely objective audio magazine you'll find.  (www.theaudiocritic.com).  Lesser lights include The Sensible Sound (about half of the people there seem to drink the hi-end cool-aid) and Sound & Vision (David Ranada is a DBT advocate, Ian MAsters is a no-nonsense objectivist, Ken Pohlmann is a god of digital, but the mag as a whole seems ever more tilted towards marketing hype -- I don't see much by Ranada in there recently)
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-10-19 20:12:48
Quote
Well, I think expectation effects in listening tests actually make people perceive the sound as truly different, so they cause an equivalent effect to the placebo effect in medicine. I mean, both them cause a real effect: patients think that they are having a medicine, and that cures them. Listeners think that their equipment improves or deteriorates the sound, and that makes them truly perceive these effects.
[a href="index.php?act=findpost&pid=335745"][{POST_SNAPBACK}][/a]


The audio placebo effect also makes people 'feel' differently, and that doubtless has some neurophysiological correlate -some measureable brain activity correlated to the feeling -- so it's 'real' in that sense.  But the point is it's entirely self-generated -- it is not due to some real difference between sounds.  (Just as the beneficial medical placebo effect isn't 'due', in any meaningful way, to the sugar pill itself -- it's due to the idea that the sugar pill was medicine.)
Title: What's the problem with double-blind testing?
Post by: Danimal on 2005-10-19 20:20:23
Quote
The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.
[a href="index.php?act=findpost&pid=335617"][{POST_SNAPBACK}][/a]


The point of double-blind testing is to make sure that you're getting goosebumps over something you actually hear, and not something you expect to hear.  The whole point is to try to evaluate what you're hearing without knowing whether it's the original or the lossy encode.  That shouldn't remove the "emotional" part of it, just make sure that you're reacting based upon something that is really there, and not what you want to hear or what you expect to hear.
Title: What's the problem with double-blind testing?
Post by: Halcyon on 2005-10-19 20:29:41
There is nothing inherently bad about double blind.

However, 'double blind' is not a test method per se, it's a test methodology requirement (or super class)  to rule out bias.

ABCHR is a rough outline of a test methodology in sensory evaluation.

So is ABX.

Hoever, even they can be set up (i.e. implemented) improperly and used to produce useless tests (that prove nothing).

For example, if you set up a test for ABX with persons who are CONVINCED they will hear NO difference, you have a strong tester bias that a simple ABX test method (even when double blind) does not protect against.

There are many inherently problematic issues about sensory evaluation of stimuli at the treshold of detection.

Currently, most set ups crudely ignore this and often err on the side of ruling out type II errors and as such, increasing the likelihood of type I errors in the test setup. This is esp. true of most statistical "analysis" (it's not really analysis, it's crude mechanistic calculation) of the test data, when you look at most of the published studies.

The problem is that many people fail to make a conceptual distinction between the following concepts:

- Double blind test
- ABX test
- Sensory evaluation test

They are not all interchangeable and correct procedure in one, does not necessarily translate to a correct procedure in another.
Title: What's the problem with double-blind testing?
Post by: ff123 on 2005-10-19 20:53:29
Tests that try to distinguish very small effects *and* aim for small type I and type II errors either require lots of listeners or lots of trials.  See my test sensitivity calculator:

http://ff123.net/export/TestSensitivityAnalyzer.xls (http://ff123.net/export/TestSensitivityAnalyzer.xls)

Column "D" shows the Proportion of Distinguishers, that is, the proportion of listeners in a randomly sampled group of people who would be expected to hear a difference in an audio comparison.  The smaller the proportion of distinguishers, the smaller the effect.  If the proportion of distinguishers is 30% (a pretty small effect), you'd need 119 listeners out of which 69 have to make a correct response to limit both the type I and type II errors to 0.05.  I assume this would be the criteria that would satisfy both the "objective" and the "subjective" crowds.

But obviously there is already a huge problem in trying to assemble that many listeners or in attempting such a large number of trials.

ff123
Title: What's the problem with double-blind testing?
Post by: fcmts on 2005-10-19 22:15:00
There is another problem similar to wine blind tests: it is very hard to believe that wine experts can't recognize the origin of the samples by the taste. Although those tests are not abx, that bias is also present in sound abx tests for the testers are, in general, experts in codecs.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-19 22:22:46
I don't see how that is a problem. The goal of an ABX test is to determine if there is an audible difference, it is not intended to determine the quality of a track. The answer is "yes" or "no", not "good" or "bad" as in wine tests.
Title: What's the problem with double-blind testing?
Post by: Yaztromo on 2005-10-19 23:05:44
Quote
Edit: Hello goon

[a href="index.php?act=findpost&pid=335556"][{POST_SNAPBACK}][/a]


How in the world of intuition did you figure that bit out? I don't see no stairs!
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-10-19 23:40:21
Quote
Tests that try to distinguish very small effects *and* aim for small type I and type II errors either require lots of listeners or lots of trials.  See my test sensitivity calculator:

http://ff123.net/export/TestSensitivityAnalyzer.xls (http://ff123.net/export/TestSensitivityAnalyzer.xls)

Column "D" shows the Proportion of Distinguishers, that is, the proportion of listeners in a randomly sampled group of people who would be expected to hear a difference in an audio comparison.  The smaller the proportion of distinguishers, the smaller the effect.  If the proportion of distinguishers is 30% (a pretty small effect), you'd need 119 listeners out of which 69 have to make a correct response to limit both the type I and type II errors to 0.05.  I assume this would be the criteria that would satisfy both the "objective" and the "subjective" crowds.

But obviously there is already a huge problem in trying to assemble that many listeners or in attempting such a large number of trials.

ff123
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=335768")



Interesting.  Check out this AES preprint about a comparison SACD and DVD-A , whihc seems to be derived from a German student's masters thesis:

[a href="http://www.hfm-detmold.de/eti/projekte/diplomarbeiten/dsdvspcm/aes_paper_6086.pdf]http://www.hfm-detmold.de/eti/projekte/dip..._paper_6086.pdf[/url]

Four out of 45 listeners comparing stereo recordings passed an ABX test.  None of 100 listeners (include the four mentioned above) passed an ABX using surround material.  The material listened to by each of the four 'successful' listeners was different.  Also, they all onl;y heard the difference using headphones.  There may have been some switch noise affecting the results.

Can you make any evaluation of the senstivity of these tests?
Title: What's the problem with double-blind testing?
Post by: Donunus on 2005-10-20 01:51:43
Quote
Quote
The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.
[a href="index.php?act=findpost&pid=335617"][{POST_SNAPBACK}][/a]


The point of double-blind testing is to make sure that you're getting goosebumps over something you actually hear, and not something you expect to hear.  The whole point is to try to evaluate what you're hearing without knowing whether it's the original or the lossy encode.  That shouldn't remove the "emotional" part of it, just make sure that you're reacting based upon something that is really there, and not what you want to hear or what you expect to hear.
[a href="index.php?act=findpost&pid=335756"][{POST_SNAPBACK}][/a]


Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
Title: What's the problem with double-blind testing?
Post by: ff123 on 2005-10-20 05:38:43
Quote
Interesting.   Check out this AES preprint about a comparison SACD and DVD-A , whihc seems to be derived from a German student's masters thesis:

http://www.hfm-detmold.de/eti/projekte/dip..._paper_6086.pdf (http://www.hfm-detmold.de/eti/projekte/diplomarbeiten/dsdvspcm/aes_paper_6086.pdf)
Four out of 45 listeners comparing stereo recordings passed an ABX test.  None of 100 listeners (include the four mentioned above) passed an ABX using surround material.  The material listened to by each of the four 'successful' listeners was different.  Also, they all onl;y heard the difference using headphones.   There may have been some switch noise affecting the results.

Can you make any evaluation of the senstivity of these tests?
[a href="index.php?act=findpost&pid=335793"][{POST_SNAPBACK}][/a]


Proponent of one format vs the other:  "Aha, 4 people definitely heard a difference!" and "That low-level crackling sound when switching didn't affect the test's validity."

Someone in the no-difference camp:  "Aha, 141 out of 145 20-run ABX tests did not show a difference!"

Choose your interpretation.

ff123
Title: What's the problem with double-blind testing?
Post by: Lyx on 2005-10-20 06:07:44
Quote
Quote
The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.
[a href="index.php?act=findpost&pid=335627"][{POST_SNAPBACK}][/a]

What do you mean by long term abx? how do I do this? Play entire songs?
[a href="index.php?act=findpost&pid=335628"][{POST_SNAPBACK}][/a]

Well, this is a two-sided sword, because long-term DBTs are indeed theoretically no problem - but in practice, most ABX-tools aren't designed for this purpose, which makes a bit more difficult to setup.

It basically goes like this - any abx-test may take as long as you want - there is no limit on how long the timespan has to be between trials - there is no limit in how long the entire test takes - and there is no limit on the number of trials.

What this means is, that you could for example do a 6months long DBT, where you just pick one of your fav-albums, and just go your normal daily schedule.... BUT the application which you use for playback should make it invisible to you if the original or the encoded version is currently played. Then, during those 6 months, when you just normally listen to that fav-album, and think "this is the encoded version!", then you just click a button, and afterwards continue your normal daily life. Over the course of the whole 6 months, you repeat that a dozen times.... and afterwards you end up with a longterm DBT which has so many trials, that its accuracy is incredibly high.

The only thing to keep in mind in that case is that there should be no way for you during those 6 months to see the intermediate results. Another way would be to not limit the test by time, but instead define beforehand on the number of trials..... i.e. say "this test will end after 100 trials were done". If you go with the predefined number of trials approach, then you are not allowed to end the test before you complete the 100 trials, else the results will be invalid.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-20 08:48:43
Quote
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[a href="index.php?act=findpost&pid=335816"][{POST_SNAPBACK}][/a]

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
Title: What's the problem with double-blind testing?
Post by: Donunus on 2005-10-20 16:51:12
Quote
Quote
Quote
The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.
[a href="index.php?act=findpost&pid=335627"][{POST_SNAPBACK}][/a]

What do you mean by long term abx? how do I do this? Play entire songs?
[a href="index.php?act=findpost&pid=335628"][{POST_SNAPBACK}][/a]

Well, this is a two-sided sword, because long-term DBTs are indeed theoretically no problem - but in practice, most ABX-tools aren't designed for this purpose, which makes a bit more difficult to setup.

It basically goes like this - any abx-test may take as long as you want - there is no limit on how long the timespan has to be between trials - there is no limit in how long the entire test takes - and there is no limit on the number of trials.

What this means is, that you could for example do a 6months long DBT, where you just pick one of your fav-albums, and just go your normal daily schedule.... BUT the application which you use for playback should make it invisible to you if the original or the encoded version is currently played. Then, during those 6 months, when you just normally listen to that fav-album, and think "this is the encoded version!", then you just click a button, and afterwards continue your normal daily life. Over the course of the whole 6 months, you repeat that a dozen times.... and afterwards you end up with a longterm DBT which has so many trials, that its accuracy is incredibly high.

The only thing to keep in mind in that case is that there should be no way for you during those 6 months to see the intermediate results. Another way would be to not limit the test by time, but instead define beforehand on the number of trials..... i.e. say "this test will end after 100 trials were done". If you go with the predefined number of trials approach, then you are not allowed to end the test before you complete the 100 trials, else the results will be invalid.
[a href="index.php?act=findpost&pid=335853"][{POST_SNAPBACK}][/a]


Damn! What regular consumer audio equipment supports this? There should be car decks for example that have this voting button
Title: What's the problem with double-blind testing?
Post by: bryant on 2005-10-20 18:11:09
Quote
Quote
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=335816")

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
[a href="index.php?act=findpost&pid=335879"][{POST_SNAPBACK}][/a]


It turns out that the science of perception is rich with results that run counter to common sense. Think how wrong common sense is when trying to understand quantum mechanics! Well, it is equally bad when trying to understand cognitive psychology and perception. To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782]http://www.hydrogenaudio.org/forums/index....ndpost&p=321782[/url]
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-10-20 20:48:27
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782 (http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782)
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?
Title: What's the problem with double-blind testing?
Post by: Lyx on 2005-10-20 21:10:30
Quote
Quote

...
The only thing to keep in mind in that case is that there should be no way for you during those 6 months to see the intermediate results. Another way would be to not limit the test by time, but instead define beforehand on the number of trials..... i.e. say "this test will end after 100 trials were done". If you go with the predefined number of trials approach, then you are not allowed to end the test before you complete the 100 trials, else the results will be invalid.
[a href="index.php?act=findpost&pid=335853"][{POST_SNAPBACK}][/a]


Damn! What regular consumer audio equipment supports this? There should be car decks for example that have this voting button :P
[a href="index.php?act=findpost&pid=335972"][{POST_SNAPBACK}][/a]

Well, yeah - i think its actually sad that most DBT-tools(be it player-plugins, hardware or dedicated apps) aren't designed with doing longterm "relaxed" tests... there are a handful of people who went through the hassle to setup 'em, but they're really a minority... :-/

Oh, to clarify about the "dont see the intermediate results" and "limit number of trials beforehand"-stuff....... this is because of the following:

If you are able to see the intermediate results AND are able to decide about when to end the test whenever you want, then you can just manipulate the test by ending it when you like the current results. Therefore, there are just two valid approaches to do an ABX:

1. You may end the test whenever you feel like it, but you may not see the intermediate results in this case.

2. You decide about the number of trials beforehand and may see the intermediate results. If you end the test before the defined number of trials were completed, then the test is invalid and must be discarded.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-20 21:43:04
Quote
It turns out that the science of perception is rich with results that run counter to common sense. Think how wrong common sense is when trying to understand quantum mechanics!

Not really, it is just that in the area where quantum mechanics is applicable we have no emperical knowledge/experience at all.

Quote
Well, it is equally bad when trying to understand cognitive psychology and perception. To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782 (http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782)
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]

I couldn't read in those articles that our subconcious is turned off during an ABX test... in fact, it seemed to me the tests in the second article were performed under conditions ABX-test worthy.
Title: What's the problem with double-blind testing?
Post by: singaiya on 2005-10-20 22:13:31
Thanks for the papers, David. The first one is a good read, the second one was way over my head so I skipped that one.

But I understand Pio's question: given that humans respond differently to conscious & subconscious stimuli, how is casual listening, aka the sighted listening setting where WAVs give Donunus goosebumps, a subconscious stimulus while a DBT setting is not?

Donunus, maybe you are not getting goosebumps from mp3 because you can pass ABX tests between original vs 320. I would be more concerned if you were failing the ABX, because in that case you should get goosebumps from either format. Seems like your problem is not with DBT, but the curse of your golden ears 
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-20 22:52:08
Quote
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782 (http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782)
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?
[a href="index.php?act=findpost&pid=336039"][{POST_SNAPBACK}][/a]


It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be. Because of the increased effort required of perceptual processes, due to compressed information that must be resolved, listening experiences might have an uncomfortable element associated with them that uncompressed stimulus perception doesn't. An ABX paradigm might not be able to tap into that very well, if at all (because the effort is perceptual, not in a decision processs). But there are other methodologies that could measure whatever processing differences might exist.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-20 23:00:51
One distinction relevant to this issue is the difference between perceptual and decision processes. Whenever you use a judgment paradigm requiring a subject to provide a verbal report, you invoke decision processes that necessitate additional processing beyond perceptual processing. In other words, in an ABX task when you ask a person to match two stimuli (presumably based on auditory quality), the person essentially relies on memory, which must be processed through a language system. There are other methodologies that attempt to tap into perceptual processes more directly, such as reaction time paradigms, or other body movements like jabbing. These methodologies often elicit data at odds with verbal report data for the same stimuli.

A good example of this is provided by research showing how early visual information gets parsed and differentially distributed to other information processing systems. One finding is called the Roelof’s effect. This is a phenomenon where visual distracters affect location judgments of objects for one system (a “cognitive” system you access by asking someone to verbally describe a location), but the distracters do not affect another system (a “motor” system where you ask subjects to point to the location). Imagine two vertical lines and a dot in between them. These stimuli appear for a couple of seconds, then disappear. The subject is then asked to identify the location either verbally or manually. Judgments of the dot’s position become biased by the relative placement of the vertical lines (e.g., if the dot is closer to the left line, then the location is judged to the right of its actual location), but this bias only happens to the cognitive system, not the motor system.

This illustrates that decision processes later in the information processing chain are often subject to top down influences, or bottom up deficits, that other systems, like motor systems, are not. Point being, decision processes rely on particular pre-processed information, and are often not a reliable source for determining differences in perceptual processing, whereas motor systems often are.

In terms of audio, this could mean that DBT might fail to distinguish auditory stimuli that in fact do have relevant differences in their acoustic properties. For example, perhaps processing auditory stimuli with degraded resolution might be more difficult, and result in subtle fatigue or other effects that aren’t reflected in verbal reports, but might be revealed in reaction time experiments. There are principled reasons why different systems have access to different information, and the details of this are obviously beyond the scope of the discussion. However, perceptual processes involved in the judgment of sound quality are not designed to be sensitive to fine distinctions between different digitized stimuli, so such stimuli might not affect subjective perception. That is, whatever distinctions that can be made will be byproducts of auditory system design features that might or might not be affected by digitization manipulations. But this doesn’t rule out other possible effects on processing that compressed stimuli might have on a listening experience.

The overall message from cognitive science is that our conscious notions are quite often deceiving, and sometimes just outright made up. So while DBT is methodologically sound, its implementation in perception research assumes incorrectly that subjective experience has access to the processes underlying it. That said, I am not arguing this perspective to justify bizarre audiophile notions, instead David and I are proposing that some of the elusive effects described by audiophiles might be based in reality. It’s a testable idea that is on the agenda, by the way.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-20 23:11:38
Quote
It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be.

Might be... or might be not. You can't conclude or even suggest that based on the articles mentioned above. They only prove that subconcious stimuli can influence our behaviour. But when and where that happens is not mentioned. There is no reason to assume that we are more sensitive to subconsious stimuli during casual listening than during an ABX test.

Quote
Because of the increased effort required of perceptual processes, due to compressed information that must be resolved,

Is that so? Does listening to MP3 require more effort than listening to PCM?
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-20 23:36:39
Quote
Might be... or might be not. You can't conclude or even suggest that based on the articles mentioned above. They only prove that subconcious stimuli can influence our behaviour. But when and where that happens is not mentioned. There is no reason to assume that we are more sensitive to subconsious stimuli during casual listening than during an ABX test.


I'm not really referring to implications of just those articles. But the articles are quite specific about when and where unconscious information affects our behavior. In fact, there are five experiments in that second paper that show quite clearly how phase correction occurred in synchronized motor movements as a function of unconsciously perceived elements in an auditory stimulus. Anyway, as I described in my last post, the idea is that the ABX task relies on processing that has no direct access to the relevant parameters that are manipulated in some compression schemes.

Quote
Is that so? Does listening to MP3 require more effort than listening to PCM?


I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems. In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.

You might be confusing conscious effort from processing effort (the latter of which you have no conscious access to, and might only be revealed through implicit measurements like reaction time data to auditory stimuli).
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-10-21 00:00:34
Quote
I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems. In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.
[a href="index.php?act=findpost&pid=336092"][{POST_SNAPBACK}][/a]

Our ear doesn't work that way. Our auditory processes don't derive any kind of waveform. They are more like a realtime frequency analyzer, from which only part of the data is sent to the brain. Lossy compression just tries to discard that information that is not audible, just that.

As to the process of having to take a decision being different from just listening, well, that should apply too to sighted listening tests, don't you think so?

Also, psychoacoustic research already found that quick switch, short length stimuli, blind  listening tests have proved to be more much more sensitive that casual listening.
Title: What's the problem with double-blind testing?
Post by: rjamorim on 2005-10-21 00:15:32
Quote
Also, psychoacoustic research already found that quick switch, short length stimuli, blind  listening tests have proved to be more much more sensitive that casual listening.[a href="index.php?act=findpost&pid=336097"][{POST_SNAPBACK}][/a]


Right, more sensitive to certain artifacts, but maybe less sensitive to others?
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-21 00:29:28
Quote
Quote
I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems. In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.
[a href="index.php?act=findpost&pid=336092"][{POST_SNAPBACK}][/a]

Our ear doesn't work that way. Our auditory processes don't derive any kind of waveform. They are more like a realtime frequency analyzer, from which only part of the data is sent to the brain. Lossy compression just tries to discard that information that is not audible, just that.

As to the process of having to take a decision being different from just listening, well, that should apply too to sighted listening tests, don't you think so?

Also, psychoacoustic research already found that quick switch, short length stimuli, blind  listening tests have proved to be more much more sensitive that casual listening.
[a href="index.php?act=findpost&pid=336097"][{POST_SNAPBACK}][/a]


The auditory system must derive, from a waveform, the characteristics that are relevant to any number of functions. In speech and music, the generated representation has all the characteristics of a waveform. From the stimulus, the auditory cortex represents spectral and temporal features, so for all intents and purposes, this is a waveform. Lossy compression tries to discard information that is not audible, but that doesn’t speak to the acoustic deficits in the original signal that the system must handle.

Take for instance your pitch perception system. It generates a pitch percept based on information in a waveform. The system sends a percept to your conscious awareness, but the effort required to derive that information might not be revealed in the percept. For example, there is a phenomenon called the ‘missing fundamental’ where people perceive pitch (fundamental frequency, or F0) in a high band pass filtered segment that does not contain enough information to calculate a proper F0 through Fourier analysis (e.g., F1-F4 is missing). But the percept is there because the system can make an unconscious inference using limited information. Of course to the listener, it just seems like there’s a pitch. Degraded signals require more effort to process. I think that is a rather uncontroversial point.

I'm not sure what you mean by a sighted listening test?
Title: What's the problem with double-blind testing?
Post by: rjamorim on 2005-10-21 01:03:03
Quote
I'm not sure what you mean by a sighted listening test?[a href="index.php?act=findpost&pid=336106"][{POST_SNAPBACK}][/a]


The opposite of "blind", I suppose. I.E, a test where placebo can take effect.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-21 01:27:22
Quote
Quote
actually can you give me a link that explains placebo. thx[a href="index.php?act=findpost&pid=335628"][{POST_SNAPBACK}][/a]

The word placebo, when used in audio circles, is a little different from the medical term, where people can actually benefit a little from dummy treatments.

According to ff123, it's closer to expectation bias, where a person's expectation can influence his perception, if I understand correctly.
[a href="index.php?act=findpost&pid=335725"][{POST_SNAPBACK}][/a]


While I don't disagree, I'd go farther.

If you look at what the auditory system detects as an "event", it appears from the work done to be set to detect many 'false events' at a much lower risk of missing a real event.

This kind of performance (tradeoff between misses and false detections) is obligatory for any probabilistic system, of which the auditory system is quite.

I have heard this explained to others (somebody said this at a tutorial at the NY aes, for instance) as being for a simple reason:

If you hear the tiger coming, and hide, you live.

If you don't hear the tiger coming, you are out of the gene pool.

If you hear the tiger coming, and it isn't, all you do is look a bit silly.

If you consider that we are wired to detect nonevents in favor of missing any events, then that would explain quite nicely why changes cause  "differences".
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-21 01:27:27
Quote
The opposite of "blind", I suppose. I.E, a test where placebo can take effect.


Haha. Of course. Audiophile jargon. 

Well, the distinction between decision and perception applies regardless of the presentation conditions, but I don't see any reason to have any other ABX paradigm besides a double-blind design.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-21 01:38:21
Quote
If you consider that we are wired to detect nonevents in favor of missing any events, then that would explain quite nicely why changes cause "differences".


One has to be careful about generalizing this principle across all domains. Signal detection applies to many perceptual phenomena, but the costs and benefits associated with the four possible outcomes (hit, miss, false alarm, correct rejection) vary depending on what you are attempting to detect. So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!
Title: What's the problem with double-blind testing?
Post by: Donunus on 2005-10-21 04:10:15
Quote
Quote
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[a href="index.php?act=findpost&pid=335816"][{POST_SNAPBACK}][/a]

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
[a href="index.php?act=findpost&pid=335879"][{POST_SNAPBACK}][/a]

What I'm saying is not contradictory. All I'm saying is one cannot abx enjoyment because the enjoyment factor can only be realized when listening to the music casually. How can one enjoy the music as a whole when abx testing when youre busy listening to different aspects of the sound of a clip. Its a clinical procedure, not an emotional one.

I do not mean that  I "hear" less of a difference. What I mean is that the differences that I hear during analytical mode don't mean as much to me at that moment compared to when I listen to the music for enjoyment. When enjoying the music sometimes I think that I am tired of the song that I am listening to or feel bored but get more emotional or even cry when listening to the original. The analogy is exactly like listening to music in a different mental state. try listening when you are drunk and sober. The experience is different. Just like listening when you are analyzing isnt as  emotional as listening deeply to the soul of the music. At the time you are listening emotionally, you are not pinpointing the artifacts. The music speaks for itself. Sort of like placebo but this time it is not expectation bias, Its just the way it is whether or not I know which source cd is playing. To ramble even more, Its just like listening to music with different stereo systems. one is a more emotional experience, one is dull and boring. I hope you got what I meant. Its audiophile hocus pocus at work
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-10-21 05:25:08
Quote
Quote
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782 (http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782)
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?
[a href="index.php?act=findpost&pid=336039"][{POST_SNAPBACK}][/a]


It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be. Because of the increased effort required of perceptual processes, due to compressed information that must be resolved, listening experiences might have an uncomfortable element associated with them that uncompressed stimulus perception doesn't. An ABX paradigm might not be able to tap into that very well, if at all (because the effort is perceptual, not in a decision processs). But there are other methodologies that could measure whatever processing differences might exist.
[a href="index.php?act=findpost&pid=336072"][{POST_SNAPBACK}][/a]



Might .  Could.  If you suppose X.

How about, when evaluating the endless number of claims of difference between audio,  considering what we *know* DOES happen when people hear two things they think are different  : various forms of 'unconscious' bias that can and DO lead to such absurdities as subjectively perceiving the same sound the be different.  Of *course* unconscious stimuli affect our behaviour -- particularly our *beliefs* and our *claims* about what we hear and why.  These 'unconscious' stimuli are things like: stuff we've heard or read about the gear, their brands, their cost, their appearance.
They easily lead us to believe, and claim, that A sounds much different (better/worse) than B -- even if A and B are actually the same

This really happens.  It's not suppositional. 

Guy buys a new CD player.  Oooh, ahhh, so much better than the previous one!  But is it?  How does he know?  Well, he paid more for it.  Or it does 'upsampling'.  Or he read that it's better.  Gosh darn it, it *SOUNDS* better to him.  That;s proof enough.

Was this guy listening in 'uncompressed' fashion?  Where's the evidence that this overrides the nuisance factors that are ALWAYS present when listening 'sighted'?y
Title: What's the problem with double-blind testing?
Post by: saratoga on 2005-10-21 05:56:10
Quote
The auditory system must derive, from a waveform, the characteristics that are relevant to any number of functions. In speech and music, the generated representation has all the characteristics of a waveform.


All sounds are waveforms.

Quote
From the stimulus, the auditory cortex represents spectral and temporal features, so for all intents and purposes, this is a waveform. Lossy compression tries to discard information that is not audible, but that doesn’t speak to the acoustic deficits in the original signal that the system must handle.


Its still a waveform, so whats the difference.  Its not like you're reading the samples.  From your ears perspective its all just sound.


Quote
Take for instance your pitch perception system. It generates a pitch percept based on information in a waveform. The system sends a percept to your conscious awareness, but the effort required to derive that information might not be revealed in the percept. For example, there is a phenomenon called the ‘missing fundamental’ where people perceive pitch (fundamental frequency, or F0) in a high band pass filtered segment that does not contain enough information to calculate a proper F0 through Fourier analysis (e.g., F1-F4 is missing). But the percept is there because the system can make an unconscious inference using limited information. Of course to the listener, it just seems like there’s a pitch. Degraded signals require more effort to process. I think that is a rather uncontroversial point.


More effort to recover meaning from a distorted signal, sure.  But to say that processed audio is intrinsically harder to percieve is something entirely different.  What exactly are you basing this conclusion on?  I find it extremely unlikely that a signal that has been so marginally degraded that you cannot detect it via ABX would pose such a problem for one's senses as to consistantly fool a listener.
Title: What's the problem with double-blind testing?
Post by: Jun-Dai on 2005-10-21 07:22:53
I think it's pretty clear that double-blind test cannot "disprove" anything--they can only disprove something beyond reasonable doubt and, more importantly, there's really no better option out there.  They can prove the presence of something, at least up to the point at which one begins to doubt whether the test was performed correctly. 

The very fact that one is taking a double-blind test is going to affect one's perception.  Whether that difference is significant enough to make something audible inaudible (or something inaudible audible) isn't really knowable, as far as I know.  The mere fact that you are scrutinizing your perception of the music is going to affect your findings.  Because of this, there's no way that you can prove that something that causes a very minor change in the sound waves hitting your ear doesn't have any audible effect.  You can prove, via a double-blind test, that someone trying to hear the difference cannot, which for me, at any rate, is proof enough that any change in the sound is negligible.

As for mp3-decoded sound waves causing your brain to work harder to interpret the signal:  it's certainly possible that an mp3-decoded sound wave would require a different amount of effort to interpret on your brain's part, without you knowing it.  It's not reasonable to conclude, however, that an mp3-decoded sound would be harder; it could as well be easier for your brain to interpret.  That said, the only way to disprove this would be to monitor brain activity.  Proving it, on the other hand, might be possible through some carefully controlled double-blind tests (polling tiredness after extensive sample-listening).  Whatever the difference is, if there is any, is going to be pretty damn subtle.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-10-21 08:10:21
Quote
From the stimulus, the auditory cortex represents spectral and temporal features, so for all intents and purposes, this is a waveform.

OK, for a wider meaning of "waveform" than the one I was thinking of.

Quote
Take for instance your pitch perception system. It generates a pitch percept based on information in a waveform. The system sends a percept to your conscious awareness, but the effort required to derive that information might not be revealed in the percept.

You are assuming that it takes some kind of "effort" to "derive" such information, as if our auditory system had to struggle just to perceive sound. I don't know if that is true. Even if it was to some extent, which I guess can be... see my next replies.

Quote
For example, there is a phenomenon called the ‘missing fundamental’ where people perceive pitch (fundamental frequency, or F0) in a high band pass filtered segment that does not contain enough information to calculate a proper F0 through Fourier analysis (e.g., F1-F4 is missing).

True, and also true that this seems happen due to brain processing, not the ear.

Quote
But the percept is there because the system can make an unconscious inference using limited information.

I guess it can be called unconscious, because we have no conscience that is happening. But it can also be an automatic process that requires no extra effort, it's just the way the brain processes all sound.

Quote
Degraded signals require more effort to process. I think that is a rather uncontroversial point.

I'm no expert, but can't see why it's uncontroversial. Is there any actual evidence that supports this? Edit: I mean, in case of degraded signals where degradation is not audible. Because in case of missing fundamental, the difference is clearly audible.

Quote
I'm not sure what you mean by a sighted listening test?
[a href="index.php?act=findpost&pid=336106"][{POST_SNAPBACK}][/a]

The usual evaluation of audio equipment people do at home, not blind. The one with which many ones are able to perceive such great differences.

As Jun-Dai said, even if there were some problems with blind testing, it's still the least bad method.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-10-21 08:30:57
For helping doing long-term, casual listening-like listening tests, it can be useful the utility fileABX (http://www.kikeg.arrakis.es/fileabx), that makes "blind" copies of .wav test files. Those copies can be burn to CD, or copied to a DAP, in order to perform such a test.

It's also possible to generate copies of different sets of test files, and use all the sets to perfom the test, in order to avoid listening over just one pair of test samples. As long as the identification of each file of a set of files is unknown, and a proper statistical analysis of the final result of the listening evaluation is done, it will be a reliable blind test.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-21 08:34:51
Quote
I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems.

Present me good articles that describe this. I'm not interested in dogmas.

Quote
In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.

Sorry, this is not even true. I would like you to show me how you calculate that an 128 kbps mp3 has 90% less information (define information too please) than the original PCM data. Then show me that our auditory system must derive more from a decoded mp3 than from the original PCM AND explain exactly what it is that is being derived. If anything, it seems to me the opposite must be true and working with less data is also is less fatiguing

Quote
You might be confusing conscious effort from processing effort (the latter of which you have no conscious access to, and might only be revealed through implicit measurements like reaction time data to auditory stimuli).

And I think you are making too many assumptions and speculations that you just can't reasonably back up right now.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-21 08:43:40
Quote
I do not mean that  I "hear" less of a difference. What I mean is that the differences that I hear during analytical mode don't mean as much to me at that moment compared to when I listen to the music for enjoyment. When enjoying the music sometimes I think that I am tired of the song that I am listening to or feel bored but get more emotional or even cry when listening to the original. The analogy is exactly like listening to music in a different mental state. try listening when you are drunk and sober. The experience is different. Just like listening when you are analyzing isnt as  emotional as listening deeply to the soul of the music. At the time you are listening emotionally, you are not pinpointing the artifacts. The music speaks for itself. Sort of like placebo but this time it is not expectation bias, Its just the way it is whether or not I know which source cd is playing. To ramble even more, Its just like listening to music with different stereo systems. one is a more emotional experience, one is dull and boring. I hope you got what I meant. Its audiophile hocus pocus at work

But the problem is that this is not blinded. You KNOW you have put the CD in the tray. You KNOW what gear you are using. While I'm not gonna deny you your goosebumps caused by this, this is an emotional factor that lies indeed outside of the scope an ABX test. Sorry, but I don't I get those fuzzy feeling when listening to 10.000$ equipment, I would be constanly worried if what I bought was really worth the price and with that degrading my listening experience.
Title: What's the problem with double-blind testing?
Post by: user on 2005-10-21 09:20:45
I recall, we have had successful abx tests even here at ha in the high end area. The guy, who abxed 1 trial each morning, when his ears were fresh.
You don't need special software,
instead of software you need then a helper person, who burns a CD with first 2 files as references, AB, you know, which is which, and then other tracks as several X in random order, but the helper writes somewhere down, which is which.
Then you can play with this cd, and if you think, you are finished, the helper can reveal you the results.




I like the University of Music in Detmold, Germany.
How they carry out abx tests in diploma thesis.
And that they write directly in thier papers against te marketing hypes, now they revealed, no perceivable difference between DSD and 192/24 PCM.
hm, would be interesting also, that they test DSD vs. 96/24 and maybe properly dithered 16/44...

I presented some time ago thier results about testing possible differences between  16/48, 24/48, 24/96 music.
It was shown, that with some probability 24/48 is betetr than 16/48. And that 24/96 does not improve anthing comapred to 24/48. Unfortuentaly, they did that test, when 24/96 dacs were very fresh and not yet optimized.
Their very good 24/48 dacs performed better than 24/96 at that time, their test showed it.
They wrote in the paper also, that the limit of perceivable difference is probably at 20 bit, so 24 bit is overkill already. They referred to another paper for this statement. Unfortunately, this paper was only in german available, and it had limited value, as the 24/96 dacs had not yet their fully potential, and 16/44 was not included in that test as low anchor.
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-10-21 12:18:21
Quote
I recall, we have had successful abx tests even here at ha in the high end area. The guy, who abxed 1 trial each morning, when his ears were fresh.[a href="index.php?act=findpost&pid=336189"][{POST_SNAPBACK}][/a]


It was not here, but in another english forum. It was a 16 bits truncated vs 16 bits dihered test. I recall the sample being guitar only, with little dynamics, and no fade-out.
It was an interesting success.

It also took me a very long time to confirm Xerophase success with MPC standard on the Smashing Pumpkins sample. Not one trial per day, but several hours for the whole ABX test, as far as I remember. It is still online here, search for Xerophase's first posts in hydrogenaudio.

Quote
I presented some time ago thier results about testing possible differences between  16/48, 24/48, 24/96 music.
It was shown, that with some probability 24/48 is betetr than 16/48. [a href="index.php?act=findpost&pid=336189"][{POST_SNAPBACK}][/a]


I remeber that we couldn't figure out their protocol, nor the "probability" of their result, because it was all in german. Unfortunately, the original webpage is now offline.
Title: What's the problem with double-blind testing?
Post by: antz on 2005-10-21 13:39:03
Quote
Quote
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782 (http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782)
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?
[a href="index.php?act=findpost&pid=336039"][{POST_SNAPBACK}][/a]


It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be. Because of the increased effort required of perceptual processes, due to compressed information that must be resolved, listening experiences might have an uncomfortable element associated with them that uncompressed stimulus perception doesn't. An ABX paradigm might not be able to tap into that very well, if at all (because the effort is perceptual, not in a decision processs). But there are other methodologies that could measure whatever processing differences might exist.
[a href="index.php?act=findpost&pid=336072"][{POST_SNAPBACK}][/a]

Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"? Maybe also it makes no difference. If you can't ABX the difference then it's possible that you actually took away the "data" that you genuinely couldn't hear.

I'd be prepared to believe that an ABX test might be unreliable if the circumstances of the listening varied on each test and the difference was hovering around a level that you could genuinely distinguish. However, your results would tell you exactly that - that your test was statistically unsound.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-21 16:48:47
Quote
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?


Lossless and lossy compression schemes are just that, compression (http://www.computerhope.com/jargon/c/compress.htm). It is in this sense that I mean that an mp3 file, for example, is compressed. (http://www.computerhope.com/jargon/l/lossy.htm).

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision). I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-21 17:29:59
Quote
The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
[a href="index.php?act=findpost&pid=336296"][{POST_SNAPBACK}][/a]

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
Title: What's the problem with double-blind testing?
Post by: Lyx on 2005-10-21 17:35:53
Quote
Quote
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?


Lossless and lossy compression schemes are just that, compression (http://www.computerhope.com/jargon/c/compress.htm). It is in this sense that I mean that an mp3 file, for example, is compressed. (http://www.computerhope.com/jargon/l/lossy.htm).

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision). I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
[a href="index.php?act=findpost&pid=336296"][{POST_SNAPBACK}][/a]

This still does not show any evidence that this applies to lossy audiocompression which makes use of psychoacoustic models. It may apply, or it may not apply.

Why does it not necessarily apply? Because the purpose of lossy audio compression is to throw away unperceivable parts of the audio. So if the sensors aren't missing anything, then why should it make any difference? Anyways, discussing this wont really solve it because we simply lack DATA..... we need evidence and analysis..... not extrapolated from other fields.....  else we are just speculating.

I definatelly dont think that current test-methods are the end of the road. And i dont think that current DBT-implementations are perfect. But i do think that they're the best ones we have available right now. I am interested in better methods and flaws in current methods, but only if its backed by hard evidence - not extrapolated speculations ala "well, it could be possible that....". Why the hard line? Well, i've seen too many of those "speculations" which in the past ended up being exposed as flawed. Show me something significant and applicable and i'll become interested.
Title: What's the problem with double-blind testing?
Post by: ff123 on 2005-10-21 17:43:42
Quote
Quote
I recall, we have had successful abx tests even here at ha in the high end area. The guy, who abxed 1 trial each morning, when his ears were fresh.[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=336189")


It was not here, but in another english forum. It was a 16 bits truncated vs 16 bits dihered test. I recall the sample being guitar only, with little dynamics, and no fade-out.
It was an interesting success.
[a href="index.php?act=findpost&pid=336237"][{POST_SNAPBACK}][/a]


Results of this test are described here:

[a href="http://ff123.net/24bit/24bitanalysis.html]http://ff123.net/24bit/24bitanalysis.html[/url]
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-21 20:06:58
Quote
Quote
The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
[a href="index.php?act=findpost&pid=336296"][{POST_SNAPBACK}][/a]

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
[a href="index.php?act=findpost&pid=336310"][{POST_SNAPBACK}][/a]


It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

I will post studies that support a number of things I have claimed, including quite notably the neural dissociation of sensory and decision processes. The "assumptions" I'm making are rooted in my training as a cognitive psychologist who studies speech processing. Many of the things I've said are well supported in experimental studies...I'm not just making it up. Also, what I'm saying is testable. I'm just trying to explain the logic of why I believe a different methodology might be necessary to rescue at least some of the claims made by audiophiles. I'm not an audiophile.

Quote
Why does it not necessarily apply? Because the purpose of lossy audio compression is to throw away unperceivable parts of the audio. So if the sensors aren't missing anything, then why should it make any difference?


Again, the main point here is that fatigue effects caused by processing issues (not accessible to systems that verbally report differences in ABX listening tests) could have residual effects on listeners in more long term ways. Reaction time experiments could verify this processing claim I'm making, and a positive result (where an ABX test shows nothing) could indicate a relationship between perceptual processing and vague ideas of discomfort experienced by some listeners of compressed audio.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-21 20:21:42
Quote
Quote

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
[a href="index.php?act=findpost&pid=336310"][{POST_SNAPBACK}][/a]


It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

Now you are saying something completely different. I don't see how our brain would try to represent anything else than what it is perceiving. I don't see how a wave form produced by an mp3 decoder is conceptually different from one might encounter in other situations. But ok, I will wait for studies.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-21 21:15:07
Quote
So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!
[a href="index.php?act=findpost&pid=336117"][{POST_SNAPBACK}][/a]


Really, now, would you like to show your evidence?

The evidence that partial loudness differences are overdetected has been in for years.

All auditory stimulii start out as a set of partial loudnesses, as expressed as pulse position modulation in the auditory nerves.

If you're right, this would require a complete revision of the entire understanding of how the human auditory system works.  Have you this revision prepared yet?
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-21 21:19:01
Quote
Sorry, this is not even true. I would like you to show me how you calculate that an 128 kbps mp3 has 90% less information (define information too please) than the original PCM data. Then show me that our auditory system must derive more from a decoded mp3 than from the original PCM AND explain exactly what it is that is being derived. If anything, it seems to me the opposite must be true and working with less data is also is less fatiguing
[a href="index.php?act=findpost&pid=336175"][{POST_SNAPBACK}][/a]


For information, Shannon Entropy will do.

Of course, since MP3 is a middling source coder on top of its perceptual encoding, there's a lot more of the original "information" in the signal than most people realize, simply because it is also a source coder.

I've stayed out of this whole "waveform" thing, because, frankly, it seems based entirely on a heap of confusion in the first place, but at this point, somebody claiming that the ear has to "make up data" just simply denies everything we know about how the auditory system works.

It doesn't "make up" anything, it detects, in a very lossy and mathematically imperfect fashion, whatever is there.  Since that's a lossy fashion, sometimes it imagines that things are there when they aren't, but that's not "making up the missing information" at all.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-21 21:20:53
Quote
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all.
...
[a href="index.php?act=findpost&pid=336251"][{POST_SNAPBACK}][/a]


I agree that information is lost, no doubt. I must, though, point out that the mere act of frequency analysis also provides some "compression" in the LMS sense, of course that is undone by the decoder, not by the ear, something our earstwhile correspondent seems to forget about.
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-10-21 21:54:52
Quote
I recall, we have had successful abx tests even here at ha in the high end area. The guy, who abxed 1 trial each morning, when his ears were fresh.
You don't need special software,
instead of software you need then a helper person, who burns a CD with first 2 files as references, AB, you know, which is which, and then other tracks as several X in random order, but the helper writes somewhere down, which is which.
Then you can play with this cd, and if you think, you are finished, the helper can reveal you the results.




I like the University of Music in Detmold, Germany.
How they carry out abx tests in diploma thesis.
And that they write directly in thier papers against te marketing hypes, now they revealed, no perceivable difference between DSD and 192/24 PCM.


If you're talking about the same paper I am, four out of 145 people passed an ABX (which was defined as at least 15/20 correct).  Those four passed only when using headphones to compare stereo material (SACD vs. 24-bit/176.4 kHz DVD-A).  They each passed using a different musical program.  There was some question as to whether they might have been unconsciously cued by differences in switching noises. 

No one passed an ABX using surround-sound material.

http://www.hfm-detmold.de/eti/projekte/dip..._paper_6086.pdf (http://www.hfm-detmold.de/eti/projekte/diplomarbeiten/dsdvspcm/aes_paper_6086.pdf)

The full thesis in German is somewhere at

http://www.hfm-detmold.de/hochschule/eti.html (http://www.hfm-detmold.de/hochschule/eti.html)
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-10-21 22:01:26
Quote
Quote
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?


Lossless and lossy compression schemes are just that, compression (http://www.computerhope.com/jargon/c/compress.htm). It is in this sense that I mean that an mp3 file, for example, is compressed. (http://www.computerhope.com/jargon/l/lossy.htm).

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision).



People can easily parse sentences with 'degraded' content, such as sentences where 'the the' appears, as well as other typos involving *missing* words or data.
Has it been determined what level of degradation has to be reached before the *time* it takes to parse a sentence is affected?  If two tasks take an effectively indistinguishable amount of time for the same person, then one cannot be said to be more 'difficult' than another, can it?  The only other measure of 'difficulty' I could imagine would be something like, the number of neurons engaged, or the amount of blood flow involved.

Quote
I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.


You haven't demonstrated that it's even an *issue* for lossy schemes. Horses before carts, please.

Ans since we seem still to be in the realm of the suppositional, suppose perception is a matter of stimuli crossing sensory and cognitive *thresholds*.  It could be that as long as the 'lossy' (lossy only in comparison to the original stimulus) stimulus still crosses the right thresholds, it's all the same to the brain.
This would mean that the 'best' 'lossy' representations' are simply those that are *good enough* -- and they don't require any more 'effort' to process.
Title: What's the problem with double-blind testing?
Post by: bryant on 2005-10-21 22:53:28
Quote
Quote
Quote

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.
[a href="index.php?act=findpost&pid=336310"][{POST_SNAPBACK}][/a]


It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

Now you are saying something completely different. I don't see how our brain would try to represent anything else than what it is perceiving. I don't see how a wave form produced by an mp3 decoder is conceptually different from one might encounter in other situations. But ok, I will wait for studies.
[a href="index.php?act=findpost&pid=336351"][{POST_SNAPBACK}][/a]

You have to keep in mind how MP3 encoders (and other lossy encoders) work. To say they "throw away" information is true, but it doesn't make it clear what they are really doing. What they do is reduce the resolution at certain frequencies and times (which is how they save space) and this has the effect of adding quantization noise to the signal. The idea is to essentially add noise to the signal as close as you can get to (but still below) the threshold where it's directly, consciously, audible. What's surprising is that this can be a relatively high level of noise (which is why these lossy codecs work so well). You can easily do a test where you subtract an original music signal from its encoded version. You might be surprised how much noise is added.

So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

What is in question is whether or not noise that is below the threshold where you can actually hear it still has an effect on processing, and what research is showing over and over is that whether or not something is directly perceived has very little to do with the underlying processing. The brain is processing the sounds into a "reality" for your conscious mind to act on, and does not want to bother you with unimportant details that might distract you. What if every stimuli registered by your senses was consciously perceived!!

Imagine walking blindfolded into a large room with sound reflective walls. You instantly become aware of the rough dimensions of the room, but you don't hear every echo of every footstep. A blind person would be able to "hear" in much greater detail, and in fact claim to be able to almost "see" the room. This is the kind of thing that the subconscious mind is constantly doing for us in the background, and what finally gets presented to our conscious mind is rather independent of what processing was required to achieve it. I admit that without at least a working knowledge of the literature, this may seem counterintuitive.

Another argument that was made is that if this extra processing automatic and we are not aware of it, then maybe it has no other effect. The problem with this is that the brain (and all parts of the body) don't work that way. The processing consumes resources (they actually measure this activity by measuring the radiation it generates) and all the body's systems try to minimize resource use. If I ask to you perform a math calculation in your head you would have to concentrate to do it (and you might refuse). If I asked you to do calculations over and over you would probably get tired and annoyed. Just because other processing is unconscious does not mean it's any less taxing, and the brain will always attempt to minimize it.

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.

The reason that I linked to the first of those two papers is that I find this field fascinating from a scientific viewpoint and I suspect that some HA members might be less aware of the richness of research discoveries in this area (compared to some other "harder" scientific realms).

The reason I linked to the second paper is that it directly relates to the auditory system, and points out a case where subconscious perception of a certain auditory stimuli turns out to be much more sensitive than what is consciously perceived. What could be more closely related to this discussion? I can imagine a version of the experiment where levels of noise might be added that, even when well below the threshold of conscious audibility, would have an effect on the measured subconscious perception of the timing. That would be the end of the discussion! That would prove that distortions below the threshold of conscious perception can effect how we hear music (at least to the somewhat open-minded).

Finally, and this was another motivation for me, there seems to be a tendancy here to oversimplify things that really are not that well understood. I am not trying to push any agenda and I am not an audiophile, but I am familiar enough with the scientific research to know that some of the claims often made here and taken as fact (at times in very condescending language) are simply not well supported at this time, and may in some cases turn out to be flatly wrong. I think it's fine if people have different opinions, and the free and lively interchange of ideas is what this board is all about. But I don't think it would hurt for a slightly more flexible attitude to prevail (and it would certainly reduce the possibility of many egg covered faces in the future).

Have a good weekend! 

edit: grammer
Title: What's the problem with double-blind testing?
Post by: rjamorim on 2005-10-21 23:12:36
Quote
But I don't think it would hurt for a slightly more flexible attitude to prevail (and it would certainly reduce the possibility of many egg covered faces in the future).[a href="index.php?act=findpost&pid=336381"][{POST_SNAPBACK}][/a]


You are so absolutely right...
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-21 23:43:14
Quote
Quote
So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!
[a href="index.php?act=findpost&pid=336117"][{POST_SNAPBACK}][/a]


Really, now, would you like to show your evidence?

The evidence that partial loudness differences are overdetected has been in for years.

All auditory stimulii start out as a set of partial loudnesses, as expressed as pulse position modulation in the auditory nerves.

If you're right, this would require a complete revision of the entire understanding of how the human auditory system works.  Have you this revision prepared yet?
[a href="index.php?act=findpost&pid=336364"][{POST_SNAPBACK}][/a]


When I said we aren't "wired" to detect "non-events" in general, I was referring to any phenomena that signal detection theory can be applied. This is a fundamental tenet of the theory; that is, the criterion location depends on the associated costs of different sorts of errors. How that manifests in various auditory contexts is quite variable depending what the task is. Also, it relates to attention and what the sound source is. 

I'm not making myself clear if you think the ideas I've presented require a revision of how hearing works. What I am proposing is a result of processing in the auditory cortex, not the ears, and is consistent with what we know about auditory perception. Your account of partial loudnesses doesn't really seem particularly relevant to the issue of whether processing differences between compressed and uncompressed audio could be measured in a way that ABX testing misses.

Quote
It doesn't "make up" anything, it detects, in a very lossy and mathematically imperfect fashion, whatever is there. Since that's a lossy fashion, sometimes it imagines that things are there when they aren't, but that's not "making up the missing information" at all.


My example of the missing fundamental phenomenon illustrates nicely how unconscious inference works. There are many other examples as well. Our brains construct a good deal of what we "perceive" based on what is often quite degraded information. For example, your visual experience is radically different than the information on your retina. Perception has a huge filling in component.

Quote
People can easily parse sentences with 'degraded' content, such as sentences where 'the the' appears, as well as other typos involving *missing* words or data.
Has it been determined what level of degradation has to be reached before the *time* it takes to parse a sentence is affected? If two tasks take an effectively indistinguishable amount of time for the same person, then one cannot be said to be more 'difficult' than another, can it? The only other measure of 'difficulty' I could imagine would be something like, the number of neurons engaged, or the amount of blood flow involved.


Actually, there is a ton of research on language processing using written sentences, and even something like two "the's" will slow down the system in a statistically significant way. What is often considered a significant amount of time in a cognitive sense is in reality quite fast (e.g., 100 ms. is a big effect), but people generally don't have a conscious sense of these sorts of time issues. Basically it boils down to neural activity. So two different tasks can seem indistinguishable to someone, but have radically different time courses. So yes, this issue is about brain activity, and because the brain is such an energy hog, most roads to easier processing are taken. In terms of the argument David and I are presenting here, the neural processing differences might manifest in long term fatigue effects that could be responsible for some people's discomfort with lossy formats.

One idea that many people seem to be resisting is the notion that very subtle processing differences can result in subtle long term feelings. There seems to be a tendency to overestimate the amount of information that is accessible to consciousness, and as a result, let conscious judgment be the last word on what is actually perceived. But this is definitely not the case as psychophysical researchers (including psychoacoustics people) have known for over a century.

Quote
You haven't demonstrated that it's even an *issue* for lossy schemes. Horses before carts, please.

Ans since we seem still to be in the realm of the suppositional, suppose perception is a matter of stimuli crossing sensory and cognitive *thresholds*. It could be that as long as the 'lossy' (lossy only in comparison to the original stimulus) stimulus still crosses the right thresholds, it's all the same to the brain.
This would mean that the 'best' 'lossy' representations' are simply those that are *good enough* -- and they don't require any more 'effort' to process.


I think I've presented a lot of converging evidence that suggests it could very well be an issue. Like I've said, it's a testable idea. David just presented one idea that would speak to it, and we have other ideas as well. You have to find a cart before the horse can pull it.

 

The threshold theory is just another version of the idea discussed earlier concerning why it is more labor intensive, so to speak, to perceive a degraded stimulus rather than a less noisy stimulus. David's example of listening to speech in a noisy bar is a good example. But I will throw up some references soon about neural processing and stimulus quality. As one should expect, degraded signals are more difficult to process.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2005-10-22 00:00:11
Quote
So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

This is also what isn't debated. Audible differences lie within the scope of an ABX-test.

Quote
What is in question is whether or not noise that is below the threshold where you can actually hear it still has an effect on processing, and what research is showing over and over is that whether or not something is directly perceived has very little to do with the underlying processing. The brain is processing the sounds into a "reality" for your conscious mind to act on, and does not want to bother you with unimportant details that might distract you. What if every stimuli registered by your senses was consciously perceived!!

Well, yes, this is what psycho-acoustic models try to take advantage of.

Quote
Imagine walking blindfolded into a large room with sound reflective walls. You instantly become aware of the rough dimensions of the room, but you don't hear every echo of every footstep. A blind person would be able to "hear" in much greater detail, and in fact claim to be able to almost "see" the room. This is the kind of thing that the subconscious mind is constantly doing for us in the background, and what finally gets presented to our conscious mind is rather independent of what processing was required to achieve it. I admit that without at least a working knowledge of the literature, this may seem counterintuitive.

No this is not what sounds counterintuitive. It sounds counterintuitive that my brain would have to work harder on a lossy compressed signal than on the original. Also note that duff and you are not helping each other, because you say the brain needs to work harder to filter out the noise, and duff says the brain needs to work harder to somehow interpolate the signal and thus creating info. While I might be misunderstanding duff and I found your explanation more plausible, this still doesn't keep me from wondering why it would require more effort. In both cases, all the info must be processed and a decision must be made if it is passed through to my conscience or not. But for as long as both signals carry the same amount of info then the same processing "power" is required, independantly of what my consious receives.

Now I cannot say if a noisy signal would contain more info than a clean one (probably yes, but can we measure this? if as you say a lossless compressed wav is a reasonable guess of the amount of info in a signal then in the 3 tests I have done quickly now the recompressing the lossy files to lossless results in a smaller size than the original thus indicating a smaller amount of info), but all I am asking is some proof for these claims. When people start using phrases like "It goes without saying" and "uncontroversial" I always immediately start asking myself: "Is that so?". And since these claims are made so a hard I don't think it is unfair of me to ask for some sources that confirm this. Enfin...

Quote
opinions, and the free and lively interchange of ideas is what this board is all about. But I don't think it would hurt for a slightly more flexible attitude to prevail (and it would certainly reduce the possibility of many egg covered faces in the future).

Oh I'm flexible and I wasn't throwing with eggs, but I can't do much with mere assumptions if that is all they are. If you say I can't trust my intuition in this area, I have no reason to believe one assumption more than the other if neither is backed with any kind of proof. This has nothing to do with flexibility, but practicality.

Quote
Have a good weekend! 

You too. 
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-22 00:25:04
Quote
What's surprising is that this can be a relatively high level of noise (which is why these lossy codecs work so well). You can easily do a test where you subtract an original music signal from its encoded version. You might be surprised how much noise is added.

hmm, I dont think that is a fair demonstration. Reminds me of the distinction between what is a sound and what is a difference between sounds - is the difference between sounds a sound? Like our stereo perception between our ears perceives the difference between sounds in a totaly different way that we percieve sounds in each ear. This difference between the wav and the encoding although it can be represented [audibly rendered] as a sound, isnt really such a thing.

Then im wondering what its like to percieve a lossy representation of sound in one ear and the lossless version in the other, wonder if that has been tested? 
Quote
So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing.

Perhaps but its not a sure bet.
Quote
Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

That is about interference and masking, but lossy encoders developed performance is dependant on not damaging 'clarity' by estimating those considerations and keeping within ~at least~ 'concentratively imperceptable' limits.

Quote
What is in question is whether or not noise that is below the threshold where you can actually hear it still has an effect on processing, and what research is showing over and over is that whether or not something is directly perceived has very little to do with the underlying processing. ...  Just because other processing is unconscious does not mean it's any less taxing, and the brain will always attempt to minimize it.

I agree with alot of this. Probably no surprise 

A big problem for the forum with these possibilities, is they are the preferred retreat of the (usually unwitting) snake oil technology sellers - despite them being areas difficult to document by nature making instore demos that rely on them ridiculous.

I have this weird sensation sometimes when Im ~half asleep, where I feel sharp sounds from unexpected events like clangs and ticks as a wave of light travelling through my body. Ive wondered if that could be tranquil nerves in the body feeling the sound as it passes through them, or if it is just a kind of hallucination. Perception and conciousness are surely still bewildering and mysterious things.

Regards'
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-22 00:30:19
Quote
It sounds counterintuitive that my brain would have to work harder on a lossy compressed signal than on the original. Also note that duff and you are not helping each other, because you say the brain needs to work harder to filter out the noise, and duff says the brain needs to work harder to somehow interpolate the signal and thus creating info.


Filtering and unconscious inference processes take place in the brain on all sinusoidal waves (light and sound). These processes are taxed by noise.

Quote
This is also what isn't debated. Audible differences lie within the scope of an ABX-test.


David's example addressed the contention that noisy stimuli aren't harder to process. Why would this principle change just because the noise isn't audible? Again, you need to separate the processing from the product of that processing (i.e., the percept).

Quote
When people start using phrases like "It goes without saying" and "uncontroversial" I always immediately start asking myself: "Is that so?".


Frankly, I find it surprising that I need to convince people that degraded signals require more effort to process. This is not the speculative element of the idea. But I understand that our use of the term "effort" is not one people ordinarily consider. I will provide examples of research demonstrating this soon.
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-22 00:47:04
Quote
Frankly, I find it surprising that I need to convince people that degraded signals require more effort to process. This is not the speculative element of the idea. But I understand that our use of the term "effort" is not one people ordinarily consider. I will provide examples of research demonstrating this soon.
[a href="index.php?act=findpost&pid=336405"][{POST_SNAPBACK}][/a]

Because they arent 'degraded' - they are changed and filtered. They arent 'messier', maybe they are notably artificial in respects but - if you are used to refering to the necessarily perceptibly subtle changes allowed by lossy encoding, with perjorative terms as degrade and elsewhere 'impoverished' you'll have a problem with seeing them fairly. That a perceptibly subtle change can be 'strong' or 'powerful' doesnt mean its 'noisy' just different. Im open to the idea there are unconcious differences in their perception, and maybe some unconcious stresses to watch out for in psychoacoustics, but different sound doesnt have to be messier sound.
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-22 00:59:05
A simplistic example - a sine wave of exactly 3122.4873 Hz might be encoded as a sine of 3121 Hz. Loads of waves in a signal might have their accuracies reduced like this, some removed completely, some allowed to be a bit louder or quieter to fit a nicely compressable huffman block. Each sine is still smooth, even though the difference between the encode and the wave will sound like white noise, -there is no sound of white noise in the encode
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-22 01:01:16
-Sorry for flooding a bit there. Tasty subject
Title: What's the problem with double-blind testing?
Post by: duff on 2005-10-22 01:14:41
Quote
Quote
Frankly, I find it surprising that I need to convince people that degraded signals require more effort to process. This is not the speculative element of the idea. But I understand that our use of the term "effort" is not one people ordinarily consider. I will provide examples of research demonstrating this soon.
[a href="index.php?act=findpost&pid=336405"][{POST_SNAPBACK}][/a]

Because they arent 'degraded' - they are changed and filtered. They arent 'messier', maybe they are notably artificial in respects but - if you are used to refering to the necessarily perceptibly subtle changes allowed by lossy encoding, with perjorative terms as degrade and elsewhere 'impoverished' you'll have a problem with seeing them fairly. That a perceptibly subtle change can be 'strong' or 'powerful' doesnt mean its 'noisy' just different. Im open to the idea there are unconcious differences in their perception, and maybe some unconcious stresses to watch out for in psychoacoustics, but different sound doesnt have to be messier sound.
[a href="index.php?act=findpost&pid=336407"][{POST_SNAPBACK}][/a]


The reduction of resolution and introduction of noise into a signal, to my mind, qualifies as a degradation. That's what lossy encoders do. I use this in a technical sense, so I don't mean to disparage any lossy format with "pejorative" intentions! Bad mp3 file! Bad!
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-22 01:26:01
Quote
The reduction of resolution and introduction of noise into a signal, to my mind, qualifies as a degradation. That's what lossy encoders do. I use this in a technical sense, so I don't mean to disparage any lossy format with "pejorative" intentions! Bad mp3 file! Bad!

I know you dont mean that, but you are placing the change in sound in an troublesome context with the terms, when the change is virtualy sculpted as best it can to result in no perceptible troubles. I think it is only an added noise in one abstract sense,  fundamentaly its just a change.

When we listen to a difference track, we hear a jumbled up compound of lots of individual differences, and it sounds bad, but we are not percieving them in context, in their context they were (at least consciously) imperceptable. In their context they are ratios, they are the details of a relationship between one sound and another. Relating to them as the medium which they reference is misleading. 
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-22 07:52:18
Quote
So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

Well, actually, it is.

First, reverberation is way, way above threshold.

A decent perceptual coder (AAC, WMA Pro for instance) is very near, or under, threshold at a reasonable rate.

The two issues are not in any fashion comparable. If there is no way to detect the difference in two signals coming down the auditory nerve, there is noting for the CNS to work on.

Quote
BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.
[a href="index.php?act=findpost&pid=336381"][{POST_SNAPBACK}][/a]


No that's wrong, too.

With a decent lossy compressor, you get about 50% compression. So your 90% would become 45%, even with the odd metric you're using.

A better way is to point out that lossless compression gets you to 50%. THAT is "no loss at all" in the real signal. Ergo, something that runs at 88% is throwing away 38%.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-22 08:00:27
Quote
When I said we aren't "wired" to detect "non-events" in general, I was referring to any phenomena that signal detection theory can be applied. This is a fundamental tenet of the theory; that is, the criterion location depends on the associated costs of different sorts of errors. How that manifests in various auditory contexts is quite variable depending what the task is. Also, it relates to attention and what the sound source is. 

And I am talking about what the auditory system actually seems to do.

Yes, you can change detection thresholds, in theory, but you know what?  The auditory system at its basic level, i.e. partial loudness, seems to clearly overdetect.
Quote
What I am proposing is a result of processing in the auditory cortex, not the ears, and is consistent with what we know about auditory perception. Your account of partial loudnesses doesn't really seem particularly relevant to the issue of whether processing differences between compressed and uncompressed audio could be measured in a way that ABX testing misses.

You argue that there must be a way to filter out overdetections (that always happen in the partial loudness domain) upstream. This is a de-noising question. While I can't prove it does not happen, my experiences (extensive) with psychometric testing is that auditory overdetection is the rule, not the exception.

Given any straw to grab, even null tests (i.e. same stimulus) will show "results". It's been done. Over and over.
Quote
My example of the missing fundamental phenomenon illustrates nicely how unconscious inference works.

It is irrelvant and shows nothing relating to what we are talking about. The partials are way, way above threshold.  A good perceptual coder does not inject noise at that level, period.

They are two entirely different scenarios, perceptual coding and phantom pitch.
Quote
There are many other examples as well. Our brains construct a good deal of what we "perceive" based on what is often quite degraded information. For example, your visual experience is radically different than the information on your retina. Perception has a huge filling in component.

And, again, you talk about information way, way above threshold.
Quote
The threshold theory is just another version of the idea discussed earlier concerning why it is more labor intensive, so to speak, to perceive a degraded stimulus rather than a less noisy stimulus. David's example of listening to speech in a noisy bar is a good example. But I will throw up some references soon about neural processing and stimulus quality. As one should expect, degraded signals are more difficult to process.
[a href="index.php?act=findpost&pid=336391"][{POST_SNAPBACK}][/a]


Before you argue that, explain how something for which the information to enable detection at all (the SNR and integration constants of the auditory system are, after all, pretty well known from phenominon) does not exist can make a difference.

In a noisy bar, the noise, again, is far, far above threshold.

Every one of your examples is using noise that is far, far above threshold. If your coded signals fit that bill, please get a new encoder.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-22 08:07:22
Quote
Filtering and unconscious inference processes take place in the brain on all sinusoidal waves (light and sound). These processes are taxed by noise.

Whoa there.  "Filtering" takes place in the organ of Corti, not in the brain, in the sense of frequency filtering like would be involved for a sine wave.

The eye is not a frequency analyzer, all frequency analysis in the eye is secondary in the brain.

The ear is, fundamentally, a frequency analyzer at the periphery. There's no point in going further with eye/ear comparisons, they aren't the same.
Quote
Frankly, I find it surprising that I need to convince people that degraded signals require more effort to process.

May I politely suggest, then, that you reconsider your idea?

A signal below threshold in the auditory system is GONE.

Noise below threshold is GONE.

AT least 80 years of research shows that it's gone in the PERIPHERY, in the organ of Corti, in the basic transduction.

Quote
This is not the speculative element of the idea. But I understand that our use of the term "effort" is not one people ordinarily consider. I will provide examples of research demonstrating this soon.
[a href="index.php?act=findpost&pid=336405"][{POST_SNAPBACK}][/a]


Your idea is unsupported and beyond speculative.

You will have to show that frequency analysis and information loss takes place after the cochlea if you wish to proceed successfullly.

Greenwood, Zwislocki, Fletcher, Zwicker, etc, all conclude (and I think accurately so) that this happens on the basilar membrane, and that things that are below the basic 30dB SNR of a given neuron in the auditory nerve are gone.

This is supported by a host of evidence, from BLMD to high-frequency unmasking on envelopes, to tone masking noise in ERB's ...

You're pushing way upstream here, and your suggestions would require a complete revamping of both the understanding and, frankly, the evidence of how the human auditory system works.

Again, for signals ABOVE threshold, especially quite above, I agree that you're right. I think, however, that you don't quite grasp what "under threshold" can mean in the auditory system.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-22 08:08:16
Quote
A simplistic example - a sine wave of exactly 3122.4873 Hz might be encoded as a sine of 3121 Hz.
[a href="index.php?act=findpost&pid=336409"][{POST_SNAPBACK}][/a]


Um, no, that's not what happens. The short-term frequency may be varied due to noise (phase noise, FM noise, whatever), but that's a different issue.
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-22 14:09:26
Quote
Quote
A simplistic example - a sine wave of exactly 3122.4873 Hz might be encoded as a sine of 3121 Hz.

Um, no, that's not what happens. The short-term frequency may be varied due to noise (phase noise, FM noise, whatever), but that's a different issue.

The key phrase was 'simplistic example'
(I want duff to understand the 'noise' he is refering to, not run him out of town  )
I know that wasnt your intention woodnville, but..

..wait till we get the links

Lossy digital processes affect the waveform at least with digital rounding error of phase resolutions / frequencies & powers.
The difference which rounding error makes is a very tiny noise, containing only hazy information about the bulk density of the position over the waveform, or not even that perhaps.
Thats a noise that if it was loud would be very noisy*
The 'noise' of psychoacoustic 'info reduction' -attempts to be (or , mostly is) generated in sync with the major details of the source.
On top of the noisy syncronisationInfo loss which is minimised by encoders,
much stronger info losses can be applied to power - a reason for this could be that power signatures can change without causing as big kinks in the waveform as phase/freq signatures
And there are linear charts of masking&pinpointing perception levels used to hack into the power info even more.
Those economisations do not interfere with the waveform, they fit into, when they are removed the reduced waveform carries no details of the economisation any more.
They leave no discernable pattern on the waveform other than the very quiet rounding error chaos.
That would be the ideal, but it might not be a securely achieved idea by some formats & encoders.
Title: What's the problem with double-blind testing?
Post by: bryant on 2005-10-22 23:32:37
Quote
Quote

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.
[a href="index.php?act=findpost&pid=336381"][{POST_SNAPBACK}][/a]


No that's wrong, too.

With a decent lossy compressor, you get about 50% compression. So your 90% would become 45%, even with the odd metric you're using.

A better way is to point out that lossless compression gets you to 50%. THAT is "no loss at all" in the real signal. Ergo, something that runs at 88% is throwing away 38%.
[a href="index.php?act=findpost&pid=336466"][{POST_SNAPBACK}][/a]

Sorry, you're the one who's wrong.

The only relevant measure of information here is the losslessly compressed wav. If we assume, like you did, 50% compression, then the bitrate would be 705 kbps (1411 kbps / 2). To get that down to 128 kbps requires eliminating 82%.

As for your other points, you may be taking the examples a little too specifically. If we knew exactly what effect was going to be discovered we would design an experiment and get it over with. The point I am trying to make is much more general, and is simply that listening tests involving only conscious subject choice may not be sufficient to measure all aspects of auditory perception, and that those areas it cannot properly measure may be involved with music enjoyment. If you think that's a preposterous stretch, good for you.
Title: What's the problem with double-blind testing?
Post by: bryant on 2005-10-22 23:39:40
Quote
I have this weird sensation sometimes when Im ~half asleep, where I feel sharp sounds from unexpected events like clangs and ticks as a wave of light travelling through my body. Ive wondered if that could be tranquil nerves in the body feeling the sound as it passes through them, or if it is just a kind of hallucination. Perception and conciousness are surely still bewildering and mysterious things.
[a href="index.php?act=findpost&pid=336402"][{POST_SNAPBACK}][/a]

I have these too! Sometimes I'll wake up and be certain that the doorbell just rang, and sometimes I feel like something popped inside my head. Very unpleasant! Fortunately these sensations don't happen when I'm fully awake... 
Title: What's the problem with double-blind testing?
Post by: ChiGung on 2005-10-23 00:34:47
just thinking -i do get all sorts of weird sensations when im half awake....
wont go there now,//
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-10-24 21:51:51
Quote
The only relevant measure of information here is the losslessly compressed wav. If we assume, like you did, 50% compression, then the bitrate would be 705 kbps (1411 kbps / 2). To get that down to 128 kbps requires eliminating 82%.[a href="index.php?act=findpost&pid=336612"][{POST_SNAPBACK}][/a]



Well, I think first you need to define what you're measuring.

Now, as to listening to differences, are you aware that DBT's with forced-choice and short stimulii have shown listener sensitivity down to the levels mathematics and science just barely predict are possible?

Look at the sensitivity of the hearing system at absolute threshold, and compare it to the ambient noise due to shot noise of air, for instance. The absolute thresholds are gotten via DBT's.

Look at the in-band sensitivity to tone-masking-noise, compare it to the resolution of a nerve fiber (considering firing rate and the 200 millisecond integration time of the auditory system).
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-10 19:10:52
So sorry for the long delay. I didn't forget about this thread....

I have compiled a number of articles that provide anybody here resources and empirical support for the various claims David and I have been making. I claimed that there is a distinction between sensory and decision processes. This recent Nature article (http://psych.ucsc.edu/~gbryant/ha/decisionsensory.pdf) provides empirical support for this. Cognitive scientists have understood this distinction for decades, but here is recent evidence demonstrating the neurological basis for it. This is a fundamental issue in our criticism of the validity of ABX testing as the last word on ultimate differences in the auditory processing of lossy audio. Acoustic differences that could matter for a listener’s overall experience might not be audible.

Here are two articles that show the difference between decision processes linked to the discrimination of stimuli versus sensory processes that are affected when there is no discrimination. There is clear enhanced brain activity that occurs during the presentation of noisier stimuli even when it cannot be distinguished from a less noisy counterpart.

Effects of Low-Pass Noise Masking on Auditory Event-Related Potentials to Speech (http://psych.ucsc.edu/~gbryant/ha/noisemasking2.pdf)
The Effects of Decreased Audibility Produced by High-Pass Noise (http://psych.ucsc.edu/~gbryant/ha/noisemasking3.pdf)

Woodinside claimed that no filtering processes happen after the ear. This is definitely incorrect, as filtering processes do happen in the auditory cortex (http://psych.ucsc.edu/~gbryant/ha/spatialauditory.pdf) (one of the many analogues between auditory and visual processing), but I’ll assume he meant that the relevant filtering for audio codecs involves processes that happen only in the ear. This is fine, as there’s no reason to distinguish peripheral (inner and outer ear) processes from central processes (auditory cortex) when talking about metabolic costs of neural effort. The neurons must work harder (wherever this happens) to resolve the signal and encode it for later processing. And this does not deal with the likely possibility that the resulting representations might differ depending on stimulus quality (not a necessary feature of our argument).

There has been quite a bit of skepticism regarding the claim that compressed or degraded stimuli require more effort to process. I could cite a list of papers a mile long showing reaction time increases as a function of stimulus complexity (which heavily imply that processing difficulty increases), but there is also work showing that neurons respond to increased attention demands (http://psych.ucsc.edu/~gbryant/ha/attentionneural.pdf). Additionally, there seems to be specialized neural systems for resolving degraded signals (http://psych.ucsc.edu/~gbryant/ha/learningdegraded.pdf), which in turn contribute to learning. These processes require effort, and have associated metabolic costs. This is exactly the sort of thing we have been arguing.

Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-10 19:23:46
Quote
Woodinside claimed that no filtering processes happen after the ear. This is definitely incorrect, as filtering processes do happen in the auditory cortex (http://psych.ucsc.edu/~gbryant/ha/spatialauditory.pdf) (one of the many analogues between auditory and visual processing), but I’ll assume he meant that the relevant filtering for audio codecs involves processes that happen only in the ear. This is fine, as there’s no reason to distinguish peripheral (inner and outer ear) processes from central processes (auditory cortex) when talking about metabolic costs of neural effort. The neurons must work harder (wherever this happens) to resolve the signal and encode it for later processing. And this does not deal with the likely possibility that the resulting representations might differ depending on stimulus quality (not a necessary feature of our argument).[a href="index.php?act=findpost&pid=340974"][{POST_SNAPBACK}][/a]


I'm sorry, but you're simply misusing the term "filtering" from the point of view of a signal-processing person. You are not, I trust, suggesting that the "filtering" that goes on in the CNS results in anything like the outcome of a linear filter, are you?

Psychologists often take words with specific meanings and add new meanings. Personally, I wish they'd use a different word, but in any case, the spatial capture is not a filter in any sense that I would normally use the word. I might use a filter to do spatial capture, that's a different problem altogether.

What you are, once again, ignoring, is that if there is NO INFORMATION AVAILABLE about a given distortion at the auditory nerve, you're arguing for something paranormal to happen later.

The evidence is in for outer hair cells, they depolarize at higher levels. So, there is no more effort going on in the basilar membrane, either.  So, missing information means "no stimulus", ergo if anything it's "less effort" in the periphery.  I think it's safe to say that the "cochlear amplifier" mechanism has gone FORD. (That's "found on road dead" for non-'mercans)

There is simply no evidence, no mechanism, and no non-sighted listener responses that in any fashion support this contention of "extra effort". None.  That's what this all boils down to. One one side we see a bunch of suppositions, hypotheses, and speculation, based on the idea that there must be something wrong with DBT's because they don't observe some things that people think exist.

You show two sets of evidence for noise WELL OVER THRESHOLD. You're using data from a completely different, and irrelevant, situation in order to make your argument.

On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.

That is really what I see on the table here, supposition vs. a century of evidence. If there is something here, some extraordinary evidence for its existance, that addresses the known flaws of human perception, has to come forth.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-10 23:41:26
I don't understand why you would want to maintain a definition of filtering that does not include the action of neurons that differentially respond to frequency-time information in an encoded signal. This is hardly unusual, and psychophysical researchers have been using "filter" in this way for over a century.

[span style='font-size:8pt;line-height:100%']Kowalski, N., Depireux,D. A. & Shamma, S. A. (1996). Analysis of dynamic spectra in ferret primary auditory cortex. II. Prediction of unit responses to arbitrary dynamic spectra. J. Neurophysiol. 76, 3524±3534.[/span]

The reason I pointed that out at all is because you made a blanket statement about the auditory cortex and filtering. Just as you made a blanket statement about privileging "non events" in signal detection. You later said you only meant that in a restricted way, but you originally said something much more general. Anyway, onwards...

A paragraph from here (http://www.healthsystem.virginia.edu/internet/corwin-lab/auditorysystem%20revised%202002%20by%20JTC.doc) speaks at least indirectly to an important issue:

[span style='font-size:8pt;line-height:100%']Measurements carried out in living anesthetized lab animals revealed that the waves of maximal displacement produced by moderate sound levels in the living cochlea are actually much, much sharper than can be accounted for by the physical resonant properties of the basilar membrane in cadavers. That work led to an objective method for screening for hearing disorders in newborns that has become widely adopted in the past 5 years. It turns out that cochlear hair cells can be both receptors and effectors. In response to signals from their efferent innervation outer hair cells exhibit slow contractions and elongations. In addition, outer hair cells exhibit fast contractions in response to sound. Those contractions occur when the cells are depolarized and hyperpolarized.  Currently, the molecular mechanisms that underlie this motility are under intense investigation. Two years ago the apparent motor protein that powers outer hair cell motility was identified and named prestin. Physiological experiments have established that the local resonance of the basilar membrane is directly modulated as a result of active changes in the length of outer hair cells, but just how that results in highly-tuned mammalian hearing performance is not yet understood.  [/span]

Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal. But auditory processing doesn't work that way. I'd like to see any empirical support for your argument surrounding effort and noise even at that level. But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The articles I provided show that even when discrimination cannot be made between two stimuli that differ in how much noise they contain, neural systems are treating them differently. That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes. All we have to show to make our basic case, like the articles I presented do show, is that people can be unable to discriminate between two stimuli (rendering them equivalent in ABX terms), while their brains act differently during the presentation of them. The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

Quote
On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.


Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-11-11 13:34:04
Sorry, you still have provided no evidence that what you say applies to inaudible differences. So, you have no basis to say that an ABX test (which is just about audible or inaudible differences) may not be adequate because of these reasons.

Related to this, I believe that very subtle audible differences have been detected in DBTs even when the subjects at test couldn't consciously hear a clear difference.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-11 17:37:59
Let's back up a moment...

There are at least two different arguments going on here. One is whether there is any evidence that degraded stimuli, or lossy compressed stimuli require greater effort to resolve by perceptual systems. I think I have provided evidence that is true. There is no principled reason why this would not apply to stimuli that we are not consciously aware of (e.g., inaudible). Earlier I discussed examples in vision where reaction times were longer in a reading task on a computer screen as a function of flicker rate, but subjects had no conscious awareness the stimuli were different. This is a perfect example of increased effort affecting processing that is not consciously accessible.

The other point relates to whether aspects of stimuli that we cannot see or hear affect our perception, and this is obviously true. There is a huge literature on the attentional blink (http://en.wikipedia.org/wiki/Attentional_blink), there are studies like the flicker rate study I just mentioned, as well as various other phenomena where subjects do not consciously notice some stimuli, but the perception of that stimuli is revealed by paradigms that tap into unconscious processes. For example, people look at a cross-hatch and are asked to judge whether the vertical line is longer than the horizontal line, and a small shape appears very close to the cross-hatch. Subjects claim they do not see the shape but in a yes/no recognition task, they take longer to say no to shapes they saw, than shapes they didn't see.

And the list goes on. Point being, conscious reports (which ABX tests rely on) are not totally reliable indicators of what affects perceptual systems.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-11-11 18:57:56
About the first point, your degraded examples are *hugely* degraded, as opposed to what average quality lossy compression does. Then you say that there is no reason why the phenomena you describe would not apply to inaudible difference stimuli. Well, if it's inaudible, there's a great difference. I think it is too much to assume that it would apply equally, and I think I'm not being over-skeptical here.

About the second point, would those visual "degradations" (not consciously perceived) be perceived as some kind of difference when the subject is actually trying to see them, or when subjected to an A/B test? I mean, is it that those differences are not consciously perceived, but just under casual "looking", or without a reference? For example, subtle audible differences are sometimes not perceived when doing casual listening, but when paying attention, or in a close comparison with a reference (A/B test) they are easily distinguishable.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-11 20:19:26
Quote
Well, if it's inaudible, there's a great difference


This represents a general feeling many people have, but in fact, I'm not so sure. In terms of information processing, there's no reason to invoke different principles to describe the relationship between stimulus characteristics and processing effort. The fact that it's "inaudible" in an ABX sort of way just means that the percept is not processed by a system that is consciously accessible. I can't stress enough how many processes happen that are NOT conscious.

Quote
About the second point, would those visual "degradations" (not consciously perceived) be perceived as some kind of difference when the subject is actually trying to see them, or when subjected to an A/B test? I mean, is it that those differences are not consciously perceived, but just under casual "looking", or without a reference? For example, subtle audible differences are sometimes not perceived when doing casual listening, but when paying attention, or in a close comparison with a reference (A/B test) they are easily distinguishable.


In phenomena like the attentional blink, attention has no impact on one's percept. Same with the flicker rate study. People cannot tell that the flicker rates are different, even when you draw their attention to it. The reason their reaction times are different is because certain relationships between their eye movement speeds and the flicker rates can cause people to repeat scans over text in extremely small ways (we're talking eye movements here, so these are tiny distances). There is no feeling associated with this other than rather long-term fatigue.

In other paradigms, like the cross-hatch/shape studies, of course when you direct someone's attention, they see the shape. The point of that work is to show that items not consciously perceived are actually processed. So, as I keep repeating, aspects of compressed stimuli might affect processing, and conscious reports might not reflect it.

Look, this is a testable hypothesis, and is not unreasonable given what we know. I keep trying to explain it because it's clear many people here are not very familiar with research in psychophysics and cognition. It is hardly a radical proposal that something like an ABX test does not capture all aspects of perceptual processing...in fact, it's a given. Whether or not this applies to lossy audio remains to be seen.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-11 20:52:16
Quote
Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal.

I propose nothing whatsoever of that sort. There is lots of processing. My point is simple, if there is no detectable remains of an original signal at the auditory nerve, then nothing the cortex does can recover what isn't there in the first place.

And that's what it seems quite clear that basic auditory masking is, the limits of the cochlea to encode the stimulus in the first place.
Quote
But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The fact that their firing is graded with level is irrelevant. We're talking about the situation in which the noise from the PPM encoding (that's what neural transmission is, after all, PPM) is large enough that a change in firing rates or first-arrivals is not detectable given the known integration times of the auditory system.

First arrival is indeed the most sensitive, and if you think a minute, it's obvious that a rapid increase in firing rate after a period of low rate is quite easily detectable.  The converse not so easily detectable.
Quote
That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes.

I have done no such thing. Please do not presume to lecture me or to put elementary, obvious fallacies in my lap.

I am ignoring nothing.

The signals in the experiments you cite are way, way over both absolute and masked thresholds.  They are audible.  They absolutely create detectable information in the auditory nerve.

A signal below masking threshold does not.

Your experiments are, I will presume, correct for what they measure, but they are not in any fashion relevant to this discussion.
Quote
The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

Now, exactly what is your concern with, DBT's, or ABX tests, and for what use? Detecting thresholds or something else?
Quote
Quote
On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.


Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.
[a href="index.php?act=findpost&pid=341041"][{POST_SNAPBACK}][/a]


You can not say "will not be reflected", you haven't shown that.  What you've shown is a discrimination test between two impaired signals with easily audible impairments, not a test of JND's.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-11 20:56:47
Quote
Sorry, you still have provided no evidence that what you say applies to inaudible differences. So, you have no basis to say that an ABX test (which is just about audible or inaudible differences) may not be adequate because of these reasons.

Indeed.
Quote
Related to this, I believe that very subtle audible differences have been detected in DBTs even when the subjects at test couldn't consciously hear a clear difference.
[a href="index.php?act=findpost&pid=341106"][{POST_SNAPBACK}][/a]


Indeed they have. Subjects have more than once reported that they were "mostly guessing", could not articulate differences or sometimes even internalize the idea that they heard difference, but some of those subjects have still discriminated against a very slightly impaired signal at the 99.9% level. (18/20, for instance)

What's interesting is that in my experience, quite some of these subjects, who insisted that they were guessing, were right.  Quite a few more of the subjects who insisted that they heard "obvious differences" hadn't.

Trained listeners are very often humble, having been through those experiences, and will "sally on" even when convinced they didn't hear anything, only to find out that they did.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-11 20:59:53
Quote
In other paradigms, like the cross-hatch/shape studies, of course when you direct someone's attention, they see the shape. The point of that work is to show that items not consciously perceived are actually processed. So, as I keep repeating, aspects of compressed stimuli might affect processing, and conscious reports might not reflect it.[a href="index.php?act=findpost&pid=341166"][{POST_SNAPBACK}][/a]


You keep avoiding the obvious point.  The stimulus, crosshatch in this case, is way, way, way, way way above threshold, both absolute and masking.

The issue here is what happens to parts of a signal that are below absolute or masking thresholds.

And we are talking about the ear.

You must recall that unlike the eye, the periphery in the ear is a frequency analyzer. No such facility exists in any substantial form (lateral inhibition being the only thing to speak of) in the eye.  The eye, of course, is a 2D transducer, the ear is time/frequency, which you can argue is 2D, but with one dimension as time, you can't "hold" a stimulus there like you can for the eye.
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-11-12 01:32:47
Quote
Subjects have more than once reported that they were "mostly guessing", could not articulate differences or sometimes even internalize the idea that they heard difference, but some of those subjects have still discriminated against a very slightly impaired signal at the 99.9% level. (18/20, for instance)[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=341174")


Like Garf ? [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=6651&view=findpost&p=70284]http://www.hydrogenaudio.org/forums/index....indpost&p=70284[/url]

Getting 99.8 % just guessing. Fine...
But he was not only guessing, the stumulus was just not being presented to him  !
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-11-12 02:00:14
Quote
The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. [a href="index.php?act=findpost&pid=341041"][{POST_SNAPBACK}][/a]


An ideal ABX test should be run under the exact same conditions under which the difference tested was first observed. This way, only differences relying on a decision process will be tested.

This works if ABX tests are used to check the validity of people's claims about sonic differences.

I can understand your criticism of ABX if ABX tests are used in order to evaluate a threshold of hearing, for example. In this case, there is no pre-existing decision process that the ABX test is supposed to validate. Differences smaller and smaller are just submitted to the listener.
I find this method rather flawed. The listener must have the possibility to train himself. For example, it becomes easy to pass ABX tests about MP3 compression once you have heard MP3 artifacts, while you can fail the same test if you don't know what to listen to. It happened to me with the Smashing Pumkins sample provided by Xerophase. I could hear no differences, but since he managed to get a successful score quite easily, I listened and listened to the samples again, until I finally found a small difference.

Once the listener is trained, it all becomes a decision process, and the ABX test can be applied. If you apply the ABX test without first training the listeners (and ITU-R BS.1116-1 recommendation for blind listening tests says that It is important that data from listening tests assessing small impairments in audio systems should come exclusively from subjects who have expertise in detecting these small impairments), then you take the risk to get a failure while your subjects are actually capable of passing your test. This could indeed be a situation where the reaction time detects a difference that the decision process doesn't detect (yet).
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-12 08:33:10
Quote
I can understand your criticism of ABX if ABX tests are used in order to evaluate a threshold of hearing, for example. In this case, there is no pre-existing decision process that the ABX test is supposed to validate. Differences smaller and smaller are just submitted to the listener.
I find this method rather flawed. The listener must have the possibility to train himself.[a href="index.php?act=findpost&pid=341210"][{POST_SNAPBACK}][/a]



Your objection is invalid.

Curiously enough, it has been shown in very graphic ways (i.e. by doing it) that one of the most effective ways of training on simple signals is to just decrease the probe signal from obvious, slowly, trial after trial, until the subject fails to hear it at some given level, is given at least two tries after going up and back down a bit, and then taking that as threshold.

It's called "signal detection", it's old and well-established in the art.

Did it occur to you that what you, yourself, describe, when done in a sensible fashion, is a self-training test?

Now, nobody's ever argued the need for training that I know of, well, nobody who has done testing and observed subject performance, so I must have missed something about your point, of COURSE the () subject has to be trained one way or another.
Title: What's the problem with double-blind testing?
Post by: user on 2005-11-12 09:52:45
There aren't theoretical flaws with abx or DB tests.
The practical and theoretical problems and limits of these tests are well known in various sciences.
I think, they were all mentioned already here at ha and also in this topic.

To the problem of abx-ing unconcious differences:
differences you cannot even listen by short A/B comparison,
you can vary the setup of an abx-test.
Extend the abx test to a long time period, listen each day for 30-xxx minutes, and rate eg. your fatigue.
I don't know now, if this has been tried already for good  (or even medium/low quality) lossy vs. CD.

of course, if such a test results to 50% , ie. pure guessing and same fatigue values for lossy and orignal,
it doesn't prove, that lossy eg. is not more fatiguing than original.
As known, abx test can only prove (with certain probability) one side of  a hypothesis (that was to be tested).
Title: What's the problem with double-blind testing?
Post by: Cartoon on 2005-11-12 12:40:01
Quote
I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[a href="index.php?act=findpost&pid=335816"][{POST_SNAPBACK}][/a]


That is why, IMO, all this focus on lossy encoders is a bit outdated. Encode to lossless into your music archive/repository. Then play from there at home, stream to your stereo or connect a PC directly to the stereo. Filesize of lossless is not a big deal these days, very few have music collections that would require disk storage worth more than a few hundred to maybe a thousand dollars for a fully mirrored setup. Most could do with much less. The CDs are the backup anyway, but it takes time to rip, so some extra money into harddisk-crash protection is nice. The advantage is that you now also got a backup of your CDs, since you can recreate a bit-perfect(1) copy of the original CD if it dies from scratches of UV-exposure.

Then for portable use, transcode from the repository to mp3, aac, or whatnot.

In this context, what is "best" is moot. Not happy, just re-transcode to a different format or bitrate.

(1)Audio ripping is error prone,  but if using EAC in secure more and/or test&copy with AccuRip there it is quite likely that your copies are bit-perfect. There are also other factors, but please don't nit too much 
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-11-12 14:24:00
Quote
Your objection is invalid.  [...]

Curiously enough, it has been shown in very graphic ways (i.e. by doing it) that one of the most effective ways of training on simple signals is to just decrease the probe signal [{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=341258")


You're right, my example is the opposite of what I wanted to illustrate.
My point was that in some way, training oneself = bringing the process from the "reaction" realm into the "decision" realm. But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction. If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

As I use to tell in french forums, I've never seen a hifi cable saleseman arguing "Buy this cable, I guarantee you'll never realize that it improves your high fidelity system" !

Quote
I don't know now, if this has been tried already for good  (or even medium/low quality) lossy vs. CD.[a href="index.php?act=findpost&pid=341272"][{POST_SNAPBACK}][/a]


It's currently done with power cables. Each listener has got 7 days to give one answer in a A/XYZ test (is X, Y or Z equal to A ?).
[a href="http://www.hifiwigwam.com/forum1/1614.html]http://www.hifiwigwam.com/forum1/1614.html[/url]
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-12 20:05:33
Quote
If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

This is one of the main points I have been trying to address. Just because some perception is unconscious does NOT mean that it cannot affect one's enjoyment. It most certainly can.

Long term ABX approaches might be better at getting at it.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-12 20:16:39
Quote
Quote
If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

This is one of the main points I have been trying to address. Just because some perception is unconscious does NOT mean that it cannot affect one's enjoyment. It most certainly can.

Long term ABX approaches might be better at getting at it.
[a href="index.php?act=findpost&pid=341374"][{POST_SNAPBACK}][/a]



I have to agree with this to some extent. If people can think they are "guessing" and be getting 18/20 on a binomial choice, (not one, or two, but enough to be way, way outside the joint or entire distribution by lots and lots of sigma no matter how you cut it), one can't argue that they are unaffected.

On the other hands, they ARE getting the right answer in the test, too, so you can't use that to argue that the test does not work.
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-11-14 03:45:10
Quote
Quote
If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.

This is one of the main points I have been trying to address. Just because some perception is unconscious does NOT mean that it cannot affect one's enjoyment. It most certainly can. [a href="index.php?act=findpost&pid=341374"][{POST_SNAPBACK}][/a]


The difference is between an unconcious perception, and a perception with no concious effects.
The first can affect enjoyment, thus can be ABXed, if enough effort is put into the test.
The second can't affect enjoyment and can't be ABXed.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-16 18:31:04
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction. If the listener can't make a decision about it, then the difference must be completely unconciuous, then it can't contribute to his enjoyment while listening to music.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.

Quote
The difference is between an unconcious perception, and a perception with no concious effects.


These are the same. It is not understood to what extent unconscious percepts affect experience. They might in quite subtle ways.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-16 18:41:36
Quote
My point is simple, if there is no detectable remains of an original signal at the auditory nerve, then nothing the cortex does can recover what isn't there in the first place.


If the auditory cortex is involved in inferential processing, then the more degraded the signal, the harder it is to infer acoustic properties of it. An example is the missing fundamental phenomena. That is a clear example of the cortex recovering (i.e., through inference) information that is not present in the stimulus.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-16 19:49:28
Quote
If the auditory cortex is involved in inferential processing, then the more degraded the signal, the harder it is to infer acoustic properties of it. An example is the missing fundamental phenomena. That is a clear example of the cortex recovering (i.e., through inference) information that is not present in the stimulus.
[a href="index.php?act=findpost&pid=342343"][{POST_SNAPBACK}][/a]


Irrelevant. You've reversed stimulus and probe.

The NOISE is below threshold, not the SIGNAL.

There is nothing "missing" to work on.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-16 20:20:42
Reversed stimulus and probe? This phenomenon just refers to the percept associated with a stimulus that requires unconscious inference. This has nothing to with noise, or probe stimuli. I am addressing your claim that the auditory cortex cannot recover missing information from a stimulus. Perhaps you only mean that in a very limited sense, but as you write it, it is relatively sweeping, and incorrect.

The reason I point this out is because you seem to disregard the importance of cortical processing (and instead emphasize peripheral processes), which is curious since we are talking about decision processes used in ABX tests which are by definition not just perceptual.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-17 01:02:32
Quote
Reversed stimulus and probe? This phenomenon just refers to the percept associated with a stimulus that requires unconscious inference. This has nothing to with noise, or probe stimuli. I am addressing your claim that the auditory cortex cannot recover missing information from a stimulus. Perhaps you only mean that in a very limited sense, but as you write it, it is relatively sweeping, and incorrect.
[a href="index.php?act=findpost&pid=342363"][{POST_SNAPBACK}][/a]


In the future, you should learn the terminology before you criticize.

In my point about masking, the information on the MASKED signal, which is not the music signal, but is rather the noise (coding otherwise). This noise is not expected and is not present in the stimulus (the music) but is the interfering (probe) signal.

The information about these masked signals is not distinguishable from the information available at the far end of the auditory nerve.  My statement is qualified, is as sweeping as appropriate, and given all results from the psychometric community, precisely correct.

You insist on "inference" in situations which no inference is being made. You can start there with your presumptions. The "inference" is only going to occur if you attempt to hear noise, and then you have exactly the same problem with or without noise.

That, in and of itself, shows fully the internal contradiction inherent in your presumptions.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-17 06:45:47
In the future you should learn the appropriate context in which you make your arguments.

You clearly have little understanding of the role of unconscious inference in perception, and you instead interpret everything through your understanding of the audibility of masked noise in the creation of audio codecs (without realizing that's not what I'm talking about). Granted, this is one element of the debate here, but only one among many. The more general point that you fail to get is that decision processes involved in ABX judgments can be oblivious to information processing issues not well measured by conscious judgments. I have provided abundant evidence showing this principle in a variety of ways/domains.

I might add that you have provided no evidence supporting any of your positions.

Quote
The "inference" is only going to occur if you attempt to hear noise, and then you have exactly the same problem with or without noise.


No. Unconscious inference is a basic feature of low level auditory and visual processing, and is independent of conscious behavior. As a signal is encoded, characteristics are inferred, and these inferences vary as a function of the stimulus characteristics and the expectations of the system. For example, light information on the retina is quite impoverished compared to the eventual percept that people experience. We construct much of what we perceive. It has little-to-nothing to do with what people "attempt" to do. You are misunderstanding this use of the term "inference."

The missing fundamental phenomenon has nothing to do with masked noise, nor expectation, other than the "expectation" pitch perception systems have when encountering a complex sound (which is not conscious). People perceive a fundamental frequency in degraded spectral information that does not contain a fundamental. You claimed that the auditory system cannot recover information that is not there, and this is without a doubt false. Instead of acknowledging that, you just start talking about masked signals. It's non sequitur.

You also don't address the issue that there might be metabolic costs associated with processing noisy signals, other than just saying that can't be. Where's your falsifying evidence? I've shown that there are specialized neural systems that deal with degraded stimuli in the visual system (which require extra energy to implement), and all you can say is that because the noise is above threshold, there is an important distinction to be had with regard to metabolic costs of neural processing. I'm not convinced of that at all, and I don't think there is any clear research supporting it.

We do seem to agree that different implementations of the ABX paradigm could better capture long-term effects of processing on listeners' experiences. But given how ABX tests are currently used to test listeners' discriminations, tasks involving motor responses might capture differences that cognitive decision (i.e., verbal report) tasks don't capture.

Other than that, I don't think we are accomplishing anything by continuing. I think you have made good points about the role of peripheral hearing processes in the creation of lossy codecs, but you don't have a familiarity with psychophysics or cognitive science research that examines the full chain of processing from sound to percept, and this is causing us to argue in circles.
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-11-17 20:19:58
Quote
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.


Yes, but I was looking for an audio example.

Quote
Quote
The difference is between an unconcious perception, and a perception with no concious effects.


These are the same. It is not understood to what extent unconscious percepts affect experience. They might in quite subtle ways.
[a href="index.php?act=findpost&pid=342342"][{POST_SNAPBACK}][/a]


I don't mean the same thing.
A sound without concious effects would be a sound that we didn't pay attention to, and that don't affect our behaviour. It was not concious when it was perceived, and then had no further effect at all.
But an uncoucious sound can sometimes have concious effects. I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre). If we don't have the "absolute ear", we are not concious that the pitch is higher (about half a tone higher), but we find the vocal expession of the actors less "serious". The whole movie can seem less dramatic. We can be concious of this effect, while not being concious of its cause.

In an ABX test, if we try, several monthes apart, to recognize the pitch of the movie, we usually fail (exept some rare musicians who practice every day, and can notice the scale difference in the music score). But if we concentrate on the voice timbre, we can succeed. This is an example of stimulus with a concious side (the voice timbre), and an unconcious side (the pitch).

This way, if a sound difference can lead, by unconcious ways, to a perceived difference in quality for a given listener, there is always a possibility to ABX it by decision, granted that the decision is made on the right perception.
In last resort, the test must be run in the exact conditions that lead the listener to realize the difference in quality.
To tell "this seems to sound better to me" is a decision process in itself. If it only comes after several monthes of listening experience, then it will always be possible, though impractical, to ABX it in several monthes per trial.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-18 01:53:34
Quote
Quote
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.


Yes, but I was looking for an audio example.


I did provide two studies showing that people can be unable to discriminate audio stimuli, but brain reactions were different. David has also linked to an article (http://www.apa.org/journals/features/xhp273600.pdf) showing how phase correction occurred in synchronized motor movements due to unconsciously perceived elements in an auditory stimulus. Anyway, we are going to do the critical specific experiment I think you want.

Quote
A sound without concious effects would be a sound that we didn't pay attention to, and that don't affect our behaviour. It was not concious when it was perceived, and then had no further effect at all.
But an uncoucious sound can sometimes have concious effects.


And an unconscious sound can have unconscious effects. Careful about privileging conscious effects though, that is a bit of an illusion. Most of what our brains are doing is unconscious. But your example is good, and shows that long term effects need long term testing strategies (to extend as much as possible the sensitivity of decison-based tests).

Quote
This way, if a sound difference can lead, by unconcious ways, to a perceived difference in quality for a given listener, there is always a possibility to ABX it by decision, granted that the decision is made on the right perception


I agree that having people attend to different aspects of stimuli can affect ABX results in important ways. But that doesn't mean any two stimuli, no matter how slight the difference, can be tested with a decision based test. Many experiments show how motor tasks reveal perceptual differences where verbal tasks fail. ABX-minded people should be open to this possibility - it's not the end of the world, and despite what some folks here would have you believe, it is entirely possible in the context of lossy codecs!

Did I mention that I was NOT an audiophile? I don't care what the ultimate truth is - I have no investment emotionally, materially, or otherwise. I record my own music 16/44.1 for godsakes. But some people do have motivations to believe what they believe beyond plain curiosity. They should be prepared to revise their view I think. I am.

Title: What's the problem with double-blind testing?
Post by: Woodinville on 2005-11-18 03:08:23
Quote
In the future you should learn the appropriate context in which you make your arguments.

The context of these remarks is the idea, the unproven, extraordinary assertion that perceptual codecs make people somehow have to listen harder.
Quote
You clearly have little understanding of the role of unconscious inference in perception,

You have no basis whatsoever to make that accusation.  As shown below, you are attempting to escape from the context of this discussion, and introduce issues irrelevant to this context.  You have shown no evidence for the idea that percpetual codecs make listeners work harder, and when compelling evidence has been handed you, you have attempted to change the subject, while making egregious professional accusations.

Quote
and you instead interpret everything through your understanding of the audibility of masked noise in the creation of audio codecs (without realizing that's not what I'm talking about).

The context of this discussion is the contention that somehow perceptual codecs cause listeners to  work harder.  You are attempting to escape from context./
Quote
Granted, this is one element of the debate here, but only one among many.

It is the context of this discussion. You may wish to introduce other issues, but I am continuing to examine your original, unproven, and extraordinary claim.  Before we can move on to more of your theories and contentions, you must prove your extraordinary claim, and you have yet to produce either a relevant argument or any evidence whatsoever of your claim.


Quote
The more general point that you fail to get is that decision processes involved in ABX judgments can be oblivious to information processing issues not well measured by conscious judgments. I have provided abundant evidence showing this principle in a variety of ways/domains.


Oddly enough, now you're talking about ABX tests, even though the context is DBT's.

Furthermore, it is I who has referred informally to data that shows, incontrovertably, that DBT judgements can easily and trivially show evidence of more than concious judgement.  Therefore, your accusation is preposterous, contentious, and completely false.

Furthermore, I have provided evidence in a germane setting, that of an audio DBT, wherein subjects thought they heard nothing, but did, in fact, get very accurate identifications.

You, on the other hand, have introduced discussions of tests where interferers are very far above threshold, rather than at or below threshold, which, at least for most audio testing, wherein it is attempted to discover any difference.  Your results may be accurate, but they are simply not germane, and do not apply to testing wherein the interfering signal is at or below threshold.

Quote
I might add that you have provided no evidence supporting any of your positions.

You are supporting the extraordinary position, not I. Do not attempt to shift the burden.

It is you who owe us evidence, lots of evidence, testable, verifiable, falsifiable evidence that can be repeated by others.  Evidence, that is, for the contention IN THE CONTEXT OF THIS THREAD, not for some other attempt to evade the issue as here stated.
Quote
No. Unconscious inference is a basic feature of low level auditory and visual processing, and is independent of conscious behavior.

First, we are talking about audio.  So, let's stick to audio.

Second, you appear to be claiming that the organ of corti does "inference". That is the "low level auditory processing" that actually exists.  What evidence for "inference" can you show for the organ of Corti?

Let's start there. Because my point about masked signals applies at that level.

If you're talking about once you're past the periphery, well, certainly many things happen. One of the things made clear by much subjective testing is that training can alter the concious (and, since subjects can clearly obtain some information that they can not verbalize and regard as "guessing" even though it's not, unconcious as well) response of everything once we leave the auditory nerve.

That's not in doubt.  It does not, however, impact the validity of DBT's in any fashion.

Your assertion, however, exactly requires that information be discernable at the end of the auditory nerve.  When a signal is below absolute threshold, or masked below tone-masking-noise thresholds, there is no evidence that any discernable information exists regarding such a signal, and there is nothing for this unconcious process to use.

THAT is my point.
Quote
[a href="index.php?act=findpost&pid=342482"][{POST_SNAPBACK}][/a]


As I haven't said the missing fundamental had anything at all to do with masking, your reply is simply disingenious and deceptive. The harmonics that are processed into the missing fundamental are very far above threshold, generally, thank you, so why do you introduce the issue at all? 

Your fallacy of the excluded middle, in suggesting my position of "no metabolic cost" is based entirely on your attempt to evade the context, which is the claim that there is a metabolic cost for listening to perceptual codecs.  There is undoubedly metabolic cost for hearing speech in noise, etc, that news is as old as the hills, and was determined before convenient things like PET allowed us to measure it directly.

That, however, is irrelevant in its entirety to the issue of the "effort' required to hear the music, or some alleged missing information, from a perceptual coder.

You failed to notice that I did point out that if one EXPECTS noise, and thereby tries to find auditory features related to the noise, that that EXPECTATION may indeed cause more work.

finally:
Quote
but you don't have a familiarity with psychophysics or cognitive science research that examines the full chain of processing from sound to percept, and this is causing us to argue in circles.

You have absolutely no grounds to make this egregious professional insult.

You have, time and again, avoided context, argued at straw men, engaged in name-calling rather than offer substantive responses, and you still have failed to produce a single shred of evidence, ordinary, let alone extraordinary, for the claim made in this thread, that perceptual codecs (I will cheerfully stipulate at reasonable rates, let's not get into these low-rate noisemakers, please) cause a metabolic load because some process is trying to extract the "missing information".

What have you done? Shifted context. Tried to shift the burden of proof away from your extraordinary claim.  Issued extremely crude, offensive, and absolutely disgusting professional accusations. Fallacies of the excluded middle, attempts at a magician's force question, yes. Evidence? No.

But no evidence. None, at least related to the subject at hand.
Title: What's the problem with double-blind testing?
Post by: krabapple on 2005-11-18 03:27:13
Quote
Quote
Quote
But I fail to find an example of stimulus that could not be ABXed by decision, but only by reaction.


I have provided a couple of examples, most notably the flicker rate study I described. This represents possible stimulus pairs that could not be ABX'd by a decision process, while reaction times studies show a difference.


Yes, but I was looking for an audio example.

Quote
Quote
The difference is between an unconcious perception, and a perception with no concious effects.


These are the same. It is not understood to what extent unconscious percepts affect experience. They might in quite subtle ways.
[a href="index.php?act=findpost&pid=342342"][{POST_SNAPBACK}][/a]


I don't mean the same thing.
A sound without concious effects would be a sound that we didn't pay attention to, and that don't affect our behaviour. It was not concious when it was perceived, and then had no further effect at all.
But an uncoucious sound can sometimes have concious effects. I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre). If we don't have the "absolute ear", we are not concious that the pitch is higher (about half a tone higher), but we find the vocal expession of the actors less "serious". The whole movie can seem less dramatic. We can be concious of this effect, while not being concious of its cause.



There is a much more notorious example:  the tendency for a slightly *louder* of two otherwise identical sound presentations, to be perceived/reported as having higher *quality*  (rather than being perceived/reported as louder).

It is a case, like the one you cite, where an *effect* is perceived ('conscious') but its cause is misinterpreted ('unconscious').
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-18 06:55:09
Quote
You have shown no evidence for the idea that percpetual codecs make listeners work harder


If you cannot see the relevance of the many lines of research that speak to this issue, that's too bad. I have explained the idea, shown suggestive evidence that the idea could be right, and have described how it can be tested. I am going to test it. Can't really do much more than that. I'm not attempting to evade any issues - I am merely trying to get you to see the connection between different sorts of research that relates to the problem at hand.

You think I've resorted to name calling? That's weird, because I've done no such thing. I have noted that I think you have a weak grasp of the cognitive science research we are discussing. That's not name calling - that's just me calling it as I  see it. Sorry if that offended you - I just think you are misunderstanding me, and you say stuff that is just wrong.

For example, processing that takes place after the Organ of Corti is properly considered low-level. The nature of that encoding process and the subsequent effects downstream are poorly understood scientifically by anybody. Your professed certainty about auditory encoding demonstrates you aren't familiar with the literature. For instance, you acted like it was preposterous that cortical neurons can act like a linear filter. I showed you an article demonstrating how neural activity in the auditory cortex behaves as a linear filter, and you came up with some bizarre accusation that isn't the kind of filter you mean. It is well established that A1 neurons act like a linear filter, and this is meant, by neuroscientists, in the mathematical sense of the term. That was just one article amongst many demonstrating such a thing. I included that not to divert the discussion, but to refute what was a blatantly incorrect assertion on your part.

(http://hyperphysics.phy-astr.gsu.edu/hbase/sound/imgsou/anerv2.gif)
Sound localization happens in the auditory cortex, and the mechanisms implementing that function (in the Superior Olives) are considered low-level.

I don't ever change the subject ultimately - I correct you when you make mistakes regarding the various topics under discussion, which, by the way, all relate to my hypothesis (something by definition "not proven"). Somehow you are concerned with one proposal that I made about lossy codecs being harder to process, but really this is secondary to my main point about the limits of ABX testing. You are over-concerned about that detail.

Also, while this thread is about double-blind testing, all that really refers to is the situation where neither the experimenter nor subject are aware of the experimental condition on any give trial. The primary method under discussion is ABX testing, which is a specific method, not a principle. So that's why I am talking about ABX testing - not odd at all I would say.

Quote
It is the context of this discussion. You may wish to introduce other issues, but I am continuing to examine your original, unproven, and extraordinary claim.


Everything I have brought up is relevant, but all you keep claiming is that when noise is below threshold, it cannot affect the encoding process. I disagree. So you can stop saying that now.

When I get a result that shows people are unable to discriminate between two stimuli with different levels of inaudible noise in an ABX paradigm, but they respond differently on a reaction time task, then what will you say? All of my conjecture is geared towards explaining why this result is very possible. Again, if you disagree, fine, but you can stop with the Organ of Corti business, even though I know you love it.
Title: What's the problem with double-blind testing?
Post by: Pio2001 on 2005-11-19 23:51:27
Quote
I did provide two studies showing that people can be unable to discriminate audio stimuli, but brain reactions were different. David has also linked to an article (http://www.apa.org/journals/features/xhp273600.pdf) showing how phase correction occurred in synchronized motor movements due to unconsciously perceived elements in an auditory stimulus. [a href="index.php?act=findpost&pid=342715"][{POST_SNAPBACK}][/a]


Yes, it is always interesting to learn, and I thank you for providing all these informations.

I had a look at the first one that deals with masking ba/da sounds with noise. In fact, the presence of brain activity does not mean that a reaction process is possible, while a decision one is not.
In the article about finger tapping, I just read the first experiment, with the change in tempo (Figure 2). The change has not been conciously perceived, but I don't agree with interpreting the adjustment of the finger tempo as an unconcious perception of it. It is just the permanent mechanism of adjustment that follows the change.
Looking at the diagrams that show the synchronization error during the unaltered sequences (Figure 2 a), we can see that people's synchronization is better that the change in tempo. The error bars in the left part are smaller that +/- 10 ms. Not adjusting the finger tapping in order to follow the change would have eventually lead to a conciously perceived asynchonicity.
My conclusion is that the change was conciously perceived, but wrongly interpreted as a random variation of the finger tempo, that was, as the diagram shows (figure 2b, 2c, 2d) corrected the beat following the detection. Thus hidden among all the other corrections that were made by the subject in order to keep the right tempo.
People were asked if they perceived a change in tempo, and said that this was not the case. But maybe if they had been asked "did you always tap in phase, or did you readjust your tempo at a given time", people would have answered that they had to readjust in b, c, and d case only.
Furthermore, in music class, we are trained, for rhythm dictation, to ignore small deviations, and only report musically relevant information. These people all had a solid musical background. Maybe they actually perceived the change in tempo, but interpreted the question as "did the score went from allegro to presto" (which was indeed not the case), while the question actually ment "was the conducer as accurate as a metronome" (which was not the case).

Quote
And an unconscious sound can have unconscious effects. Careful about privileging conscious effects though, that is a bit of an illusion.
[a href="index.php?act=findpost&pid=342715"][{POST_SNAPBACK}][/a]


Maybe. But in fact I am more interested in debunking false claims about sound quality. We are surrounded by unfounded information about 15,000 € CD players, or "unlistenable" 320 kbps MP3. And all these claims assume that the difference is conciously audible. Showing that this is not the case can be acheived with ABX tests.
Maybe there are also unconcious differences, that work in these cases, but it is already such a hard work to show that there are no "kickass differences" between two interconnect cables, and that silver interconnect cables do not emphasize treble, that I'm not planning to go beyond ABX (or AXY, or any other decision pattern) for the time being.
Title: What's the problem with double-blind testing?
Post by: duff on 2005-11-20 04:42:17
Quote
Maybe. But in fact I am more interested in debunking false claims about sound quality. We are surrounded by unfounded information about 15,000 € CD players, or "unlistenable" 320 kbps MP3. And all these claims assume that the difference is conciously audible. Showing that this is not the case can be acheived with ABX tests.


I agree. There are certainly many false claims out there, and ridiculously priced cables and gear are based largely in pseudoscience. I'm totally with you there. All I'm suspecting is that some of this nonsense might be rooted in auditory processing issues that ABX testing doesn't tap into. I have no interest in arguing for something that real audiophile freaks could latch onto. I just think it's an empirical question worthy of study.

Quote
In fact, the presence of brain activity does not mean that a reaction process is possible, while a decision one is not.


You're right - the point of that article is about how sensory and decision processes are dissociated, and the brain is doing different stuff on noisier stimuli even though decision processes are treating them the same.

I think it is a bit of a stretch to claim that the participants in the tempo synchronization study actually perceived the change, and just needed to be asked differently. For one, with studies like these (and with this researcher), you can rest assured it has been replicated before it was published, and that sort of confound would have been dealt with early. But more importantly, the more parsimonious story, scientifically, is that the participants' motor movements were driven by a system that had access to different information about the stimulus than the cognitive/decision system. This is well established in psychophysics research. The Roelof's effect I've talked about before is a good example.
Title: What's the problem with double-blind testing?
Post by: Halcyon on 2005-11-20 16:40:54
Quote
Also, psychoacoustic research already found that quick switch, short length stimuli, blind listening tests have proved to be more much more sensitive that casual listening.


Hey KikeG, do you have some reference for this?

I'm by no means claiming it's not true, but I haven't come across this myself, but would very much like to read any relevant papers on the issue.
Title: What's the problem with double-blind testing?
Post by: KikeG on 2005-11-20 18:22:40
I can't cite any papers, but I've read this many times from people knowledgeable in psychoacoustics.
Title: What's the problem with double-blind testing?
Post by: djcombes on 2009-12-16 00:45:46
I'm new to the forums here, but I must say I'm finding the contributions here very, very interesting. I'm an engineer/scientist by trade, developing technology for a living so I'm interested in getting to the bottom of things. Recently I've been investigating hi-fi with the intention of trying to work out what's worth worrying about and what isn't.

I found this discussion very interesting so I thought I'd try to resurrect a long dead argument. I'm sure that no-one is interested in my opinion, not being a cognitive scientist or having much idea about how lossy compression works, but for what it's worth, I found some arguments from both sides convincing and learned a lot.

I don't think it's reasonable to argue with his assertions (and evidence) showing that ABX testing can't pick up on some unconscious things. Even I've heard of reaction times as a method to by pass the decision gubbins to try to get a better handle on what's going on.

So it seems entirely reasonable to suggest that a test methodology for audio compression that does not rely on conscious inference would be interesting, since it might show effects below the threshold at which they are available for conscious inference.

What I'm not clear on, and I don't think can be taken as a 'given' at this stage (before the proposed experiments) is that such effects will occur for the case of compression (at bitrates above 128kbps). I think the examples that have been shown so far show cases in which a stimulus with a clear mechanism for interfering with perception is present. From what I understand, the principles on which audio compression routines are based avoid such mechanism as far as they are known.

There seems to be an assertion that the brain will certainly recognise a compressed signal as degraded, and therefore work to reconstruct it. I'm not convinced that the brain will be able to infer that there is anything to be reconstructed in the sound, and I reckon the understanding of these processes are not well enough understood to be able to predict that there is likely to be a metabolic overhead associated with compressed audio.

So I can see the point of carrying out some experiments, but I don't think that the outcome can be pre-judged.

I think a lot of the negative reaction may have stemmed from the suggestion that inaudible acoustic differences might 'matter' for a listener's overall experience. I really don't see that this is supported by the evidence, and would be music to the snake oil salesperson's ears. If the metabolic overhead is unconscious, it's hardly been shown to be significant or material to the experience.

Anyway - can anyone report with an update? Have any such experiments been carried out?
Title: What's the problem with double-blind testing?
Post by: Arnold B. Krueger on 2009-12-19 15:11:13
I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre).


Just because the FPS change, there's no necessity for the pitch or timing to change. People working with films and video have been managing this situation for several decades.

This article explains how FPS changes are have been managed for decades so that they don't affect the length or running speed of film or video:

http://www.zerocut.com/tech/pulldown.html (http://www.zerocut.com/tech/pulldown.html)

In the old days, the production paths for sound and video were separated, the video or film was altered using pull down techniques, and the audio was added back into the finished product with  no changes. Or, the sound was added back into the finished product with a different FPS from a more origional source.

These days, most video production software manages things like this automagically.  You just specify the FPS of the finished product, and it is produced in accordance with your spec under the covers, as it were.

In short, your example has a rather serious flaw.


Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 14:28:01
I too found these discussions interesting.  I think one relevant intuition is this.  Someone has just failed an ABX test where he (or she) switches quickly back and forth between two signals.  He says to himself, "Still, there might be a difference between these sources that matters to me.  When I listen to one signal for a long stretch, perhaps in a relaxed way, I might get a different experience from it than from the other signal.  I might hear things in one that I do not hear in the other.  Or I might get greater pleasure from one than the other.  I might not be very good at comparing those experiences, since they are separated in time.  But what matters to me might be that I have certain experiences, not that I know or be able to report that I do." 

The intuition, then, is that something in those signals could make a difference to perception, even if that difference is not itself perceived, or the subject of awareness or report.

That is the skeptical worry, and I have never seen a really convincing response to it.

Mark
Title: What's the problem with double-blind testing?
Post by: psycho on 2010-01-16 14:59:38
I have an idea as how to test what djcombes and Mark DeB are trying to get tested... There should be a reality show that lasted for a month or so and the contestants would have to listen to a lot of music and they would never know the source and they would always have to grade their experience level, aswell as guess if it came from original source or if it was a compressed sample.

All in all, I'm sure the results would show that there would be no difference in experience grades between original sources and compressed ones. I am sure that somedays the contestants would grade compressed samples better and then other days they would grade the originals better...

Anyway... such a prolonged test would be interresting and would definetly bring more insight and more evidence for this debate.

Maybe some of the hydrogenaudio forum readers are rich enough or influent enough to make this kind of test possible?

EDIT: typos
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-16 19:26:02
It's a BS argument simply because there are no time limits to ABX testing.  Listen to X for a day and then Y for another day (or whatever period of time makes you happy), and give your answer; repeat the process 16 times regardless of how long it takes.  If you truly perceive a difference it will be apparent in your score.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 20:23:25
It's a BS argument simply because there are no time limits to ABX testing.  Listen to X for a day and then Y for another day (or whatever period of time makes you happy), and give your answer; repeat the process 16 times regardless of how long it takes.  If you truly perceive a difference it will be apparent in your score.


Not sure what argument you are referring to, but the skeptic in my example is not supposing that he perceives a difference, only that there might be a difference in what he perceives.

Consider:

(1) You perceive a as X and you perceive b as Y, and X is different from Y.
(2) You perceive that a is different from b.

There is a logical difference between these.  In (2), "different" falls within the scope of "perceives."  In (1), it does not.

The skeptic does not think that (1) implies (2).

EDIT: typo.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-16 20:36:39
My perception of X as A and not B implies that I believe I can differentiate either X from Y or A from B since by definition X either equals A or B, and consequently Y equals whatever X does not (assuming that A does not equal B).  Regardless of what your skeptic thinks, he has no credible argument.

Semantics aside, if the "experience" between A and B is truly different to the experiencer, then there is no reason it can't be measured by a double-blind test.
Title: What's the problem with double-blind testing?
Post by: pdq on 2010-01-16 20:41:56
You have completely missed the point of ABX. You are presented with X which is the same as either A or B, and is different from the other one! The latter statement is the same as your statement #2, but to prove that you can hear a difference between A and B you must choose which one it sounds different from, A or B.

Edit: greynol beat me to it.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 22:00:31
My perception of X as A and not B implies that I believe I can differentiate either X from Y or A from B since by definition X either equals A or B, and consequently Y equals whatever X does not (assuming that A does not equal B).  Regardless of what your skeptic thinks, he has no credible argument.

Semantics aside, if the "experience" between A and B is truly different to the experiencer, then there is no reason it can't be measured by a double-blind test.


That may be your definition of "perceives X as A and not B" (to change the variables in the original example), but there is no necessity for the skeptic to accept that definition.  Why does perceiving necessarily imply anything about having beliefs about what you can differentiate?  Animals perceive, but they likely do not have any beliefs about what they can differentiate.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 22:03:57
You have completely missed the point of ABX. You are presented with X which is the same as either A or B, and is different from the other one! The latter statement is the same as your statement #2, but to prove that you can hear a difference between A and B you must choose which one it sounds different from, A or B.

Edit: greynol beat me to it.


No, I was saying that the person in the example acknowledges that he can't hear a difference.  Please look again at my post.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-16 22:21:43
What part about semantics aside do you not comprehend?

Perceive a difference can be the result of anything.  It could be a feeling in your stomach or a taste in your mouth.

The ability to perceive something as one thing and not the other means you can differentiate the two.

If the skeptic is not interested in understanding how ABX works, why should anyone care what he has to say about it?
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-16 22:22:05
I think that ending up essentially arguing about what consciousness is in order to find something wrong with ABX-type tests is pretty funny.

But maybe I am too literal-minded.

Anyway, no offense meant to anybody.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 22:36:46
Look, I just did an ABX of the same piece at different sampling rates and I couldn't hear a difference.  And I asked, What exactly does this tell me?  Does it tell me that those files don't differ in properties that, when I listen in 10-minute stretches, cause me to have different types of experiences?  (Where this difference is one that matters: where I have an interest in having experiences of one type over the other.)

As far as I can see, it does not tell me that, and I have never seen a convincing argument that it does.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-16 22:51:11
It certainly doesn't demonstrate that you actually perceive a difference, either.  If you think you can do better beyond 10 minute stretches, feel free.

Personally I think you're just grasping at straws in an effort to not accept the fact that you are merely deluding yourself.

Either way, you have failed in your attempt to discredit ABX as a tool to identify one's ability to perceive/distinguish/whateveryouwannacallit differences.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 22:51:40
I think that ending up essentially arguing about what consciousness is in order to find something wrong with ABX-type tests is pretty funny.

But maybe I am too literal-minded.

Anyway, no offense meant to anybody.


None taken.  FWIW, I wasn't saying that there was anything "wrong" with ABX tests.  The question is what they do or do not prove.  One thing that raised my eyebrows is the assertion way back in this thread that negative ABX results "debunk" certain claims.  I suppose it depends on what one means by "debunk" and for what purposes.  The question here is, do the files in my example differ in their properties in a way that causes you to hear different things, when you listen normally for long stretches of time?  I have never seen a convincing argument as to why a negative ABX test settles that question.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 22:59:15
It certainly doesn't demonstrate that you actually perceive a difference, either.  If you think you can do better beyond 10 minute stretches, feel free.

Personally I think you're just grasping at straws in an effort to not accept the fact that you are merely deluding yourself.

Either way, you have failed in your attempt to discredit ABX as a tool to identify one's ability to perceive/distinguish/whateveryouwannacallit differences.


Fine, but please note that that isn't what I was attempting to do in the first place, so I don't think you understood what I was saying.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-16 23:12:17
What part about semantics aside do you not comprehend?

Perceive a difference can be the result of anything.  It could be a feeling in your stomach or a taste in your mouth.

The ability to perceive something as one thing and not the other means you can differentiate the two.

If the skeptic is not interested in understanding how ABX works, why should anyone care what he has to say about it?


RE: "The ability to perceive something as one thing and not the other means you can differentiate the two."

Says you.  I walk into a museum and something looks like a Picasso to me.  Does that mean that I can (reliably) differentiate Picassos from non-Picassos? 

RE: "If the skeptic is not interested in understanding how ABX works, why should anyone care what he has to say about it?"

I don't know; who said the skeptic wasn't interested in that?
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-16 23:17:10
I think that ending up essentially arguing about what consciousness is in order to find something wrong with ABX-type tests is pretty funny.

But maybe I am too literal-minded.

Anyway, no offense meant to anybody.


None taken.  FWIW, I wasn't saying that there was anything "wrong" with ABX tests.  The question is what they do or do not prove.  One thing that raised my eyebrows is the assertion way back in this thread that negative ABX results "debunk" certain claims.  I suppose it depends on what one means by "debunk" and for what purposes.  The question here is, do the files in my example differ in their properties in a way that causes you to hear different things, when you listen normally for long stretches of time?  I have never seen a convincing argument as to why a negative ABX test settles that question.


Well, induction can't actually prove anything anyway. Obviously an a priori argument that results obtained by performing double-blind (for instance) tests on 1-minute pieces apply also to listening to a whole symphony is impossible to construct; just how much generalisation one is willing to allow will depend on one's assumptions and model of what is going on. Using "my model" of what is occurring here, it seems obvious that if something can't be ABXed using eg 15s of music then I won't be able to ABX it with longer pieces either. Clearly you have a different underlying model.

At any rate, one can ABX over longer periods as well.

Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them
Title: What's the problem with double-blind testing?
Post by: LaserSokrates on 2010-01-16 23:23:51
ABX is about whether you can notice the difference between two similar stimuli. It does not prove if two stimuli are the same. An ABX test determines the probability that you actually can make out the difference between the two. However, if the same ABX test is being done by a larger user group, certain assumptions can be made. It's all about statistics.

Quote
Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them

No, one camp uses scientific evidence within a certain, small range of uncertainty while the other camp is zealous about it's unfounded belief.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-16 23:24:51
I have never seen a convincing argument as to why a negative ABX test settles that question.

Individual ABX tests that are negative don't prove the impossibility that someone will never perceive a difference.  This has never been disputed by those who understand the methodology.  It's hardly any basis to demonstrate that you can perceive a difference, however.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-16 23:31:30
Says you.  I walk into a museum and something looks like a Picasso to me.  Does that mean that I can (reliably) differentiate Picassos from non-Picassos?
Non sequitur.  I never suggested that one had to identify the precise origin of a sample under test, nor are we comparing apples to oranges, either.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-16 23:33:40
Mark DeB, your point just doesn't hold. You claim there might be an longterm benefit to A that B hasn't and that the difference between A/B might not be noticeable by direct (short term) comparison. As others have said, nothing prevents you from extending test exposure to longer periods, say a whole day. After, for example, a month, you should have marked several days in your calendar with "experienced effect" and some with "no effect" then compare the result with the actual settings.

The strongest effect I have experienced so far is sight. The effect is so strongly perceivable, especially when comparing audio stuff, that it's almost funny. I have to bind myself to blind protocol to not be fooled.

The effect, which you believe to be possibly at work here, is very improbable. Nevertheless it would be verifiable with extended testing. I also would not call your position truly sceptic, rather the opposite. You seem to just resort to a ultrasceptic position to sustain a non-sceptic belief (possibly induced by sighted comparison).

Not being able to perceive a difference between two things means there is no perceivable difference for that subject. The domain of reality outside that scope of that subject's perception has nothing to do with it.
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-17 00:31:24
ABX is about whether you can notice the difference between two similar stimuli. It does not prove if two stimuli are the same. An ABX test determines the probability that you actually can make out the difference between the two. However, if the same ABX test is being done by a larger user group, certain assumptions can be made. It's all about statistics.

That's obviously a true statement, but I am not sure why this was posted as a reply to my comment. Unless you're trying to say that one can increase the probability of something being different by having an arbitrarily large group take part in the test, which is also true but also irrelevant.

Quote
Quote
Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them

No, one camp uses scientific evidence within a certain, small range of uncertainty while the other camp is zealous about it's unfounded belief.


Actually it is literally true that both camps (here, I am not talking about magazine editors lying through their teeth for instance) are in fact using their common sense, and it is just that it's telling them different things. I do agree that it's much more reasonable to assume that results obtained by using samples of (for instance) 20s apply also to general music listening than not to, but that's because my commons sense tells me so (ie because I can "watch" myself listening to music and it seems that no integration occurs over such timescales, while also I've observed myself get fooled  by my hearing often enough to not trust subtle results indicating counterintuitive effects--the results would have to be compelling).

Anyway, you can't actually prove it except by induction, ie performing the experiments and showing that the same conclusion is reached. Given that I consider this generalisability to be obviously true, I won't bother  Or you could show that no mechanism exists that would result in long-term listening revealing features that cannot be noticed on short-term listening, but I don't think such an approach would work in this type of psychoacoustical research.

All I am saying is that in this particular case, it seems that Mark DeB is not doing this to defend his irrational beliefs but rather seems to actually be wondering about it. And if you think this is being stupid, that's fine, but I don't. I disagree with what he believes and think it's obviously incorrect, but that's all.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 01:04:31
Says you.  I walk into a museum and something looks like a Picasso to me.  Does that mean that I can (reliably) differentiate Picassos from non-Picassos?
Non sequitur.  I never suggested that one had to identify the precise origin of a sample under test, nor are we comparing apples to oranges, either.


No, but it is a counterexample to the principle you stated, since we do sometimes perceive paintings as Picassos.  If you want to make your principle more precise to rule out this counterexample, please feel free.

Look, you are saying (yes?) that from the fact that I can't perceive a difference between two stimuli in a quick-switch ABX test that I should conclude that those stimuli, when I listen in long stretches, don't cause different experiences in me?  Why does that follow, on the basis of either logic or science?  How do I know, for example, that when I listen in long stretches I don't have a different perceptual set from the one I have in the testing situation, and hear different things in response to the stimuli, but lack the ability to compare the experiences because they're too far separated in time?
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-17 01:14:41
lack the ability to compare the experiences because they're too far separated in time?

You don't lack the ability to compare the experiences because they're too far separated in time.

It seems like every time someone comes along with silly excuses for their inability to demonstrate they aren't simply imagining differences, the conversation devolves into nitpicking over comments that have nothing to do with the principles being discussed.

If you think you can fit your absurd Picasso example into the workings of ABX testing, feel free.  Otherwise, you're wasting our time.


Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 01:16:25
Mark DeB, your point just doesn't hold. You claim there might be an longterm benefit to A that B hasn't and that the difference between A/B might not be noticeable by direct (short term) comparison. As others have said, nothing prevents you from extending test exposure to longer periods, say a whole day. After, for example, a month, you should have marked several days in your calendar with "experienced effect" and some with "no effect" then compare the result with the actual settings.

The strongest effect I have experienced so far is sight. The effect is so strongly perceivable, especially when comparing audio stuff, that it's almost funny. I have to bind myself to blind protocol to not be fooled.

The effect, which you believe to be possibly at work here, is very improbable. Nevertheless it would be verifiable with extended testing. I also would not call your position truly sceptic, rather the opposite. You seem to just resort to a ultrasceptic position to sustain a non-sceptic belief (possibly induced by sighted comparison).

Not being able to perceive a difference between two things means there is no perceivable difference for that subject. The domain of reality outside that scope of that subject's perception has nothing to do with it.


Right, no perceivable difference, but that isn't the same as no difference between perceptions.  If you listen to a one-minute stretch of music, and then a similar one-minute stretch, you might very well perceive different things, but not be able to detect and/or report this reliably because you  can't compare them.  And there is no need to assume that experience implies the ability to apply a label (such as "experienced effect").
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-17 01:23:34
More useless semantic gymnastics. A difference is a difference and ABX tests measure differences.  Face it, Mark, you got nothing.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 01:32:37
lack the ability to compare the experiences because they're too far separated in time?

You don't lack the ability to compare the experiences because they're too far separated in time.

It seems like every time someone comes along with silly excuses for their inability to demonstrate they aren't simply imagining differences, the conversation devolves into nitpicking over comments that have nothing to do with the principles being discussed.

If you think you can fit your absurd Picasso example into the workings of ABX testing, feel free.  Otherwise, you're wasting our time.


The separation in time might or might not be the salient reason.  But in any case, the question stands: why should I conclude, from what I can or cannot do in the quick-switch ABX situation, anything about whether the stimuli cause different experiences in the long-term setting? And I see nothing in your posts that actually gives a reason why I should conclude that.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-17 01:34:54
Why are you insisting that we needlessly constrain ABX to a quick-switch situation?
Title: What's the problem with double-blind testing?
Post by: Axon on 2010-01-17 01:42:23
Settle down, chil'ren.

Mark, you are correct that under any strictly logical analysis, a negative ABX result cannot "debunk" the existence or non-existence of a difference. Nonexistence is nonfalsifiable. However, it can disprove the magnitude of such a difference, if it exists, and often times that's enough to settle a debate. If one says "OMG this is a night and day difference, I can detect the improvement in half a second", but then undergoes an ABX test under identical listening conditions and fails it miserably, the most plausible reason for the failure is not that there's some fundamental issue with ABX testing that fundamentally compromises its meaning (although that excuse is profferred all the time). The most plausible reason is that the originally perceived difference was 100% imaginary, and among many reasons why this is the most plausible, the one I prefer is that I observe that behavior in myself, regularly, in comparisons which plainly null out, and based on my experience and what I know of human psychology, I expect that behavior in everybody else.

However, when blind tests consistently fail, and there are good theoretical reasons to doubt that such tested differences would ever be audible, at some point, you've gotta step back and call a turd a turd. A while ago there was a big controlled study showing that homeopathy did not demonstrate any efficacy, and while that was used by doctors to "prove" that homeopathy was a worthless fraud, it clearly could not justify such a conclusion - and it didn't, at least not on its own. But combined with the knowledge that homeopathy treatments are diluted to the point of not containing a single molecule of the original substance, and the knowledge that no remotely scientifically plausible theory exists that would in any way justify its efficacy, even after over 100 years of study, you'd need some pretty thick blinders (or big cahones) to assert anything different than "homeopathy is dead".

In other words... bone up on your Kuhn. Paradigms have never been overturned because of evidence.

Regarding quick-switch, I might offer (anybody correct me if I'm wrong here): a) No evidence exists that long-time switching is any more sensitive than quick switching; b) Considerable anecdotal evidence (and I want to say at least one study) indicates quick switching being more sensitive than long time switching; c) The seconds-long duration of auditory memory provides an extremely powerful theoretical justification for the increased sensitivity of fast switching. There is simply no evidence supporting such a theoretical objection, some (perhaps a significant) amoung of evidence against the idea, and a very good theoretical reason against the idea.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-17 02:06:28
Why the Picasso's analogy. I've been recently convinced that people who favor pseudoscience are astoundingly bad at analogies. "Perceiving" a painting as a Picasso would be like perceiving a piece as Mozart. Never mind that you're, you know, not even comparing the same piece!

How about this, forget who painted it, and compare the original with an extremely good forgery. A blind test should tell you if you got the chops to spot the differences.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 02:13:37
ABX is about whether you can notice the difference between two similar stimuli. It does not prove if two stimuli are the same. An ABX test determines the probability that you actually can make out the difference between the two. However, if the same ABX test is being done by a larger user group, certain assumptions can be made. It's all about statistics.

That's obviously a true statement, but I am not sure why this was posted as a reply to my comment. Unless you're trying to say that one can increase the probability of something being different by having an arbitrarily large group take part in the test, which is also true but also irrelevant.

Quote
Quote
Anyway, I think both "camps" are applying common sense, and the only difference is what their common sense tells them

No, one camp uses scientific evidence within a certain, small range of uncertainty while the other camp is zealous about it's unfounded belief.


Actually it is literally true that both camps (here, I am not talking about magazine editors lying through their teeth for instance) are in fact using their common sense, and it is just that it's telling them different things. I do agree that it's much more reasonable to assume that results obtained by using samples of (for instance) 20s apply also to general music listening than not to, but that's because my commons sense tells me so (ie because I can "watch" myself listening to music and it seems that no integration occurs over such timescales, while also I've observed myself get fooled  by my hearing often enough to not trust subtle results indicating counterintuitive effects--the results would have to be compelling).

Anyway, you can't actually prove it except by induction, ie performing the experiments and showing that the same conclusion is reached. Given that I consider this generalisability to be obviously true, I won't bother  Or you could show that no mechanism exists that would result in long-term listening revealing features that cannot be noticed on short-term listening, but I don't think such an approach would work in this type of psychoacoustical research.

All I am saying is that in this particular case, it seems that Mark DeB is not doing this to defend his irrational beliefs but rather seems to actually be wondering about it. And if you think this is being stupid, that's fine, but I don't. I disagree with what he believes and think it's obviously incorrect, but that's all.


Well, thank you for the vote of confidence, but what exactly is it that you take me to believe that you disagree with, or think is obviously incorrect?  I am saying that for all I know, those stimuli could have different causal properties, of a certain kind, that I care about.  Are you saying that the quick-switch ABX results furnish enough information to rule that out?  On what basis?  Yes, of course this is a question of science, or induction.  But you seem to be saying that you believe this because it is "reasonable," or "common sense," or on the basis of introspection perhaps, or "obviously true," none of which is scientific.

I think that you have put your finger on it when you say that it would have to be proved that "no mechanism exists."  What is more, there would have to be some theory of experiences and what it is for them to "reveal features," and of the causal conditions for such experiences.

I still think that, knowing what I know about how I did on the ABX test, it's an open question whether those signals could cause different types of experiences in me, perhaps in other settings such as longer-term listening.  The general tenor of the replies seems to have been that it is not an open question and that I should conclude that those signals could not do so.  But I think it would be rash to conclude that.  And no one here has stated any real scientific basis why that conclusion should follow.  Why, then, do you think I am "obviously incorrect" in not making said inference?
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-17 02:19:04
I also would not call your position truly sceptic, rather the opposite. You seem to just resort to a ultrasceptic position to sustain a non-sceptic belief (possibly induced by sighted comparison).

I agree. Mark DeB, you're being "skeptical" about something, but completely credulous about something else for which there's no evidence (nor any clear physical or physiological mechanism through which it could credibly happen). Furthermore, you're inventing "maybes" to support your hypothesis. "Maybe there's something that no one can detect, but long-term will make you feel different." Well do you have any ideas as to what it might be? Put forward something clear so it can be tested. Anything that can be felt by your senses (and not invented by the brain), can be detected. Your body is full of detectors!

As for the long-term feeling stuff, it can also be ABX'ed as others already said. Just takes more time, but the procedure is the same. You just listen blindly for a long time, and if you believe it's the touchy-feely version, mark it on your chart. Rinse and repeat. What's the problem there?
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-17 02:24:13
Right, no perceivable difference, but that isn't the same as no difference between perceptions.  If you listen to a one-minute stretch of music, and then a similar one-minute stretch, you might very well perceive different things, but not be able to detect and/or report this reliably because you  can't compare them.  And there is no need to assume that experience implies the ability to apply a label (such as "experienced effect").


Ok, nice, let me summarize, then:

Subject S declares:



Why in hell should S then care about "hidden differences" in any scenario, where 1), 2), and 3) are true and are expected to stay true*. It is totally irrelevant wether real differences exist, for example if A has been low passed at 100000 Hz and B has been low passed at 100001 Hz. So why should S care?

* Else a long term double blind A/B could give a definitive conclusion for S as it has been repeated here already many times. S would note down his feelings and after enough data has been collected, it would be correlated against the sequence of As and Bs as they actually went along with S. The result will either be noise or show a degree of significance.

The problem is perfectly decidable. The answer is either there is a detectable difference for S or there is none. The question is just how far you are personally willing to go.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 02:33:11
Why are you insisting that we needlessly constrain ABX to a quick-switch situation?


I'm not.  But will a person always be able to report, reliably, whether two longer-term experiences were the same or different in character?  The worry is that certain experiences could be different but that we are not very good at detecting or reporting that.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 02:36:23
Right, no perceivable difference, but that isn't the same as no difference between perceptions.  If you listen to a one-minute stretch of music, and then a similar one-minute stretch, you might very well perceive different things, but not be able to detect and/or report this reliably because you  can't compare them.  And there is no need to assume that experience implies the ability to apply a label (such as "experienced effect").


Ok, nice, let me summarize, then:

Subject S declares:

  • no perceivable A/B difference
  • no perceivable side effect of A
  • no perceivable side effect of B


Why in hell should S then care about "hidden differences" in any scenario, where 1), 2), and 3) are true and are expected to stay true (else a long term ABX could give a definitive conclusion for S as it has been repeated here already many times). It is totally irrelevant wether real differences exist, for example if A has been low passed at 100000 Hz and B has been low passed at 100001 Hz. So why should S care?


Because he doesn't assume that the only differences between experiences that matter are ones he is able to monitor and report on. 
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-17 02:43:13
Well, thank you for the vote of confidence, but what exactly is it that you take me to believe that you disagree with, or think is obviously incorrect?


I think that your central point is that results from ABX-type tests with short pieces of music do not necessarily hold for long-term listening. In terms of logic this is true, and also if I just look at the "experiments" themselves, they literally say nothing about long-term listening, unless I make some assumptions about the "listening mechanism" in our heads. Apparently, given your assumptions about this mechanism, they do not, but according to mine they do.

Quote
I am saying that for all I know, those stimuli could have different causal properties, of a certain kind, that I care about.  Are you saying that the quick-switch ABX results furnish enough information to rule that out?  On what basis?  Yes, of course this is a question of science, or induction.  But you seem to be saying that you believe this because it is "reasonable," or "common sense," or on the basis of introspection perhaps, or "obviously true," none of which is scientific.

I think that you have put your finger on it when you say that it would have to be proved that "no mechanism exists."  What is more, there would have to be some theory of experiences and what it is for them to "reveal features," and of the causal conditions for such experiences.


Right, like if I claim that the fact that the sun has risen from the east for millenia (let's just assume this) does not prove that it will tomorrow. It really doesn't. Coupled with the usual models of what is actually happening, though, and given that these models are in accord with other pieces of information (falling rocks, observed motion of other lights in the sky under the assumption they're planets etc), it's hard to not conclude that it will.

Here, neither side has the equivalent model (yes a lot is known but nothing as simple, general, accurate and reliable), so it's less certain.

Quote
I still think that, knowing what I know about how I did on the ABX test, it's an open question whether those signals could cause different types of experiences in me, perhaps in other settings such as longer-term listening.  The general tenor of the replies seems to have been that it is not an open question and that I should conclude that those signals could not do so.  But I think it would be rash to conclude that.  And no one here has stated any real scientific basis why that conclusion should follow.  Why, then, do you think I am "obviously incorrect" in not making said inference?


Technically it is. I, however, am not aware of a single piece of evidence to support your thesis, nor any mechanism to suggest so. Obviously (again  ), this does not disprove it, but...

The "obviously" should probably have been quoted, as I meant it as shorthand for "in light of my mental model of the world, it would be really strange if it was not true that".

I don't think this kind of belief has the same status as believing the earth is flat, but I still would be quite surprised if it was true (as I said, lack of evidence, no personal experience indicating it etc).
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-17 02:44:28
Because he doesn't assume that the only differences between experiences that matter are ones he is able to monitor and report on.


If there is any relevant causal influence it will leave a trace. And that trace can be correlated to the actual succession of A and B wether S notes this information consciously or "dumps" it unconsciously (when reporting its emotional state). If you are talking about causal influences that are even smaller than that, well, that's usually considered as noise. Confusing noise with relevant information will actually degrade any degree of knowledge S can obtain about his world.
Title: What's the problem with double-blind testing?
Post by: lvqcl on 2010-01-17 02:51:01
déjà vu 

http://www.hydrogenaudio.org/forums/index....st&p=649043 (http://www.hydrogenaudio.org/forums/index.php?showtopic=73686&view=findpost&p=649043)
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 02:52:19
I also would not call your position truly sceptic, rather the opposite. You seem to just resort to a ultrasceptic position to sustain a non-sceptic belief (possibly induced by sighted comparison).

I agree. Mark DeB, you're being "skeptical" about something, but completely credulous about something else for which there's no evidence (nor any clear physical or physiological mechanism through which it could credibly happen). Furthermore, you're inventing "maybes" to support your hypothesis. "Maybe there's something that no one can detect, but long-term will make you feel different." Well do you have any ideas as to what it might be? Put forward something clear so it can be tested. Anything that can be felt by your senses (and not invented by the brain), can be detected. Your body is full of detectors!

As for the long-term feeling stuff, it can also be ABX'ed as others already said. Just takes more time, but the procedure is the same. You just listen blindly for a long time, and if you believe it's the touchy-feely version, mark it on your chart. Rinse and repeat. What's the problem there?


How is it credulous to resist making an inference that is not, as far as anyone has established here, warranted?

About long-term, my question is the same: if you can't tell a difference between the two signals, does it follow that they are not causing you to have different experiences (i.e., that you are not perceiving different things in them)?
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-17 02:53:27
Well, thank you for the vote of confidence, but what exactly is it that you take me to believe that you disagree with, or think is obviously incorrect?  I am saying that for all I know, those stimuli could have different causal properties, of a certain kind, that I care about.  Are you saying that the quick-switch ABX results furnish enough information to rule that out?  On what basis?  Yes, of course this is a question of science, or induction.  But you seem to be saying that you believe this because it is "reasonable," or "common sense," or on the basis of introspection perhaps, or "obviously true," none of which is scientific.


I just noticed this. To be clear: by "common sense", I do not mean the common sense of the woman in the street, but the set of beliefs that one who has looked at the evidence is forced to adopt (opening myself up to the question "what evidence etc"  ). For example, I am a theoretical physicist, and have spent most of my life studying the results of experiments in regimes where quantum mechanics is important. Well, my predecessors have been doing that for a long time, and trying to use their then-current "common sense", ie classical physics, didn't work. So they adapted it and now "common sense" tells me that QM will be needed in such and such a situation. Another way to put it is that either those results make no sense (can't be fitted into some sort of coherent whole), or QM, aspects of which are weird from an everyday point of view, is true. Thus, "common sense"... Same for "reasonable"

Basically, I am extrapolating known results in the "simplest" way, in the absence of evidence to the contrary, and calling this process "common sense" or "reasonable".

Probably clear as mud, but... And no I'm not claiming that the subject under discussion is as clear cut as what I describe above.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-17 03:01:49
About long-term, my question is the same: if you can't tell a difference between the two signals, does it follow that they are not causing you to have different experiences (i.e., that you are not perceiving different things in them)?


Your term of "can't tell a difference" is limited to direct, short-term comparison. My term of "no perceivable difference" is two sets of predicates attributed by S at/after the time of exposure to A and B, P(A) P(B) do not correlate to the actual succession of A and B. In that sense the question is decidable in regard to any chosen level of confidence for each subject.
Title: What's the problem with double-blind testing?
Post by: MichaelW on 2010-01-17 03:09:54
About long-term, my question is the same: if you can't tell a difference between the two signals, does it follow that they are not causing you to have different experiences (i.e., that you are not perceiving different things in them)?


If a subject is having "different experiences" then they should be able to "tell a difference." It might be hard; it might not be something that all subjects can do; the nature of the difference might not be verbalizable. It might be that some subjects, poorly presented with a DBT, might falsely gain the impression that they were not allowed to declare a difference if they could not locate and describe the nature of that difference; but that would be a poorly conducted DBT.

For any reasonable meaning of "experience," different experiences are detectable. There doubtless are processes in my body which vary, and which I will never be conscious of--for instance, moderate fluctuations in the flora in my gut. But that variation is not an experience, because it has no effect on my feeling of the world (large variations in flora, of course, could well be experiential). Other, normally unconscious processes, can be brought into consciousness; the state of one's peripheral circulation, for instance, is an experience because one can tell a difference ("Is it cold in here, or is it my circulation?"), even when one cannot name it. But remote unnameable possibilities are just not worth bothering about.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 03:21:04
Well, thank you for the vote of confidence, but what exactly is it that you take me to believe that you disagree with, or think is obviously incorrect?


I think that your central point is that results from ABX-type tests with short pieces of music do not necessarily hold for long-term listening. In terms of logic this is true, and also if I just look at the "experiments" themselves, they literally say nothing about long-term listening, unless I make some assumptions about the "listening mechanism" in our heads. Apparently, given your assumptions about this mechanism, they do not, but according to mine they do.

Quote
I am saying that for all I know, those stimuli could have different causal properties, of a certain kind, that I care about.  Are you saying that the quick-switch ABX results furnish enough information to rule that out?  On what basis?  Yes, of course this is a question of science, or induction.  But you seem to be saying that you believe this because it is "reasonable," or "common sense," or on the basis of introspection perhaps, or "obviously true," none of which is scientific.

I think that you have put your finger on it when you say that it would have to be proved that "no mechanism exists."  What is more, there would have to be some theory of experiences and what it is for them to "reveal features," and of the causal conditions for such experiences.


Right, like if I claim that the fact that the sun has risen from the east for millenia (let's just assume this) does not prove that it will tomorrow. It really doesn't. Coupled with the usual models of what is actually happening, though, and given that these models are in accord with other pieces of information (falling rocks, observed motion of other lights in the sky under the assumption they're planets etc), it's hard to not conclude that it will.

Here, neither side has the equivalent model (yes a lot is known but nothing as simple, general, accurate and reliable), so it's less certain.

Quote
I still think that, knowing what I know about how I did on the ABX test, it's an open question whether those signals could cause different types of experiences in me, perhaps in other settings such as longer-term listening.  The general tenor of the replies seems to have been that it is not an open question and that I should conclude that those signals could not do so.  But I think it would be rash to conclude that.  And no one here has stated any real scientific basis why that conclusion should follow.  Why, then, do you think I am "obviously incorrect" in not making said inference?


Technically it is. I, however, am not aware of a single piece of evidence to support your thesis, nor any mechanism to suggest so. Obviously (again  ), this does not disprove it, but...

The "obviously" should probably have been quoted, as I meant it as shorthand for "in light of my mental model of the world, it would be really strange if it was not true that".

I don't think this kind of belief has the same status as believing the earth is flat, but I still would be quite surprised if it was true (as I said, lack of evidence, no personal experience indicating it etc).


The difference with the "earth is flat" hypothesis is that no one here has pointed to a theory of experiences, including criteria for when they are the same or different, and of their causal conditions, that would support said inference.  That is what is actually needed to meet the skeptical worry.  The skeptic is saying, "For all I know, there could be differences to perception, whatever that would mean, despite the negative ABX."  What is needed is a scientific basis on which the notions of experience and perception get sufficiently explicated and laws stated from which it follows that there really will be no difference to experience.

But the skeptical worry is a perfectly reasonable stage of the inquiry.  Without a theory, we don't know that there isn't such a difference.  If we don't know, it is right to point out that we don't know; or at least to ask on what basis we know.  But to answer this in terms of what is "strange," or "common sense," is just arm waving. It's not science.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-17 03:26:48
...nor is the insistence that there are unidentifiable factors at play that will create an outcome that has never been reported science.


Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-17 03:32:24


I don't think this kind of belief has the same status as believing the earth is flat, but I still would be quite surprised if it was true (as I said, lack of evidence, no personal experience indicating it etc).


The difference with the "earth is flat" hypothesis is that no one here has pointed to a theory of experiences, including criteria for when they are the same or different, and of their causal conditions, that would support said inference.  That is what is actually needed to meet the skeptical worry.  The skeptic is saying, "For all I know, there could be differences to perception, whatever that would mean, despite the negative ABX."  What is needed is a scientific basis on which the notions of experience and perception get sufficiently explicated and laws stated from which it follows that there really will be no difference to experience.

But the skeptical worry is a perfectly reasonable stage of the inquiry.  Without a theory, we don't know that there isn't such a difference.  If we don't know, it is right to point out that we don't know; or at least to ask on what basis we know.  But to answer this in terms of what is "strange," or "common sense," is just arm waving. It's not science.


I think that you have either misunderstood what I wrote or are ignoring it.

And I think this is going in circles.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 03:39:30
About long-term, my question is the same: if you can't tell a difference between the two signals, does it follow that they are not causing you to have different experiences (i.e., that you are not perceiving different things in them)?


Your term of "can't tell a difference" is limited to direct, short-term comparison. My term of "no perceivable difference" is two sets of predicates attributed by S at/after the time of exposure to A and B, P(A) P(B) do not correlate to the actual succession of A and B. In that sense the question is decidable in regard to any chosen level of confidence for each subject.


Sorry, that's a bit cryptic: could you please explain? 

Suppose we are talking about comparison of two similar, but not necessarily identical, one-minute, complicated, atonal passages of music.  You may not be able to say reliably whether they were identical or not.  But surely, if they were not identical, you perceived different pitches and rhythms at corresponding times in the samples: a difference in what is perceived without a perceived difference.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 03:45:03
...nor is the insistence that there are unidentifiable factors at play that will create an outcome that has never been reported science.



Your careful reading continues apace.  Nowhere have I insisted that there are such factors.  The question is, what rules them out in the present instance?  That would be nice to know.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-17 03:49:55
Fine, replace "insistence that there are" with "suggestion that there may be".  This does not change the irony of your post or your position.
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-17 03:49:55
...nor is the insistence that there are unidentifiable factors at play that will create an outcome that has never been reported science.



Your careful reading continues apace.  Nowhere have I insisted that there are such factors.  The question is, what rules them out in the present instance?  That would be nice to know.


Same thing that rules out a giant pink bunny orbiting Uranus?  (joking here, ie, not claiming it's as unlikely)
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-17 04:00:01
But the skeptical worry is a perfectly reasonable stage of the inquiry.  Without a theory, we don't know that there isn't such a difference.  If we don't know, it is right to point out that we don't know; or at least to ask on what basis we know.  But to answer this in terms of what is "strange," or "common sense," is just arm waving. It's not science.

Without a theory, we don't know that there isn't an invisible, undetectable dragon living in my garage. Do you agree that we "don't know" that there's an invisible dragon living in my garage?

There is no reason whatsoever to assume that something outside of one's mind that can't be detected is influencing anyone, if there's no mechanism for it to affect him/her. Furthermore, the "phenomenom" is already easily explainable by far better known theories (placebo, etc.).
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 04:04:50
About long-term, my question is the same: if you can't tell a difference between the two signals, does it follow that they are not causing you to have different experiences (i.e., that you are not perceiving different things in them)?


If a subject is having "different experiences" then they should be able to "tell a difference." It might be hard; it might not be something that all subjects can do; the nature of the difference might not be verbalizable. It might be that some subjects, poorly presented with a DBT, might falsely gain the impression that they were not allowed to declare a difference if they could not locate and describe the nature of that difference; but that would be a poorly conducted DBT.

For any reasonable meaning of "experience," different experiences are detectable. There doubtless are processes in my body which vary, and which I will never be conscious of--for instance, moderate fluctuations in the flora in my gut. But that variation is not an experience, because it has no effect on my feeling of the world (large variations in flora, of course, could well be experiential). Other, normally unconscious processes, can be brought into consciousness; the state of one's peripheral circulation, for instance, is an experience because one can tell a difference ("Is it cold in here, or is it my circulation?"), even when one cannot name it. But remote unnameable possibilities are just not worth bothering about.


RE: "If a subject is having 'different experiences' then they should be able to 'tell a difference.'  It might be hard; it might not be something that all subjects can do..." 

Not sure what you mean by "should be able."  Maybe the skeptic is one of those people who can't do it.

RE: "For any reasonable meaning of "experience," different experiences are detectable."

That is just a dogmatic claim, armchair neuroscience. 

In any event, the issue is not whether, given certain perceptual facts, some kind of DBT test could be devised to determine whether they obtain.  The issue is what should be concluded from failure on the kind of test that I took, or undertook, yesterday.

Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 04:15:48
Fine, replace "insistence that there are" with "suggestion that there may be".  This does not change the irony of your post or your position.


Right, that is an "it may be" in the sense of "for all I know, it is," or an epistemic sense of "may," which is crucial to science because it needs the notion of what a theory does or does not predict.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 04:51:12
But the skeptical worry is a perfectly reasonable stage of the inquiry.  Without a theory, we don't know that there isn't such a difference.  If we don't know, it is right to point out that we don't know; or at least to ask on what basis we know.  But to answer this in terms of what is "strange," or "common sense," is just arm waving. It's not science.

Without a theory, we don't know that there isn't an invisible, undetectable dragon living in my garage. Do you agree that we "don't know" that there's an invisible dragon living in my garage?

There is no reason whatsoever to assume that something outside of one's mind that can't be detected is influencing anyone, if there's no mechanism for it to affect him/her. Furthermore, the "phenomenom" is already easily explainable by far better known theories (placebo, etc.).


Of course we don't know that there is a dragon in your garage if in fact there is no dragon there, but I think that you mean to be asking whether on my view we don't know that there isn't one, and the answer is that (obviously) there is excellent reason to think that there is no dragon in your garage.

What you refer to as "placebo" might explain why someone hears differences, but it is irrelevant to the present case, in which no differences are perceived.

In any event, I am not assuming "that something outside of one's mind that can't be detected is influencing anyone," whatever that means.  I mean, what counts as being inside or outside the mind?  What is the mind, anyway?  I am not taking any particular stand on this or making any such assumption as the one you attribute to me.  I am supposing merely that we have experiences, or perceive things as being a certain way, and that experiences can be the same or different, or that the way we perceive something as being on a certain occasion can be the same or different from the way we perceive it as being on another occasion; and I am saying I see no reason why I should conclude from the sort of ABX test I took yesterday that the relevant signals do not, in normal listening, cause different experiences in me, or cause me to perceive things in different ways, where the difference is one that matters.

It seems to me (though I could be wrong) that the sticking point is that you and others reject the notion that there could be a difference in experiences without a difference that is experienced; a difference in perceptions without a perceived difference.  Am I right (that you reject this notion, and that this is the sticking point)?
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 05:56:30
Settle down, chil'ren.

Mark, you are correct that under any strictly logical analysis, a negative ABX result cannot "debunk" the existence or non-existence of a difference. Nonexistence is nonfalsifiable. However, it can disprove the magnitude of such a difference, if it exists, and often times that's enough to settle a debate. If one says "OMG this is a night and day difference, I can detect the improvement in half a second", but then undergoes an ABX test under identical listening conditions and fails it miserably, the most plausible reason for the failure is not that there's some fundamental issue with ABX testing that fundamentally compromises its meaning (although that excuse is profferred all the time). The most plausible reason is that the originally perceived difference was 100% imaginary, and among many reasons why this is the most plausible, the one I prefer is that I observe that behavior in myself, regularly, in comparisons which plainly null out, and based on my experience and what I know of human psychology, I expect that behavior in everybody else.

However, when blind tests consistently fail, and there are good theoretical reasons to doubt that such tested differences would ever be audible, at some point, you've gotta step back and call a turd a turd. A while ago there was a big controlled study showing that homeopathy did not demonstrate any efficacy, and while that was used by doctors to "prove" that homeopathy was a worthless fraud, it clearly could not justify such a conclusion - and it didn't, at least not on its own. But combined with the knowledge that homeopathy treatments are diluted to the point of not containing a single molecule of the original substance, and the knowledge that no remotely scientifically plausible theory exists that would in any way justify its efficacy, even after over 100 years of study, you'd need some pretty thick blinders (or big cahones) to assert anything different than "homeopathy is dead".

In other words... bone up on your Kuhn. Paradigms have never been overturned because of evidence.

Regarding quick-switch, I might offer (anybody correct me if I'm wrong here): a) No evidence exists that long-time switching is any more sensitive than quick switching; b) Considerable anecdotal evidence (and I want to say at least one study) indicates quick switching being more sensitive than long time switching; c) The seconds-long duration of auditory memory provides an extremely powerful theoretical justification for the increased sensitivity of fast switching. There is simply no evidence supporting such a theoretical objection, some (perhaps a significant) amoung of evidence against the idea, and a very good theoretical reason against the idea.


The point at which you "call a turd a turd" is one where there is strong evidence for a particular hypothesis, such as that the subject cannot detect a difference, and we finally give up entertaining its negation.  But what I am asking about is different.  In this case, there is an inference from the inability to detect a difference, in a test situation, to the conclusion that there is no difference between the signals that makes a difference to perception, or a difference in the character of the experiences, in normal listening.  We are projecting from one thing to another.  What are the "good theoretical reasons" that would justify this inference?
Title: What's the problem with double-blind testing?
Post by: MichaelW on 2010-01-17 06:17:59
RE: "For any reasonable meaning of "experience," different experiences are detectable."

That is just a dogmatic claim, armchair neuroscience.


No it's not. It's not science at all. It is about the meaning of words. "Experience," in this context, means, in normal usage, something which affects the consciousness of a subject, even if it can't be precisely identified. I am reliably informed that an appalling number of neutrinos whizz through my body every day, but this is not part of my "experience," though it is something which, I believe, happens to me.

If the hypothesis is that different sound signals which are audibly, and in all other ways, indistinguishable by the subject, might produce different effects on the human organism, that would be a claim in science. To call these effects part of the human being's experience would be an idiosyncratic used of the word.
Title: What's the problem with double-blind testing?
Post by: psycho on 2010-01-17 10:38:26
If the hypothesis is that different sound signals which are audibly, and in all other ways, indistinguishable by the subject, might produce different effects on the human organism, that would be a claim in science.


I hope that this in fact is Mark DeB's hypothesis. I see it this way every since he first posted on this thread.

But this should also be ABX-able, as greynol (and others) have already stated. In my somewhat funny suggestion, I was thinkig of such a prolonged ABX test. Those, who believe that Mark DeB's hypothesis is on to something, should take such a test and get done with it already. 

If one would want to do the test without ABX, here's another one of my funny suggestions. 

In a mad scientist kind of way, this hypothesis could be tested with 10 pairs of twins, which would have been separated since birth until they stopped growing. One group of twins would be exposed to lossy samles of the same sounds as the other group would be exposed to by original samples. The mad scientist could mesure all kinds of body functions and processes and whatever he would think of. If he would want, he could try them psychologically - they would debate about experiences with certain sounds and see if they feel different about them. Between the groups there should be absolutely no other differences, besides the sounds. This test is virtualy impossible to do and I am only describing it, so people who support this hypothesis would see how absurd it is to be concerned about such subtle differences. They may be so subtle that something else can have a greater impact on us and that they don't matter at all. In fact, I believe that that is the case here - if there even is a different effect on our organism by lossy sounds.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-17 14:36:53
Sorry, that's a bit cryptic: could you please explain?


What specifically do you not understand?

Suppose we are talking about comparison of two similar, but not necessarily identical, one-minute, complicated, atonal passages of music.  You may not be able to say reliably whether they were identical or not.  But surely, if they were not identical, you perceived different pitches and rhythms at corresponding times in the samples: a difference in what is perceived without a perceived difference.


Here we come to the center of your problem. You accuse others here of inferring to much, while making the biggest unfunded jump yourself, i. e. you are inferring from the outside knowledge of factual differences to the existence of different subjective perceptions. Two super-complex, atonal passages with very little difference might be perceived as the same 'bringbrangpling' by an untrained subject with neither different perceptions nor causal differences. In contrast for someone familiar with atonal music there might be different perceptions giving one track at least another (partial) gestalt than the other, but he will be able to gain knowledge about it.

The only niche that remains for you would be:

1. Actually different tracks
2. Neither perceived difference nor different perceptions
3. Still causal differences (e.g. emotionally) between A and B in relation to S

The existence of the latter can be verified by long term, double-blind, correlated A/B testing. It's quite a task, it's very improbable that there is actually anything to find, but feel free to go that route. Findings could be revolutionary. But the could-be-would-be-space of highly improbable problems doesn't bring us forward. The space of that is just too vast*. You are welcome to present any new significant data and I'm sure this community would welcome it. But the causally-effective-but-imperceivable-in-blind-comparison-class of problems without accompanying proof is better taken care of in other forums.

* Lets start with I might be an operator assigned to your brain in tank No. 284398398A.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 15:16:21
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?
(2) How would you test this hypothesis?

If psychoacoustics rules it out or says that it is unlikely, why?  What is the explanation, or "good theoretical reasons"?  And if, though our best currently available theory does not rule it out, it describes no known mechanism by which such an effect might occur, what are we entitled to conclude from that?  Is the current state of our theory so advanced that it would describe such a mechanism if one existed?

For testing, quick-switch ABX would be useless because the phenomenon occurs only with longer stimuli.  And longer-stimuli ABX tests may not be reliable, as is illustrated by the example of the atonal piece I suggested to rpp3po in a previous post.  In the present case, how a note sounds midway through the stimulus in one presentation may differ from how it sounds in the other, but the subject would lack any way of directly comparing them, and hence of detecting or reliably reporting that they differ.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 15:31:27
Sorry, that's a bit cryptic: could you please explain?


What specifically do you not understand?

Suppose we are talking about comparison of two similar, but not necessarily identical, one-minute, complicated, atonal passages of music.  You may not be able to say reliably whether they were identical or not.  But surely, if they were not identical, you perceived different pitches and rhythms at corresponding times in the samples: a difference in what is perceived without a perceived difference.


Here we come to the center of your problem. You accuse others here of inferring to much, while making the biggest unfunded jump yourself, i. e. you are inferring from the outside knowledge of factual differences to the existence of different subjective perceptions. Two super-complex, atonal passages with very little difference might be perceived as the same 'bringbrangpling' by an untrained subject with neither different perceptions nor causal differences. In contrast for someone familiar with atonal music there might be different perceptions giving one track at least another (partial) gestalt than the other, but he will be able to gain knowledge about it.

The only niche that remains for you would be:

1. Actually different tracks
2. Neither perceived difference nor different perceptions
3. Still causal differences (e.g. emotionally) between A and B in relation to S

The existence of the latter can be verified by long term, double-blind, correlated A/B testing. It's quite a task, it's very improbable that there is actually anything to find, but feel free to go that route. Findings could be revolutionary. But the could-be-would-be-space of highly improbable problems doesn't bring us forward. The space of that is just too vast*. You are welcome to present any new significant data and I'm sure this community would welcome it. But the causally-effective-but-imperceivable-in-blind-comparison-class of problems without accompanying proof is better taken care of in other forums.

* Lets start with I might be an operator assigned to your brain in tank No. 284398398A.


No, if we are talking about a piece by Webern, say, and the sort of listener who enjoys listening to music, attends concerts, etc., though she may not know much music theory, the best available theories in music cognition tell us that she is going to have conscious mental representations of specific pitches as they go by.  She will not (only) have a perception of "bringbrangpling."  But if these are one-minute presentations, she is not going to remember the specific pitch in one presentation long enough to compare it with its counterpart in the other.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-17 16:05:25
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.

(2) How would you test this hypothesis?


Long term A/B testing. Recording when fatiguing effect has been noticed. After sufficient number of trials, correlate records with actual succession of As and Bs.

For the 100th time...
Title: What's the problem with double-blind testing?
Post by: lvqcl on 2010-01-17 16:08:01
But Mark DeB said that this fatiguing effect cannot be reliably identified...
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 16:15:02
RE: "For any reasonable meaning of "experience," different experiences are detectable."

That is just a dogmatic claim, armchair neuroscience.


No it's not. It's not science at all. It is about the meaning of words. "Experience," in this context, means, in normal usage, something which affects the consciousness of a subject, even if it can't be precisely identified. I am reliably informed that an appalling number of neutrinos whizz through my body every day, but this is not part of my "experience," though it is something which, I believe, happens to me.

If the hypothesis is that different sound signals which are audibly, and in all other ways, indistinguishable by the subject, might produce different effects on the human organism, that would be a claim in science. To call these effects part of the human being's experience would be an idiosyncratic used of the word.


I wouldn't have thought that "different experiences are detectable" is true in virtue of meaning, on the normal meaning of "experience," but be that as it may: do you mean detectable by the subject who has the experiences?  Always?  Consider the example of the atonal piece (described in another post) in which a small change is introduced.  In one presentation the subject hears one pitch; in another he hears a different pitch.  The experiences are different, but the subject may not be able to detect that they are.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-17 16:18:28
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.

(2) How would you test this hypothesis?


Long term A/B testing. Recording when fatiguing effect has been noticed. After sufficient number of trials, correlate records with actual succession of As and Bs.

For the 100th time...


But what "noticing" the effect would mean here is that the subject compares corresponding parts of the stimulus and sees that the way one sounds is different from the way the other sounds, which is hard to do reliably if they are separated in time.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-17 16:21:06
No, if we are talking about a piece by Webern, say, and the sort of listener who enjoys listening to music, attends concerts, etc., though she may not know much music theory, the best available theories in music cognition tell us that she is going to have conscious mental representations of specific pitches as they go by.
She will not (only) have a perception of "bringbrangpling."  But if these are one-minute presentations, she is not going to remember the specific pitch in one presentation long enough to compare it with its counterpart in the other.


It may seem obvious at the first look, but it isn't. Please show how you infer from different pitch at time x to different mental representation at time x. As soon as modify the test-setup to give that answer it will also be able to give the "perception of a difference" answer. This is, because your long-term effect is only a pseudo long-term effect. Your analogy just pads an easily discernible difference (two different pitches at time x) with a long timespan during which it is forgotten again. The crux is the easily discernible difference at time x. This is a flawed analogy to the hypothesis you were originally trying to make: an at no time by direct comparison detectable difference that still exists with causal effect.

PS

But what "noticing" the effect would mean here is that the subject compares corresponding parts of the stimulus and sees that the way one sounds is different from the way the other sounds, which is hard to do reliably if they are separated in time.


No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.

PPS Your excessive block citing style is not appreciated.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 05:26:10
But what "noticing" the effect would mean here is that the subject compares corresponding parts of the stimulus and sees that the way one sounds is different from the way the other sounds, which is hard to do reliably if they are separated in time.


No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.



Why should we assume that the subject can reliably report when the effect (not necessarily a feeling of fatigue) is occurring?  If she cannot, then a test such as you describe won't tell us much.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 13:14:25
Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.

(2) How would you test this hypothesis?


Long term A/B testing. Recording when fatiguing effect has been noticed. After sufficient number of trials, correlate records with actual succession of As and Bs.

For the 100th time...



No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.


Moreover, how are you going to instruct the subject as to what effect, what property of her experience, she is supposed to report?  You are assuming that there is an expression in a public language, shared by subject and experimenter, that designates this property of experience and that the experimenter knows what expression to use.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-18 14:41:56
We are turning in circles: No detectable trace, no relevant causal effect. You are like Don Quijote, endlessly defending the possible existence of a priori undetectable causal effects. Look at it from a computability perspective: The set of possibly detectable causal effects has already greater cardinality than infinity. You can't work with that. Science is essentially reducing this set to the most reasonable subsets. So your whole approach is fundamentally unscientific, because you try to extent the global set to even a priori undetectable (by your own definition) effects for no practical (or even theoretical) benefit. While you use modern terminology to describe your position, it is the same class as the clerical defense of the geocentric model. Neglecting the huge benefits of scientific discoveries by accepting the sheer fact of their possible incompleteness as an argument.

Now you even retreat to that (e. g.) upsampling may raise a feeling that there is no word for. OK, I think we finally can come to an end then. The position you are defending is so exclusive that

1. you yourself can't detect its existence, because its effects are unobservable
2. there isn't even a word for it

What's the worth of such a position?? What's the sense of defending it?

I make the following alternative claim:

Upsampling should be avoided because it causes 'gooness'. Gooness is harmful. Sadly I cannot say more because 1. and 2. (in the sense that we don't share the word in a common sense) apply.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 16:08:10
We are turning in circles: No detectable trace, no relevant causal effect. You are like Don Quijote, endlessly defending the possible existence of a priori undetectable causal effects. Look at it from a computability perspective: The set of possibly detectable causal effects has already greater cardinality than infinity. You can't work with that. Science is essentially reducing this set to the most reasonable subsets. So your whole approach is fundamentally unscientific, because you try to extent the global set to even a priori undetectable (by your own definition) effects for no practical (or even theoretical) benefit. While you use modern terminology to describe your position, it is the same class as the clerical defense of the geocentric model. Neglecting the huge benefits of scientific discoveries by accepting the sheer fact of their possible incompleteness as an argument.

Now you even retreat to that (e. g.) upsampling may raise a feeling that there is no word for. OK, I think we finally can come to an end then. The position you are defending is so exclusive that

1. you yourself can't detect its existence, because its effects are unobservable
2. there isn't even a word for it

What's the worth of such a position?? What's the sense of defending it?

I make the following alternative claim:

Upsampling should be avoided because it causes 'gooness'. Gooness is harmful. Sadly I cannot say more because 1. and 2. (in the sense that we don't share the word in a common sense) apply.


Who says it is undetectable?  Maybe there are some means by which it is detectable.  The question was how to detect or test for it, the kind of test you suggested won't work, and now you say anything undetectable by those means is not worth talking about.

It is not a "retreat" to say that there are feelings there are no words for, since I never claimed that there weren't such feelings to begin with.  Obviously, there are lots of such feelings.  Why shouldn't the listener care about them?

And even if the subject does have a word for the feeling, the experimenter has to know what feeling to ask about, in your suggested test.  It may not suffice to ask, "Are you having a feeling of fatigue now?" since the subject may not associate the effect with that expression.  How is the experimenter going to know what expression to use?
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-18 16:29:02
This is really quite pathetic.

If the person who is trying to determine whether X sounds different from Y cannot identify a "feeling" or whatever you want to call it that allows him to distinguish a difference then there is no difference to that person at that point in time.  If the person needs help from someone else who can, that's basically an admission that he cannot distinguish the two prior to getting the help, assuming the help even makes a difference.

Again, Mark, you have absolutely nothing.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-18 16:42:46
This is getting really tiring. Please let us know which effect you have observed and documented, that might be interesting to others here, and how it is reproducible. Anything else won't find much interest. Feelings should be cared about, yes, as long as they are occurrent. Completely under the radar, out of the scope of any possible documentation, they don't matter here. There is not the slightest indication or finding, that your claim is in any way relevant. If you want to change that, show some data. The ontological status of the "could" you have presented belongs to the same class as the question wether aliens live among us without us knowing. I'm not interested to follow that any further.

Bye
Title: What's the problem with double-blind testing?
Post by: Qest on 2010-01-18 19:40:13
(this thread is getting painful to lurk)

I agree that if you claim something exists then the onus is on you to prove it exists. Until that's done there is really nothing more to talk about here.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 19:55:11
This is really quite pathetic.

If the person who is trying to determine whether X sounds different from Y cannot identify a "feeling" or whatever you want to call it that allows him to distinguish a difference then there is no difference to that person at that point in time.  If the person needs help from someone else who can, that's basically an admission that he cannot distinguish the two prior to getting the help, assuming the help even makes a difference.

Again, Mark, you have absolutely nothing.


You are assuming that the experimenter is the same person as the subject.  Go back and look at the distinction between (1) and (2) in my post #125:

(1) You perceive a as X and you perceive b as Y, and X is different from Y.
(2) You perceive that a is different from b.

You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.

Would you mind being civil, though?  Thank you.
Title: What's the problem with double-blind testing?
Post by: Nick.C on 2010-01-18 20:08:44
You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.
Which other ways would there be (unless the subject is told)?
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-18 20:25:50
Would you mind being civil, though?

Calling you pathetic would have been uncivil.  Calling your failed attempt at successfully defending your point of view pathetic (and it is pathetic) is not.

Would you mind answering Nick.C's question?  Thank you.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 20:37:33
This is getting really tiring. Please let us know which effect you have observed and documented, that might be interesting to others here, and how it is reproducible. Anything else won't find much interest. Feelings should be cared about, yes, as long as they are occurrent. Completely under the radar, out of the scope of any possible documentation, they don't matter here. There is not the slightest indication or finding, that your claim is in any way relevant. If you want to change that, show some data. The ontological status of the "could" you have presented belongs to the same class as the question wether aliens live among us without us knowing. I'm not interested to follow that any further.

Bye


Fine.  But then it remains an open question whether the signals differ in properties relevant to perception, since you haven't suggested any way to resolve it.  Your request for documentation of an observed effect misses the point.  I never claimed to have observed any such effect (and, if you have been following things, you know I don't assume that the effects in question can be reliably reported).  The question is whether there are any effects of the kind described in the hypothesis, and whether tests such as you describe would be relevant to finding this out.  Evidently, they would not be.

Concerning the "ontological status" of this versus aliens, not so.  This is a question in psychology and cognition.

It may be that a hypothesis that posits things that are undetectable in principle would not be worth considering, but I am not suggesting such a hypothesis.  There might well be ways of detecting whatever properties and effects might make the hypothesis true, although those ways would probably vary with the properties/effects.  So there is no reason why some fixed method of determination should be hard wired into the hypothesis.  However, it is not hard to imagine an alternative to ABX listening tests of the kind you suggest: brain scans.  We might well have excellent theoretical reasons to think that, when a part of the brain lights up in a certain way in response to one stimulus but not another, that this corresponds to a difference in the experiences.

The fact that the hypothesis cannot be falsified using your methodology does not mean that it is unfalsifiable tout court, or cognitively meaningless.

Look, here is another way of putting the point: Why should I bother with lossy formats for listening when I can use FLAC?  I don't want to lose any relevant information.  What is the criterion for relevance?  Many people seem to assume that conscious discrimination (in certain conditions) is an adequate criterion, but I see no real argument for this.  It's just a dogma.

Thank you for the dialogue.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 20:41:50
(this thread is getting painful to lurk)

I agree that if you claim something exists then the onus is on you to prove it exists. Until that's done there is really nothing more to talk about here.


Since I am not claiming that something exists, but asking how it can be proved that it doesn't, your claim about a burden of proof is based on a false assumption.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 20:47:16
Would you mind being civil, though?

Calling you pathetic would have been uncivil.  Calling your failed attempt at successfully defending your point of view pathetic (and it is pathetic) is not.

Would you mind answering Nick.C's question?  Thank you.


They are both uncivil.  A civil discussion of some topic means sticking to the subject and giving reasons and argument, not calling names, even applied to someone's view.  And that's what you did, and continued to do in the post to which this is a response.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 20:48:48
You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.
Which other ways would there be (unless the subject is told)?


Please see post #195 for a suggestion.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 21:22:50
Oh, one other thing.  The suggestion that there can be effects of perceptual information that do not translate into conscious discrimination is not exactly an idea from Mars, but is well known in the literature on cognition.  Blindsight is a well-known example.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-18 21:36:29
We might well have excellent theoretical reasons to think that, when a part of the brain lights up in a certain way in response to one stimulus but not another, that this corresponds to a difference in the experiences.


On a side note, I suspected that you are coming from that school. Nowadays it is a quite popular position among current neurologist (maybe sometimes without a certain amount of philosophical background) and popular science. Even many proponents of the contemporary philosophy of mind follow that route. Trying to solve own philosophical inadequacies with tech seems much sexier than challenging the claims of an in-fashion discipline as neurology. I profoundly believe that the quoted claim can be refuted on phenomenological grounds alone. But that would be another discussion, at least 10x as long as the latter and not belonging here.
Title: What's the problem with double-blind testing?
Post by: ExUser on 2010-01-18 21:48:40
The question double-blind testing usually answers around here is: "Can people hear a difference?" You're demanding a pretty broad definition of "hear". I'd venture to say that your definition is almost meaninglessly broad, especially if you're extending it to mean "shows different brain scan results". Just because there is a neurological response to a physical stimulus does not imply that someone is necessarily consciously aware of the stimulus nor does it imply that a person "heard" anything.

Conscious awareness of a thing is chosen simply because it's most easily testable. There is no reason why blind-testing need be limited to conscious awareness, it's just a whole lot harder to test based on brain-scans. Anything short of conscious awareness testing cannot answer the question "Can people hear a difference?" in any meaningful way. When you start to alter the definition of "hear" beyond the intuitive awareness of what it means that all non-deaf people share, all the assumptions and axioms we start with go right out the window.

As for your FLAC analogy: I can ABX 320kbps MP3 vs. FLAC on some samples. Yet I use V4 on my portable. I can hear a difference if I try and the conditions are right. That's missing the point though. The point is that I can fit more music on my player at V4, and that I won't notice if I'm not consciously focusing on trying to differentiate the two. I can sure "hear" the difference, but unless I'm really bothered, I don't hear the difference.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 22:05:29
You are assuming that the only way to tell that X is different from Y, in the context of (1), is by the subject's own conscious discrimination. There is no necessity to make that assumption.
Which other ways would there be (unless the subject is told)?


Please see post #195 for a suggestion.


And also #199.  There are ways we tell that blindsight exists, but it is (almost by definition) not via straightforward conscious discrimination of stimuli as in the conventional ABX tests under discussion.
Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-01-18 22:23:53
This is getting really tiring.
Actually that might well be the biggest problem with double-blind testing
Let me try to help Mark a bit with an example. I have a colleague recording/mixing engineer who is convinced that 24/96 sounds better than 24/44.1 (in "his" setup). One of his motivations is that he is less tired after a day of mixing at 96k. Needless to say that I was curious to investigate further, but since his mix setups are usually rather complex, it is not at all easy to switch sampling rates quickly so an ABX test seems not possible. Besides that it's probably very difficult to separate auditory and non-auditory variables in the test (e.g. hours of sleep, disturbing phonecalls during work, duration of work, monitoring levels etc.).
What would be the best options for double-blind testing when A and B are significantly separated in time and/or likely to be different ?
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-18 22:29:07
He might have made an adequate observation. At 24/96 the computer spends more time for processing, so he has slightly longer pauses between interactions with his machine.  Also filtering is much less mentally challenging without Nyquist at your doorstep.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-18 22:30:58
As was said repeatedly(!), ABX has no time limit.  Since your friend's experience is not double-blind, you cannot rule out placebo as a legitimate reason he believes he hears differences, can you?
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-18 22:48:49
Mark do you hang a lot in the mental masturbation section of the philosophy forums? 

Clearly the burden of proof comment above is equally valid for people who say something exists, as for people to say that something might exist (i.e. it's plausible). You're being very careful (yeah, we're onto you) not to go all the way and say it exists, but you should provide a reason as to why you should think it might exist. So far I don't think you have.

About the brain scan thing. That's another blind test for you. See if the subject can differentiate when different parts of his brain light up. You're saying that he probably could. That's a more interesting test in my opinion, anyway.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-18 22:51:31
The question double-blind testing usually answers around here is: "Can people hear a difference?" You're demanding a pretty broad definition of "hear". I'd venture to say that your definition is almost meaninglessly broad, especially if you're extending it to mean "shows different brain scan results". Just because there is a neurological response to a physical stimulus does not imply that someone is necessarily consciously aware of the stimulus nor does it imply that a person "heard" anything.

Conscious awareness of a thing is chosen simply because it's most easily testable. There is no reason why blind-testing need be limited to conscious awareness, it's just a whole lot harder to test based on brain-scans. Anything short of conscious awareness testing cannot answer the question "Can people hear a difference?" in any meaningful way. When you start to alter the definition of "hear" beyond the intuitive awareness of what it means that all non-deaf people share, all the assumptions and axioms we start with go right out the window.

As for your FLAC analogy: I can ABX 320kbps MP3 vs. FLAC on some samples. Yet I use V4 on my portable. I can hear a difference if I try and the conditions are right. That's missing the point though. The point is that I can fit more music on my player at V4, and that I won't notice if I'm not consciously focusing on trying to differentiate the two. I can sure "hear" the difference, but unless I'm really bothered, I don't hear the difference.


Thanks for your response.  The thing is, I don't think I have to have a definition of "hear" in order to have a (not pointless) worry about whether compression (e.g.) makes some difference to what I hear.  I think that hearing is whatever, at the "end of science," cognitive theories would eventually say that it is.  And I don't know that compression fails to make a difference to that.  (Since I am the one doing the hearing, even though I don't have a scientific definition of it, I can still worry about it.)  Moreover, I have an open-ended attitude about what sorts of things, what experiences or cognitive effects, I ought to care about.  Suppose having certain sorts of information causes certain differences in cognitive processing or representation in me, at some level, even though I can't discriminate the stimuli.  When met with the claim, "You should not care about the difference, because you can't discriminate the two," I just see no necessity in that.  How do I know in advance that it would be silly or pointless for me to have an interest in having one sort of cognitive process go on in me, rather than the other, where the difference is not manifested in conscious discrimination?  We know more and more that what is consciously available to us is just the tip of the iceberg, so I don't know that such an interest would necessarily be pointless.

What's more, there may be effects that are conscious but context dependent, as with the "fatigue" hypothesis.  This is a cognitively meaningful hypothesis (despite the naysayers) but hard to test by ABX since quick-switch gets rid of the context and, if we use rpp3po's method on longer stimuli, valid only if the subject has a word for the effect in her language and we know to use it in explaining the protocol to her.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-18 22:51:35
And as a side note, on the "civility" argument. One very often hears this from people on the other side of science, I wonder why.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-18 23:04:24
...yet he offers no brain scans, so we're left in Mark's fantasy land where we possibly live amongst aliens, invisible garaged dragons and giant pink bunnies that orbit Uranus.

Even if brain scans show differences (I doubt they will) to different stimuli that would appear to be indistinguishable in ABX testing [which he will not specify (flac vs. mp3, 16-bit vs. 24 bit?)] it's a giant leap to conclude that one is "better" for us than the other.  Just as my ability to kill someone in a distant galaxy by snapping my fingers, it could be that the mp3 created from 16 bits will help prevent some types of cancer while the 24-bit flac will not help prevent any types of cancer.

Canar, Axon and others laid the point out pretty plainly.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-18 23:07:12
valid only if the subject has a word for the effect in her language and we know to use it in explaining the protocol to her.

Complete nonsense.  Words are not necessary to choose X as being A or B and no one ever said such a choice must be limited to what one "hears".
Title: What's the problem with double-blind testing?
Post by: ExUser on 2010-01-18 23:10:01
Quote
The thing is, I don't think I have to have a definition of "hear" in order to have a (not pointless) worry about whether compression (e.g.) makes some difference to what I hear.
If you don't know what comprises "hearing" and are unwilling to define it, you cannot even say you hear anything at all. Worrying about compression making a difference to what you hear is stupid. The evidence is clear: 320kbps is insufficent. Furthermore, there is no lossy codec that exists that does not have samples for which it fails. If the lossiness of lossy compression is an issue at all, in any way shape or form, use lossless.

When moving past the pragmatic into the realm of the theoretical, anything is possible. You're focused on the anything. You've completely lost sight of the "why". Why do we blind-test? Because it allows us to show that we hear a difference. Why do we worry about hearing a difference? Because we're trying to improve psychoacoustic audio codecs.

You talk about hearing, yet you refuse to put your finger on what hearing is, instead erecting some silly argument about "end of science" definitions. What if, at the "end of science", hearing is defined simply as what we can consciously perceive? Your whole argument falls apart. I see no reason why such a simple definition is not possible or even likely.

Instead, you insist on public mental masturbation, bringing up edge cases which are not even known to exist yet, but may exist. Okay, sure, let's suppose you're right. Now what? Oh, you've proven that you like mentally masturbating in public on topics so completely detached from reality that there is no pragmatic basis on which you base your arguments.
Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-01-19 00:03:51
As was said repeatedly(!), ABX has no time limit.
I'm just not convinced(!). In my experience, when the time between two stimuli A and B becomes large (e.g. days, weeks), many non-auditory variables are introduced (e.g. memory). Example: several years ago the seats in the Concertgebouw had to be replaced. Requirement was that the famous acoustics had to stay identical. After installation of the carefully designed seats many listeners had the impression that the acoustics had changed. How would you ABX this ?
Since your friend's experience is not double-blind, you cannot rule out placebo as a legitimate reason he believes he hears differences, can you?
Would your term "placebo" have the same meaning as my "non-auditory variables" ?
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-19 00:19:59
Placebo was just going to the broader point that once you open the door to far-fetched possibilities as this discussion has taken, we should make sure to include not-so-far-fetched possibilities such as your friend is imagining the differences (assuming his equipment performs to specification/is not broken).  It's quite easy that your friend is feeling more fatigued when working with CDDA because he has a preconceived notion that it is more fatiguing.  After all the placebo effect isn't exactly controversial and it isn't as if your friend is not aware of the bit depth and sample rate he's using, now is it?

Regarding the lack of control do you think the same things will happen only on the day that A is being presented and not B?  However, just because your friend doesn't have the luxury to dedicate his days at work to a more scientifically controlled test, doesn't mean that it can't be done.  Until it can be done, his suggestion that he can tell the difference based on fatigue holds little (if any) scientific merit if we're only supposed to go by his word.

Your room example is not exactly valid.  There is no defined A and defined B which we can go back and forth between like there is with hi-res vs. CDDA, so ABX testing clearly does not apply.  Though I suppose you could swap all the seats out but then you need to make sure the testing is still blind.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-19 00:35:21
Mark do you hang a lot in the mental masturbation section of the philosophy forums? 

Clearly the burden of proof comment above is equally valid for people who say something exists, as for people to say that something might exist (i.e. it's plausible). You're being very careful (yeah, we're onto you) not to go all the way and say it exists, but you should provide a reason as to why you should think it might exist. So far I don't think you have.


For a start, see http://www.arts.uwaterloo.ca/~pmerikle/pap...tudies.1998.pdf (http://www.arts.uwaterloo.ca/~pmerikle/papers/Merikle.JConsStudies.1998.pdf) .

If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

About the brain scan thing. That's another blind test for you. See if the subject can differentiate when different parts of his brain light up. You're saying that he probably could.


No, actually, that's the opposite of what I'm saying, because if he could do that then he probably can differentiate the stimuli to begin with and there would be no reason to do the brain scans in the first place.

That's a more interesting test in my opinion, anyway.

Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-19 01:06:42
Quote
The thing is, I don't think I have to have a definition of "hear" in order to have a (not pointless) worry about whether compression (e.g.) makes some difference to what I hear.
If you don't know what comprises "hearing" and are unwilling to define it, you cannot even say you hear anything at all. Worrying about compression making a difference to what you hear is stupid.


Please don't take this the wrong way, but that's a naive view of meaning.  The notion that in order to say things we have to have definitions in our heads, or a capacity to state necessary and sufficient conditions, was effectively demolished by Wittgenstein.  Such a capacity may be necessary for certain purposes, perhaps in scientific theory building, but it is not a general requirement for using language meaningfully.  So when you say that the worry or skeptical doubt I have expressed is "stupid," you have a false picture of meaning.  Hearing, in the sense of conscious auditory experience, is a natural phenomenon that we can refer to; science will help us discover what it is; but to insist that, unless we have an operational definition such as "shows different brain scan results" in mind (as some logical positivists might have insisted), we are not saying anything at all, is a mistake.


Title: What's the problem with double-blind testing?
Post by: ExUser on 2010-01-19 01:22:03
Entering into serious discourse without defining what it is you're talking about is asinine. Worrying about compression is what I was labelling as "stupid". Worrying about compression (which is a very small thing in the bigger picture) is not something I'd recommend to anyone. I suggest you change your tone if you wish to continue on in these forums. Language can do all kinds of internally-contradictory nonsense. Just because a construct can exist within the bounds of language does not imply that a construct can exist within the bounds of logic or science. Just because I can talk about walking on water personally does not mean that I can walk on water.

You're still missing my main point: What's the point of any of your argument? That there are phenomena that cannot be tested? That tests are specific to the test being taken? It doesn't invalidate a blind test. If we assume that there are untestable subconscious phenomena behind lossy compression, the only difference in outcome is that we have a valid reason to switch to lossless. However, there are already valid reasons to use lossless. If we assume there are untestable subconscious phenomena behind lossy compression, then blind-tests designed to test conscious phenomena are still valid in the context of conscious phenomena.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-19 07:38:19
Clearly the burden of proof comment above is equally valid for people who say something exists, as for people to say that something might exist (i.e. it's plausible). You're being very careful (yeah, we're onto you) not to go all the way and say it exists, but you should provide a reason as to why you should think it might exist. So far I don't think you have.


For a start, see http://www.arts.uwaterloo.ca/~pmerikle/pap...tudies.1998.pdf (http://www.arts.uwaterloo.ca/~pmerikle/papers/Merikle.JConsStudies.1998.pdf) .

OK before I go on and try to read the whole paper, is there any part that I should be focusing on? The abstract doesn't indicate anything that may support your position. On the contrary, for instance:
Quote from: from abstract link=msg=0 date=
In addition, recent studies of patients undergoing general anaesthesia have shown that the effects of stimuli perceived unconsciously during surgery can last for approximately 24 hours.

How do you think they tested if there were any effects of unconscious perception? We're not saying that there can't be unconscious perception, only that its effects if any can be tested, and clearly the authors of that paper think the same.

I am sorry to say that it's a common practice of pseudoscience sympathizers to quote irrelevant-but-relevant-sounding papers in the hope that (or sincerely misguidedly) people won't have the time to read them just for an internet discussion. So tell me again, which part of the paper should I be focusing on, and what do you think it states?

Quote from: Mark DeB link=msg=0 date=
If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

On the basis that you haven't made a worthwhile case to be considered. You're running on assumptions that are unwarranted and contradictory (one can "unconsciously perceive" something as different than something else, yet it is not detectable?). And, you're trying to explain a phenomenon that is easily and better explained by other known mechanisms (placebo and all that).
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-19 18:21:13
Entering into serious discourse without defining what it is you're talking about is asinine.


Right, including the topic of meaning (or serious discourse) itself.  The relevant concept, in connection with what you were saying in your previous post, is the "division of linguistic labor," which is explained by, among other writers, Hilary Putnam ("The Meaning of 'Meaning'," Philosophical Papers Vol. 2, and probably also in academic databases such as JSTOR).  This explains how "serious discourse" can go on in one part of a linguistic community thanks to activity in another part of the community, such as the scientific sector, that determines the meaning and reference of the relevant terms (that's why "definitions" don't have to be in the heads of the former).  This is directly relevant to the view you expressed.

Anyway, there's no need to be abusive.  Is this really how you were taught to talk to other people?

Worrying about compression is what I was labelling as "stupid". Worrying about compression (which is a very small thing in the bigger picture) is not something I'd recommend to anyone. I suggest you change your tone if you wish to continue on in these forums. Language can do all kinds of internally-contradictory nonsense. Just because a construct can exist within the bounds of language does not imply that a construct can exist within the bounds of logic or science. Just because I can talk about walking on water personally does not mean that I can walk on water.

You're still missing my main point: What's the point of any of your argument? That there are phenomena that cannot be tested? That tests are specific to the test being taken? It doesn't invalidate a blind test. If we assume that there are untestable subconscious phenomena behind lossy compression, the only difference in outcome is that we have a valid reason to switch to lossless. However, there are already valid reasons to use lossless. If we assume there are untestable subconscious phenomena behind lossy compression, then blind-tests designed to test conscious phenomena are still valid in the context of conscious phenomena.


As far as your main point is concerned, I don't see anything I disagree with in it, although I'm not sure I understand the last sentence (it seems kind of tautological).

In any event: my question was answered:

Consider the following hypothesis.  One of the signals has a fatiguing effect that occurs with longer stimuli, causing the way the music sounds to the person to be subtly different from the way it sounds via the other signal.

(1) Does the best currently available psychoacoustic theory say that this can't happen?


No.


I appreciate the information and thank you all for the discussion.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-19 19:03:08
Clearly the burden of proof comment above is equally valid for people who say something exists, as for people to say that something might exist (i.e. it's plausible). You're being very careful (yeah, we're onto you) not to go all the way and say it exists, but you should provide a reason as to why you should think it might exist. So far I don't think you have.


For a start, see http://www.arts.uwaterloo.ca/~pmerikle/pap...tudies.1998.pdf (http://www.arts.uwaterloo.ca/~pmerikle/papers/Merikle.JConsStudies.1998.pdf) .

OK before I go on and try to read the whole paper, is there any part that I should be focusing on? The abstract doesn't indicate anything that may support your position. On the contrary, for instance:
Quote from: from abstract link=msg=0 date=
In addition, recent studies of patients undergoing general anaesthesia have shown that the effects of stimuli perceived unconsciously during surgery can last for approximately 24 hours.

How do you think they tested if there were any effects of unconscious perception? We're not saying that there can't be unconscious perception, only that its effects if any can be tested, and clearly the authors of that paper think the same.

I am sorry to say that it's a common practice of pseudoscience sympathizers to quote irrelevant-but-relevant-sounding papers in the hope that (or sincerely misguidedly) people won't have the time to read them just for an internet discussion. So tell me again, which part of the paper should I be focusing on, and what do you think it states?

Quote from: Mark DeB link=msg=0 date=
If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

On the basis that you haven't made a worthwhile case to be considered. You're running on assumptions that are unwarranted and contradictory (one can "unconsciously perceive" something as different than something else, yet it is not detectable?). And, you're trying to explain a phenomenon that is easily and better explained by other known mechanisms (placebo and all that).


Sorry, but statements about me are not a scientific basis for anything relevant.  And the sorts of phenomena described in the article clearly do raise the plausibility level of "might" well beyond the "flat earth" or "dragon in garage" level at which you caricatured them as being; against the background of the knowledge that such phenomena exist, it is not at all pointless to ask whether the sorts of phenomena I have asked about exist. 

Yes, you are right that the authors of the article think that unconscious perception can be detected in some ways (otherwise, how would they be able to write the article?), but the point, or one point, is that those ways of detecting it need not be limited to, and might have to be more subtle and indirect, than the sorts of listening tests that are supposed to be adequate for relevant purposes here.  Also, the authors talk about the effect of unconscious perception on emotional states, which might very well be detected not by the subject herself but using behavioral or physiological criteria.  That may give an idea of why I think that article is relevant, and I hope you find it useful.  Your questions in this post were good ones, and, in principle, I would be interested to discuss these issues further, but you have been sufficiently abusive to leave me disinclined to do this; I hope you will understand.  As I noted in my previous post, my question was answered.  Thank you for the discussion.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-19 19:13:08
There is not one single instance in that paper that suggests the issues you've raised about lossy compression or different resolution are to be perceived differently depending on one's level of consciousness.

Sorry Mark, but you'll have to try a little harder.
Title: What's the problem with double-blind testing?
Post by: ExUser on 2010-01-19 19:35:56
Mark, you continue to miss the point. You can cite all the references you like. That does not make them true. This is especially true in philosophical contexts. Some philosophers argue that we can know nothing. So does that mean we do know nothing?

It is immensely ironic to me that you cite papers delineating semantic externalism while making such skeptical arguments, especially when semantic externalism is a philosophical position that stands in opposition to skepticism in many/most cases. You cannot have it both ways.  Your argument is muddled and internally-contradictory, and that's why you are under such heavy criticism. I am sorry you cannot see it that way, but I suggest that instead of writing us off, that you consider what we've written here.

We do like to argue here, but we get irritable when arguments get circular and debaters ignore important points, such as defining the topics of discussion. All the philosophical hand-waving in the world won't make it sensible to ignore a request of other people to define what it is you're talking about when you talk about a thing. Philosophers will ask you to do it. Scientists will ask you to do it. Programmers will demand you do it.
Title: What's the problem with double-blind testing?
Post by: Axon on 2010-01-19 20:40:17
I think it's not said often enough that ABX testing, and controlled testing in general, is of primary importance only to reduce cost. Either by substituting a less expensive product for a cheaper one or by quantifying the risk of some damaging event (ie that some treatment does not in fact work and cannot be distinguished from placebo). Speaking truly off the cuff, I don't think it is a coincidence that blind testing and truly large-scale manufacturing both came into existence at around the same time. That controlled testing also happens to be the best way of evaluating sensory thresholds happens to be a nice coincidence - but I don't think that's necessarily the fundamental point to be making here.

If there did exist such a large gap in covered sensory experience, between "mainstream" blind testing and the experiences of everyday life, one would expect these attempts at cost reduction to be remarkably futile. The value of the supposedly ignored sensoria would rise as its ignorance allows its further and further abuse with further attempts at cost reduction; this "lost" value detracts from the value of the product. If this lost value is negligible, that represents that the effect is probably negligible, too. A lot of money flows around the idea that people like listening to music at 48kbps. That blind testing for cost reduction works - that we don't get ill after going to McDonald's, that they are growing, and our ears don't fall off after listening to XM/Sirius, and people tend not to have any problem listening to lossy encoded music on FM stations - seems to me to be a fairly powerful observation that any sensory gap with blind testing is so small as to be meaningless, and that the "human" factors are far better explanations for preferences which cannot be explained than the results of such controlled tests.

What I'm getting at here is that, while I tend to agree with the sentiment shifting to a distinctly logical positivist angle here, I'm not even sure I'd need to take that hard of a line. Assertions of nonexistence are not falsifiable, but, the possibility of existence can be reduced to a level of unimportance that make it meaningless to everybody. While my understanding of Mark's overall question here seems somewhat reasonable - the question of how blind listening results are converted into statements of universal applicability - the corner he's painting himself into, which is to assert the hypothetical existence of an effect, which does not show up on a blind test, and yet still matters - is ad hoc.
Title: What's the problem with double-blind testing?
Post by: krabapple on 2010-01-19 20:41:58
Fine.  But then it remains an open question whether the signals differ in properties relevant to perception, since you haven't suggested any way to resolve it.



Humans can easily 'perceive' two signals that are *irrefutably* objectively  identical -- i.e., the same signal presented at two different times  -- to be 'different'.  But your main concern is that DBT is somehow missing a real difference?


Quote
Your request for documentation of an observed effect misses the point.  I never claimed to have observed any such effect (and, if you have been following things, you know I don't assume that the effects in question can be reliably reported).  The question is whether there are any effects of the kind described in the hypothesis, and whether tests such as you describe would be relevant to finding this out.  Evidently, they would not be.


Unless and until actual responses by actual people cannot be adequately explained by current models, why do we need to 'find this out'?

What phenomenon are you imagining here that is 1) real and 2) not explained either by significant measurable difference in output of the gear, or placebo-like effects in the subject?


Quote
It may be that a hypothesis that posits things that are undetectable in principle would not be worth considering, but I am not suggesting such a hypothesis.  There might well be ways of detecting whatever properties and effects might make the hypothesis true, although those ways would probably vary with the properties/effects.  So there is no reason why some fixed method of determination should be hard wired into the hypothesis.  However, it is not hard to imagine an alternative to ABX listening tests of the kind you suggest: brain scans.  We might well have excellent theoretical reasons to think that, when a part of the brain lights up in a certain way in response to one stimulus but not another, that this corresponds to a difference in the experiences.


Oy vey, I hear Oohashi coming in 5...4...3...


A 'brain scan' will also show difference due to *belief* in difference, rather than a real difference.  E.g., feelings of pleasure (and their brain scan correlates) can be greater when the subject BELIEVES he is drinking expensive wine vs plonk..even if , in fact, he's been served plonk with an expensive label.


Quote
Look, here is another way of putting the point: Why should I bother with lossy formats for listening when I can use FLAC?


Storage space/bandwidth concerns.  Or a player that does not decode FLAC.

Quote
I don't want to lose any relevant information.  What is the criterion for relevance?  Many people seem to assume that conscious discrimination (in certain conditions) is an adequate criterion, but I see no real argument for this.  It's just a dogma.


Or maybe rationality?
Title: What's the problem with double-blind testing?
Post by: krabapple on 2010-01-19 21:02:41
People are mixing, again and again and again,  'quick switching' with "short auditioning".  Quick switching means the interval between the end of test signal A and the start of test signal B is short (preferably 'zero').  Use of 'short audition' means that the audition periods of A and B are THEMSELVES short -- e.g. measured in seconds rather than minutes/hours.  Both quick switching and use of short audition/short test signals tend to ENHANCE discrimination of difference, by keeping the difference 'fresh in memory'.  That's why they're recommended.

By 'long term ABX' is meant typically that the switching remains QUICK, even though the audition periods themselves are long. 

Anyone who insists that it takes hours of 'sighted' listening to A and B in order to reliably tell them apart, is free to go ahead and do that listening until they're sure they perceive a difference.  THEN try a quick-switch ABX test.


As was said repeatedly(!), ABX has no time limit.
I'm just not convinced(!). In my experience, when the time between two stimuli A and B becomes large (e.g. days, weeks), many non-auditory variables are introduced (e.g. memory).


Yes, that would make it a 'slow switch' ABX -- where the interval between the end of A and the start of B is long .  This makes the test LESS SENSITIVE to difference. 

Quote
Example: several years ago the seats in the Concertgebouw had to be replaced. Requirement was that the famous acoustics had to stay identical. After installation of the carefully designed seats many listeners had the impression that the acoustics had changed. How would you ABX this ?


Well, they probably DID change.  Or were the seats designed to be replicas of what they replaced -- down to the wear-in on the old ones?  But ABX ing it directly would be impossible, impossible to repeatedly (or quickly)  switch between the 'old seats' and the 'new seats'.  (Conceivably one could make a carefully set up recording while the old seats were in place, and another using the same setup with the new seats in place, and play that for listeners in an ABX test.  More easily one could just measure the acoustics of old and new from the same point and see if anything 'looks like' it will cause a significant audible difference.)

This is not equivalent to your friend's problem though.  It IS possible to quickly switch between 24/96 and 16/44, using whatever test signal length your friend desires.
Title: What's the problem with double-blind testing?
Post by: krabapple on 2010-01-19 21:14:15
For a start, see http://www.arts.uwaterloo.ca/~pmerikle/pap...tudies.1998.pdf (http://www.arts.uwaterloo.ca/~pmerikle/papers/Merikle.JConsStudies.1998.pdf) .

If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?


But no one is saying unconscious influences have no effect on perception.  Indeed the primary reason for BLIND TESTING is so biases , often unconscious, do not contaminate the result  (e.g. induce unacceptably large Type I errors).

The real question is, is a 'perception' of difference a reliable indicator of objective difference?  Whatever the actual answer is, it surely is not 'YES, ALWAYS'.  Even a measured difference *in physiological response of the perceiver* is not a reliable indicator that the stimulus itself has changed (though it may indicate that *another* stimulus is at work).
Title: What's the problem with double-blind testing?
Post by: Axon on 2010-01-19 21:27:08
Thanks for the correction on short term/long term/quick audition terminology krab; I'm probably the most guilty party.
Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-01-19 22:36:00
This is not equivalent to your friend's problem though. It IS possible to quickly switch between 24/96 and 16/44, using whatever test signal length your friend desires.
Actually it's not his problem. Using 96 kHz in the studio is a no brainer these days, since there are hardly any downsides. I was just curious if his claim about listening fatigue differences could be verified. Since fatigue needs time to develop, I assume that fast ABX switching is not possible. Besides that he is willing to spend some time on testing, but not days or weeks, especially since there is nothing for him to gain apart from perhaps some insight.
Amongst the thousands of unverified subjective decisions made during each music production, this one is hardly going to make a difference
Quote
The real question is, is a 'perception' of difference a reliable indicator of objective difference?
Often I'm more interested in the perception and how it's being influenced than in the objective analysis of auditory stimuli alone (note how this is different from testing a lossy codec, where the codec is the DUT).
Title: What's the problem with double-blind testing?
Post by: Axon on 2010-01-19 23:20:52
I would imagine the best way to do that test would to do what Meyer/Moran did. Assume that your friend has a digital audio link from his DAW to speakers. Add a device/DVD player/etc of known high quality that can either pass audio through unmodified or resample to 44khz.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-19 23:58:24
Quote from: andy o link=msg=0 date=
Quote from: Mark DeB link=msg=0 date=
If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

On the basis that you haven't made a worthwhile case to be considered. You're running on assumptions that are unwarranted and contradictory (one can "unconsciously perceive" something as different than something else, yet it is not detectable?). And, you're trying to explain a phenomenon that is easily and better explained by other known mechanisms (placebo and all that).


Sorry, but statements about me are not a scientific basis for anything relevant.  And the sorts of phenomena described in the article clearly do raise the plausibility level of "might" well beyond the "flat earth" or "dragon in garage" level at which you caricatured them as being; against the background of the knowledge that such phenomena exist, it is not at all pointless to ask whether the sorts of phenomena I have asked about exist.

Huh? Where am I making an ad hominem here? You're the one raising the question, and you're the one not making a good case. Your case is bad. And you didn't address that criticism at all (from me or anyone else). You're doing classic pseudoscientific misdirection moves.

Quote
Yes, you are right that the authors of the article think that unconscious perception can be detected in some ways (otherwise, how would they be able to write the article?), but the point, or one point, is that those ways of detecting it need not be limited to, and might have to be more subtle and indirect, than the sorts of listening tests that are supposed to be adequate for relevant purposes here.  Also, the authors talk about the effect of unconscious perception on emotional states, which might very well be detected not by the subject herself but using behavioral or physiological criteria.  That may give an idea of why I think that article is relevant, and I hope you find it useful.  Your questions in this post were good ones, and, in principle, I would be interested to discuss these issues further, but you have been sufficiently abusive to leave me disinclined to do this; I hope you will understand.  As I noted in my previous post, my question was answered.  Thank you for the discussion.

Ah, yes. The civility gambit again. Yet no question remains answered.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-20 00:08:22
...not to mention another ironic comment about science again.
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-20 00:22:53
All the philosophical hand-waving in the world won't make it sensible to ignore a request of other people to define what it is you're talking about when you talk about a thing.


Exactly.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-20 00:35:41
...not to mention another ironic comment about science again.



I await his answer to the simple, easily supported observation that ABX testing, signal detection testing, and other methods, all of them double-blind, have been shown to detect stimulii down to the level that mathematics and physics suggest is possible.

Absolute thresholds in the ear canal resonance range, for instance, stand out here, given the established noise level due to the molecular nature of the atmosphere.

It's also true that the basic SNR detection ability of each individual hair cell in the ear is known to be in the range of 30dB, which is extended over about 90-120dB (depending on how you choose toe define "extended) by loudness control mechanisms due to the outer hair cells.  This kind of understanding clearly demonstrates why level changes below a given amount are inaudible, those levels being due to coding, level control, EQ, what-have-you.

Such level calculations agree very well with the actual observed performance of subjects in blind tests. There really is nothing appearing to be missing.

Now, as to the failure to define terms, etc, that is a classical gambit of the dissembler, in that it allows use of the straw man gambit in one of several ways. First, it allows a redefinition of the subject via "but that's not what I meant" dissemblement. Second, it allows  the use of "you said x, therefore you mean y, because z" wherein z is then argued to be a "natural" definition of the original issue. There are other ways in which this gambit can be used, but nobody pretending to be scientific, skeptical, or the like can claim to be such if they are not willing to actually define the assertions that they are making. That, in a nutshell, is what refusing to define a term comes down to.

Such behavior is rude, obnoxious, and does not lead to either enlightenment or concensus, unless you're to take the Derridian approach that "enlightement means there is nothing".
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-20 02:25:17
Priceless.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-22 15:07:10
Don't take my word for it...

"The experiment reported here tests a counterintuitive prediction of this linear-summation hypothesis, namely that a sound that itself is inaudible should, under certain circumstances, affect the audibility of subsequent sounds. The results show that, when two forward maskers are combined, the second of the two maskers can continue to produce substantial masking, even when it is completely masked by the first masker. Thus, inaudible sounds can affect the perception of subsequent sounds."

http://www.jneurosci.org/cgi/content/full/26/34/8767 (http://www.jneurosci.org/cgi/content/full/26/34/8767)

Masking by Inaudible Sounds and the Linearity of Temporal Summation

Christopher J. Plack, Andrew J. Oxenham, and Vit Drga

The Journal of Neuroscience, August 23, 2006, 26(34):8767-8773; doi:10.1523/JNEUROSCI.1134-06.2006
Title: What's the problem with double-blind testing?
Post by: ExUser on 2010-01-22 15:14:11
If it produces effects on the audibility of subsequent sounds, the phenomenon is tractable to blind testing and so this article does not further your claims or explain your hesitation to define what "hear" means.

That article has no bearing on blind-testing whatsoever.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-22 15:20:54
How does this relate to the debated issue for your cause in any way? While it is claimed in this paper that the presence of a masker could not be perceived, a difference could. You have instead been arguing for case where a difference cannot be perceived while still having causal effect.
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-22 15:23:04
If it produces effects on the audibility of subsequent sounds, the phenomenon is tractable to blind testing and so this article does not further your claims or explain your hesitation to define what "hear" means.

That article has no bearing on blind-testing whatsoever.


In the body of the article (as opposed to the abstract) it says: "On each trial, listeners were presented with three observation intervals separated by 300 ms. Two intervals contained the masker(s) only, and one interval (chosen at random) contained the masker(s) plus the signal."

Also, "Listeners were seated in a double- walled sound-attenuating booth and made their responses via a com- puter keyboard. “Lights” on the computer monitor indicated the time of occurrence of the observation intervals and provided feedback as to whether the response was correct or incorrect."

They were also trained on the tasks until performance was stable.

The word "inaudible" as used in this paper is being taken out of context in the posts to this thread.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-22 16:22:10
Don't take my word for it...

"The experiment reported here tests a counterintuitive prediction of this linear-summation hypothesis, namely that a sound that itself is inaudible should, under certain circumstances, affect the audibility of subsequent sounds. The results show that, when two forward maskers are combined, the second of the two maskers can continue to produce substantial masking, even when it is completely masked by the first masker. Thus, inaudible sounds can affect the perception of subsequent sounds."

http://www.jneurosci.org/cgi/content/full/26/34/8767 (http://www.jneurosci.org/cgi/content/full/26/34/8767)

Masking by Inaudible Sounds and the Linearity of Temporal Summation

Christopher J. Plack, Andrew J. Oxenham, and Vit Drga

The Journal of Neuroscience, August 23, 2006, 26(34):8767-8773; doi:10.1523/JNEUROSCI.1134-06.2006


So, lest the point not be clear, I am pointing to the hypothesis that the effects of something analogous to such maskers, which are inaudible short term, may become greater with longer stimulus duration. Now surely that is not an outlandish hypothesis, that a psychophysical effect should become greater with longer stimulus duration?  But that would be a problem for ABX, because if the compared intervals are short then the effect would not occur, whereas if the stimuli are longer then the comparison would be less reliable.

(And there is no necessity to assume that the subject can reliably report the effect using a term in her language, or that the experimenter would know what to ask for.)
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-22 16:48:13
So, lest the point not be clear, I am pointing to the hypothesis that the effects of something analogous to such maskers, which are inaudible short term, may become greater with longer stimulus duration.


Isn't that getting pathetic? Besides not supporting your point in any way (since differences could be detected), how is a <=300 millisecond experiment "analogous" to long term stimulus exposure?

But I said I'd be out and really should be. Just found it somewhat amusing that you thought that you had found something...
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-22 19:38:38
Don't take my word for it...

"The experiment reported here tests a counterintuitive prediction of this linear-summation hypothesis, namely that a sound that itself is inaudible should, under certain circumstances, affect the audibility of subsequent sounds. The results show that, when two forward maskers are combined, the second of the two maskers can continue to produce substantial masking, even when it is completely masked by the first masker. Thus, inaudible sounds can affect the perception of subsequent sounds."

http://www.jneurosci.org/cgi/content/full/26/34/8767 (http://www.jneurosci.org/cgi/content/full/26/34/8767)

Masking by Inaudible Sounds and the Linearity of Temporal Summation

Christopher J. Plack, Andrew J. Oxenham, and Vit Drga

The Journal of Neuroscience, August 23, 2006, 26(34):8767-8773; doi:10.1523/JNEUROSCI.1134-06.2006


What does this have to do with running a listening test? Such time delays are under one second, at the very most, and typically well under 200 milliseconds.  What is more, in a blind test, the subject will do this (assuming it happens) to all signals, and thereby not lose anything.

Of course, the ear just isn't a linear system, so the "linearity" comment is interesting in this light.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-23 01:23:06
And there is no necessity to assume that the subject can reliably report the effect using a term in her language, or that the experimenter would know what to ask for.

It's already been pointed out that this particular concern of yours is irrelevant.  No where in a double-blind ABX test or any of its variants is there a requirement that a person report the effect is his own language or anyone else's.
Title: What's the problem with double-blind testing?
Post by: psycho on 2010-01-23 10:33:06
Mark DeB, consider this:

The effects, you're talking about, that compressed sound COULD have, fall in the same category as the effects that COULD be caused by using different playback hardware.

For instance, let's say that sounds, which lack something undetectable, cause brain cancer. Now, let's have 2 pairs of subjects. First two are listening through the same hardware, difference being one listens to the original-uncompressed sound, the other one listens to the same sound, but compressed. The later gets brain cancer. The other two are listening to the same original-uncompressed sound, difference being one listens with best hardware there is, the other one with a hardware that provides same sounding sound, yet there is an undetectable difference. The later gets brain cancer.

You see... even if you PROVE that audio compression causes brain cancer (or whatever effect on human organism), where do you stop then? Next thing would be audio playback hardware. And if you then prove that bad hardware also causes brain cancer (or whatever effect on human organism), what's next. You go on and persue the idea that jazz causes brain cancer, whereas rock doesn't?!

Sorry, your theory concerns with effects so subtle, that it is just not worth your trouble.
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-23 11:31:12
You should reconsider your use of the term proof, psycho.
Title: What's the problem with double-blind testing?
Post by: psycho on 2010-01-23 17:35:33
rpp3po, please explain. I used proof together with if... (...if you prove...), so I think I used it OK, but I'm ready to learn.
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-23 23:23:01
For testing, quick-switch ABX would be useless

Your repetition of "quick" listening passage length is baseless. You can listen as long as you want. It is the switching time between passages that is quick. Lets quickly discard with that.
I see no reason why I should conclude from the sort of ABX test I took yesterday that the relevant signals do not, in normal listening, cause different experiences in me, or cause me to perceive things in different ways, where the difference is one that matters.

By "normal" listening, you mean sighted and uncontrolled, as you would "normally" listen. Then you don't have an understanding of "perception". Specifically, what is being tested here is hearing perception. This consists of the sound field (the sound waves that impinge upon your ears part) and then the "other" factors of the brains "hearing" process, such as visual stimuli, emotions, biases, beliefs, etc, etc.
ABX is a method of reducing you perception choices to that of the sound field only. If there is a perceptible difference in the sound field, your ABX choices will reveal that. If not,your ABX choices will reveal that also. That does not mean that during the test your brain isn't trying to do what it normally does with regards to emotions and biases, etc. You can still very much imagine hearing differences not in the sound field, that you would during sighted, uncontrolled listening, but your choices will reflect random, coin-flip scoring. In other words, guessing.
The test is not of "your perceptions". You could still "perceive" getting cold listening to A and warm listening to B. The test is of hearing perception. You either hear a difference or you don't. A formal rule of logic is that you cannot prove a negative. A null does not mean there is no difference. It means that you could not "hear" an audible difference under the test conditions.
If you hold a belief that you can "hear" a difference under uncontrolled "listening" (given that you now know that the hearing process involves factors other than the sound field) then take the ABX (which forces you to "hear" just the sound field) where you can no longer "hear" this difference...and still believe that it is the test that has failed you, rather than your belief, then you are a subjectivist. The definition of which, is a person who will evade conclusions, by refusing to believe in them.
Take as much time as you need with the ABX (though switch quickly). The conclusion will be the same, whether you accept it or not.
We'll save fallibility of acoustic memory for another day  .

cheers,

AJ
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-23 23:35:27
The test is not of "your perceptions". You could still "perceive" getting cold listening to A and warm listening to B. The test is of hearing perception. You either hear a difference or you don't.

This does not necessarily have to be the case.  When I said earlier that differences may manifest themselves in other ways I wasn't being facetious.
http://en.wikipedia.org/wiki/Synesthesia (http://en.wikipedia.org/wiki/Synesthesia)

Still, if listening to sample A causes a different conscious sensation (whatever that sensation may be and whether or not it can be properly described with language) than sample B, then this can still be demonstrated through an ABX test.
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-24 00:05:32
This does not necessarily have to be the case.  When I said earlier that differences may manifest themselves in other ways I wasn't being facetious.
http://en.wikipedia.org/wiki/Synesthesia (http://en.wikipedia.org/wiki/Synesthesia)

I thought that was exactly what I implied. So let me restate for clarity. I could care less whether you get warm/cold, whatever, listening to A or B.
Can you "hear" a difference in the sound fields. My interest is acoustics. I'll leave all the "other stuff" of interest to the psychologists.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-24 00:16:34
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-24 00:39:46
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.

Then categorize it as a "all perceptions" test, rather than an "audibility" test, which is what I understand it's purpose to be.
Don't recall an audibility test of codecs asking if whether bitrate change made you feel hotter or colder, even if this was the effect.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-24 00:59:27
The point is that there is no reason to put unnecessary limits on what the test is capable of measuring, giving our recent "skeptic" something else to which he can cling.  That we're debating unfounded speculation about what may occur on a subconscious or unconscious level is bad enough.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-24 09:42:14
The point is that there is no reason to put unnecessary limits on what the test is capable of measuring, giving our recent "skeptic" something else to which he can cling.  That we're debating unfounded speculation about what may occur on a subconscious or unconscious level is bad enough.


Agreed.

It doesn't matter what the actual percept is (i.e. the actual thing, sensory, mental, whatever...) the person percieves that helps them distinguish A from B from X.

All that matters is that they do, or that they don't.

Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-01-24 09:47:22
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.
Like ajinfla I'm confused. Can you explain how an ABX test with acoustical stimuli can reveal "the other stuff" ? Aren't you mixing up ABX and DBT ?
I think ajinfla nailed it down accurately: "My interest is acoustics. I'll leave all the "other stuff" of interest to the psychologists." IMO this is pretty much the attitude on HA, which is perfectly understandable when your field is codecs. OTOH there's the unsatisfying sensation of "I shouldn't hear a difference, but I do" (non-blindly) and ignoring or ridiculing that phenomenon is also short-sighted. Desperately trying to find acoustical explanations (like subjectivists often do) is too though.
Title: What's the problem with double-blind testing?
Post by: Arnold B. Krueger on 2010-01-24 11:43:41
To suggest that ABX, as it pertains to this discussion, is solely for demonstrating a differences limited only to the traditional sensation known as hearing is not only short-sighted, it is plainly incorrect.
Like ajinfla I'm confused. Can you explain how an ABX test with acoustical stimuli can reveal "the other stuff" ?


The idea that ABX is solely for demonstrating differences is debunked by the fact that ABX or testing methods like it are commonly used in industry for things like judging food and beverages.

The idea that ABX is solely for demonstrating the traditional idea of hearing which involves only the ears is debunked by the fact that abx naturally involves the sensations of the listener's entire body.

Some people get confused  when we say that our tests are blind, and speculate about blindfolds or poking out eyeballs. These are very badly confused people, but I have encountered them.

ABX tests are actually fully sighted except that we conceal one tiny aspect of reality for any peson with normal sensations, which is the true identity of what's playing right now. Just that one little thing!

We can even tell you what's playing as long as we tell you enough other things (like the identies of other components that might be playing right now)  that you can't figure out exactly what's playing right now.

The only thing that the subject person can't know is exactly what's playing right now. One tiny little piece of concealment in a divese, real world that is otherwise completely open to the subject person's senses. ABX tests are hardly blind at all! ;-)

Other than that, ABX tests are fully sighted, smell-enabled, tactiley enabled, taste enabled, etc., etc., etc. And, the listening sessions can go on as long as one desires, as many times as one desires.

Needless to say, there are many highly confused people who say ignorant things like ABX can't be used for long term listening. They speak out of ignorance, and I fear in many cases just a little bit of fear of the truth. ;-)
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 14:29:00
And there is no necessity to assume that the subject can reliably report the effect using a term in her language, or that the experimenter would know what to ask for.

It's already been pointed out that this particular concern of yours is irrelevant.  No where in a double-blind ABX test or any of its variants is there a requirement that a person report the effect is his own language or anyone else's.


Said concern is in response to

No, a predicate (like 'feeling fatigued') needs just be on or off. You don't need the memory of another track to assess it as on or off in your test report.


Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 15:26:05
So, lest the point not be clear, I am pointing to the hypothesis that the effects of something analogous to such maskers, which are inaudible short term, may become greater with longer stimulus duration.


Isn't that getting pathetic? Besides not supporting your point in any way (since differences could be detected), how is a <=300 millisecond experiment "analogous" to long term stimulus exposure?

But I said I'd be out and really should be. Just found it somewhat amusing that you thought that you had found something...


If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.  This difference might, to be sure, be one that the subject could readily notice if she could somehow compare those parts of the experiences side by side, but which she cannot do because they are embedded in longer experiences.  And those parts could not simply be reproduced in isolation because the effect is context dependent.

If you switched back and forth between A and B, where A is the signal with said effect, you wouldn't notice a difference since both A and B would be affected by the prior exposures to A.

I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.

Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-24 15:36:21
The idea that ABX is solely for demonstrating differences is debunked by the fact that ABX or testing methods like it are commonly used in industry for things like judging food and beverages.


Then my apologies for being confused. I thought Hydrogen Audio would be primarily interested in the perception of audibility, with regards to music, including codecs or whatever DUT.
So even if the subjects(s) can't tell a lick of audible difference in sound tracks under test, if they get hungry (food) or thirsty (beverages), well this is of interest too.

cheers,

AJ
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-24 15:46:59
If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.  This difference might, to be sure, be one that the subject could readily notice if she could somehow compare those parts

I don't see that anyone here has given any convincing argument, or scientific reason

Agreed
Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-01-24 15:51:55
The idea that ABX is solely for demonstrating differences is debunked by the fact that ABX or testing methods like it are commonly used in industry for things like judging food and beverages.
With "testing methods like it" you mean other forms of double blind testing ? In that case we're probably in complete agreement. It was just my understanding that ABX is useful for detecting perceptual differences whereas other DBT versions that include scaling and multidimensionality are more useful to quantify the variables that are responsible for various judgements.
Title: What's the problem with double-blind testing?
Post by: Soap on 2010-01-24 16:17:14
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.

See Russell's Teapot (http://en.wikipedia.org/wiki/Russell%27s_teapot).
This is a forum of scientific inquiry.  Faith is an incompatible language.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 16:28:25
Quote from: andy o link=msg=0 date=
Quote from: Mark DeB link=msg=0 date=
If, on the other hand, you think that the possibilities I'm asking about are so implausible as not to be worth considering, on what scientific basis do you make that, or any, estimate of their probability?

On the basis that you haven't made a worthwhile case to be considered. You're running on assumptions that are unwarranted and contradictory (one can "unconsciously perceive" something as different than something else, yet it is not detectable?). And, you're trying to explain


Please see below.  And I have not made any assumption that anyone "unconsciously perceives" something as different from something else; that would be to blur the distinction between (1) and (2) in my post #125.  Again, the thought is not the crude and unempirical one that some difference exists that cannot be detected in any way whatever; it is that the means that would reveal those differences would have to be more subtle and indirect than via the sorts of listening tests under discussion.  The psychology literature is replete with descriptions of situations in which such means are necessary, as in the discussion of blindsight in the first article I cited, or in J. Miller's discussion of "discrimination without awareness" (American Journal of Psychology, 1939).

Quote
Quote
Quote
a phenomenon that is easily and better explained by other known mechanisms (placebo and all that).


Sorry, but statements about me are not a scientific basis for anything relevant.  And the sorts of phenomena described in the article clearly do raise the plausibility level of "might" well beyond the "flat earth" or "dragon in garage" level at which you caricatured them as being; against the background of the knowledge that such phenomena exist, it is not at all pointless to ask whether the sorts of phenomena I have asked about exist.


Huh? Where am I making an ad hominem here? You're the one raising the question, and you're the one not making a good case. Your case is bad. And you didn't address that criticism at all (from me or anyone else). You're doing classic pseudoscientific misdirection moves.


I was referring to "On the basis that you haven't made a worthwhile case to be considered."

Look, let's look at the structure of the argument.  Just to clarify, I am not trying to "explain" anything, or make a "case" that some phenomenon exists.  I am saying that certain supposed evidence that a certain (not extremely well defined) kind of phenomenon doesn't exist does not, in fact, constitute good evidence for said conclusion.  At any rate, I have never seen any good argument that it does.

Quote
Yet no question remains answered.


Yes.  Exactly.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 17:16:25
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.

See Russell's Teapot (http://en.wikipedia.org/wiki/Russell%27s_teapot).
This is a forum of scientific inquiry.  Faith is an incompatible language.


Basic principle: believe something if you have a reason for it.

The default position, in the absence of a reason, is doubt. 

No one has actually stated a relevant reason.

Faith would come into it if I were hawking expensive cables, but I'm not.  I'm just saying X doesn't follow from Y.  Moreover, no one, as far as I know, has stated much of a reason to think that Y-and-not-X has an antecedently low probability based on the rest of what we know.  Essentially, what I have suggested is a counterexample to conventional listening tests, a situation that, if it obtained, would be hard to detect using such tests.  The impression I get is that people here simply haven't thought about such a situation and reject it simply because it is unfamiliar, not because they know it can't obtain.

But scientific inquiry includes asking whether, if a certain situation obtained, existing methods would show that it does.  That sort of internal criticism is, according to Popper and others, a key element of science.  So if you say, I'm not going to worry about this until you prove that that situation does obtain, then unless you have theoretical reasons to think that it has an antecedently extremely low probability, you are just sticking your head in the sand.

As I say, doubt, or suspension of belief, is the right attitude to have in this case.  If you disagree, then please give a reason for something other than suspension of belief.  Otherwise, you are believing something without having a reason, which is what you refer to as faith.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-24 17:32:19
The impression I get is that people here simply haven't thought about such a situation and reject it simply because it is unfamiliar, not because they know it can't obtain.

No, you mean unfounded, not unfamiliar.

As I say, doubt, or suspension of belief, is the right attitude to have in this case.

It sure is!

If you disagree, then please give a reason for something other than suspension of belief.

That's pretty ironic.  As you are the one who's making the claims that ABX doesn't work because of the possible existence of XY and Z, it's really up to you to demonstrate their existence.  You have yet to do so.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 19:02:31
The impression I get is that people here simply haven't thought about such a situation and reject it simply because it is unfamiliar, not because they know it can't obtain.

No, you mean unfounded, not unfamiliar.


Please see below.

Quote
As I say, doubt, or suspension of belief, is the right attitude to have in this case.

It sure is!


Well, then, you have conceded the point.

Quote
If you disagree, then please give a reason for something other than suspension of belief.

That's pretty ironic.  As you are the one who's making the claims that ABX doesn't work because of the possible existence of XY and Z, it's really up to you to demonstrate their existence.  You have yet to do so.


No, because the point is that the nonexistence of Y-and-not-X is not a consequence of what we know (i.e., psychoacoustics, acoustics, etc.).  If you disagree, then you must think its nonexistence is a consequence, in which case please say why.

As to the claim that ABX "doesn't work," that is your formulation, not mine.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-24 19:08:20
This is absurd.  You've come here with imaginary reasons why ABX might not work and expect us to prove how these imaginary reasons can't be true?

The onus is not on us, Mark.  It is on you to demonstrate that these reasons aren't imaginary.  You've given us nothing; you have nothing.
Title: What's the problem with double-blind testing?
Post by: Ed Seedhouse on 2010-01-24 19:49:52
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.


Well, if you insist on being scientific, in science the one who proposes something bears the burden of providing evidence.  Since it's you who is proposing such an effect it is up to YOU to provide evidence that it exists and, until you do it is, I suggest, perfectly reasonable to ignore it.

You say there "might be" such an effect?  Fine, show it and show that it makes a detectable difference.  Otherwise, "a difference that makes no difference is no difference".  Or to put it more technically, your suggestion is devoid of actual content.


Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 19:55:17
As to the claim that ABX "doesn't work," that is your formulation, not mine.


So, to elaborate, it depends on what you mean by does or doesn't "work."  If somebody were to claim that certain cables cause a certain gross and readily audible difference, in frequency response say, then I agree that ABX testing could well debunk that claim.  If it seemed to a person that there was such a difference, the ABX test could support the hypothesis that this was an expectation effect.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-24 20:15:03
OMG READING COMPREHENSION!!!
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-24 20:21:25
He's just continuing to ignore posts that don't support his fantasy.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 20:40:09
I don't know that such an effect could occur, but I don't see that anyone here has given any convincing argument, or scientific reason, to think that it couldn't.


Well, if you insist on being scientific, in science the one who proposes something bears the burden of providing evidence.  Since it's you who is proposing such an effect it is up to YOU to provide evidence that it exists and, until you do it is, I suggest, perfectly reasonable to ignore it.

You say there "might be" such an effect?  Fine, show it and show that it makes a detectable difference.  Otherwise, "a difference that makes no difference is no difference".  Or to put it more technically, your suggestion is devoid of actual content.


On "actual content," please see post #215, substituting "actual content" for "meaning."

Actually, I have taken care to distinguish the notion of a difference detectable to the subject from that of a difference in perceptions detectable to the experimenter, way back to the distinction between (1) and (2) in post #125.

As to where the burden of evidence lies, please remember that I am saying that X does not follow from Y.  For all we know, there are some events that are Y-and-not-X.  Now do you agree or disagree with that?  If you disagree, then you must think that we know that there aren't any such events.  If so, how do we know that?  If, on the other hand, you agree with me, then I'm happy with that.

You may, in addition, feel that it is entirely reasonable, nevertheless, to go on to ignore the possibility of such events, even though you don't actually know that there can't be any such events.  Fine.  It's not my concern to disabuse you of a belief in the reasonableness of ignoring that possibility given lack of knowledge one way or the other.  However, it is not at all clear why it would be reasonable to ignore it, for purposes of the present discussion, if you don't know one way or the other.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-24 21:18:01
This is absurd.  You've come here with imaginary reasons why ABX might not work and expect us to prove how these imaginary reasons can't be true?

The onus is not on us, Mark.  It is on you to demonstrate that these reasons aren't imaginary.  You've given us nothing; you have nothing.


As I said, the question of whether ABX "works" depends on the purposes to which it is put.  Consider:

....  But in fact I am more interested in debunking false claims about sound quality. We are surrounded by unfounded information about 15,000 € CD players, or "unlistenable" 320 kbps MP3. And all these claims assume that the difference is conciously audible. Showing that this is not the case can be acheived with ABX tests.
Maybe there are also unconcious differences, that work in these cases, but it is already such a hard work to show that there are no "kickass differences" between two interconnect cables, and that silver interconnect cables do not emphasize treble, that I'm not planning to go beyond ABX (or AXY, or any other decision pattern) for the time being.


"Debunking," whatever exactly is meant, is something stronger than mere doubt or suspension of belief.  It is meant to show or establish something, to show that certain claims aren't true or justified.  And so, yes, there is an onus on someone who would want to do this.  The onus is on this person to give, if requested, a reason to think that his conclusion X is warranted on the basis of the supposed evidence Y: that Y really is evidence for X.  Now if it is possible that Y-and-not-X obtains, then that raises a question as to whether Y really does warrant the conclusion X.  Of course, it is not that the matter is closed at this point, because our background theory might say that such a possibility, while conceivable, is very unlikely; but in this case there is some burden on the debunker to explain how the theory predicts this.

So I am not saying a priori that the onus is on you to prove that Y-and-not-X can't be true, because (for one thing) I don't know if you are a "debunker."  But if you want to infer X from Y, or you want to say that X follows from Y, or you want to "debunk" the claim that not-X on the basis of Y, then it is reasonable to ask you to give a reason to think that the inference is warranted.  And consideration of possible counterexamples is not at all beside the point.

Interestingly, Pio2001 seems to consider the possibility of limitations to ABX also.  I was impressed with the quality of some of the earlier discussion and how well informed some of the participants were in cognition and psychology.

Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-24 21:45:35
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.
Title: What's the problem with double-blind testing?
Post by: DonP on 2010-01-24 21:52:41
If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.


Who said ABX testing requires brief exposure?  Out of curiosity, are you talking "long exposure" like the duration of an album, or like decades? 
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-24 22:17:01
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't this all in reality just a quite inflated straw man discussion for a true, but much lamer story, started in [a href='index.php?showtopic=77910']this[/a] thread:

You had the feeling that upsampling your CD audio to 96 kHz (before it gets upsampled again to 110kHz in your DAC1), A, sounded better than letting the high quality resampler of the DAC1 do the same in a single step, B. You probably did some sighted testing while expecting yourself to sustain maximum objectivity. That all didn't quite work out, since your brain had already locked into believing that A sounded better than B in sighted comparison (a super common phenomenon), but that could basically not be held up after ABX comparison.

Now instead of accepting that your auditory system was just as easily fooled as anyone else's is by sighted comparison, you start this straw man crusade to establish a theoretical pseudo basis for the feeling you had instead of having to feeling fooled. 
Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-24 23:01:06
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't this all in reality just a quite inflated straw man discussion for a true, but much lamer story, started in [a href='index.php?showtopic=77910']this[/a] thread:

You had the feeling that upsampling your CD audio to 96 kHz (before it gets upsampled again to 110kHz in your DAC1), A, sounded better than letting the high quality resampler of the DAC1 do the same in a single step, B. You probably did some sighted testing while expecting yourself to sustain maximum objectivity. That all didn't quite work out, since your brain had already locked into believing that A sounded better than B in sighted comparison (a super common phenomenon), but that could basically not be held up after ABX comparison.

Now instead of accepting that your auditory system was just as easily fooled as anyone else's is by sighted comparison, you start this straw man crusade to establish a theoretical pseudo basis for the feeling you had instead of having to feeling fooled. 


That is what it looked like to me too, but I said nothing. Anyway, the discussion as it's being conducted at the moment is obviously endless: "you have experimental results that prove X for experimental durations up to S_1 seconds; this does not prove that X holds for durations over S_1 seconds. Thus, X may not hold for durations over S_1 seconds". Well yes, this is so painfully obvious that it is barely worth discussing. I don't think anybody has disagreed with this (ie, that it does not follow; this is not the same as "it is not true") anywhere in this thread.

Beyond that there are the following issues, in the random order they occurred to me:
a) what X are we actually talking about? This has shifted around quite a bit during the discussion.
b) is it or is it not reasonable to extrapolate our results for X for durations less than S_1 to those over S_1? Obviously, it is a matter of opinion. In mine, for instance, in the absence of evidence or plausible mechanisms to the contrary, I'd just provisionally extrapolate the results. However, this is apparently unscientific (I'd have suggested some reading of actual scientific literature, as opposed to philosophers discussing science, to see whether this is true or not, but I'd just look smug so won't).
c) There were a few random references to papers, which did not have much to do with the discussion (and it does not seem that the person mentioning them had read anything but the abstract).
d) The tactic of taking a sentence out of a hundred or so and focusing only on that was employed a few times (eg focusing on "common sense" and attacking a particular meaning of that, which differed from the intended, as explained in detail in the post being replied to).
e) others, but I've already wasted enough space.

So in conclusion, and in view of points (a), © and (d), I submit that we are being wound up
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 05:00:06
How do you think they tested if there were any effects of unconscious perception? We're not saying that there can't be unconscious perception, only that its effects if any can be tested, and clearly the authors of that paper think the same.


Yes, you are right that the authors of the article think that unconscious perception can be detected in some ways (otherwise, how would they be able to write the article?), but the point, or one point, is that those ways of detecting it need not be limited to, and might have to be more subtle and indirect, than the sorts of listening tests that are supposed to be adequate for relevant purposes here. 

By the way, unconscious priming is a good example of what I mean by indirect detection.  Priming is the phenomenon in which exposure to one stimulus (the "prime") affects the speed or accuracy of response to a subsequent stimulus (the "target").  Semantic priming is well known, where the priming effect depends on whether the target is related semantically, or by association, to the prime, for example, "dog" may be recognized with greater speed or accuracy if it is preceded by "cat" than by "table" (Healy, ed., p. 453).  It turns out that such priming can occur when the subject is not consciously aware of the prime (Marcel, 1983).  This is known as unconscious or subliminal priming.

So in a situation such as this, the experimenter may be able to detect the effects of the unconscious exposure, all right, but it is not through the subject's ability to discriminate in the normal way; rather, the route to detecting this is much more indirect, and the experimenter has to be pretty sophisticated in knowing what to test for.

Here are some references:

Alice F. Healy et al., ed., Handbook of Psychology, Volume 4: Experimental Psychology (Wiley, 2004)

http://books.google.com/books?id=VNPV0s6I6...cel&f=false (http://books.google.com/books?id=VNPV0s6I6KEC&pg=PA11&lpg=PA11&dq=unconscious+priming+marcel&source=bl&ots=N2T-0Pc7OI&sig=d2ZLK8qZXznCkkiaMb_OL2q2Heg&hl=en&ei=EQpdS8OIAc20tgfcyqCvAg&sa=X&oi=book_result&ct=result&resnum=9&ved=0CDkQ6AEwCA#v=onepage&q=unconscious%20priming%20marcel&f=false)

(see pp. 11 and 453)

Marcel, A. J. (1983). Conscious and unconscious perception: Experiments on visual masking and word recognition. Cognitive Psychology, 15, 197-237.

Greenwald, Klinger, and Liu, "Unconscious processing of dichoptically masked words."
http://faculty.washington.edu/agg/pdf/GK&L.1989.OCR.pdf (http://faculty.washington.edu/agg/pdf/GK&L.1989.OCR.pdf)

Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 05:19:26
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't this all in reality just a quite inflated straw man discussion for a true, but much lamer story, started in [a href='index.php?showtopic=77910']this[/a] thread:

You had the feeling that upsampling your CD audio to 96 kHz (before it gets upsampled again to 110kHz in your DAC1), A, sounded better than letting the high quality resampler of the DAC1 do the same in a single step, B. You probably did some sighted testing while expecting yourself to sustain maximum objectivity. That all didn't quite work out, since your brain had already locked into believing that A sounded better than B in sighted comparison (a super common phenomenon), but that could basically not be held up after ABX comparison.

Now instead of accepting that your auditory system was just as easily fooled as anyone else's is by sighted comparison, you start this straw man crusade to establish a theoretical pseudo basis for the feeling you had instead of having to feeling fooled. 


No, I think that expectation effects are considerable, and that the difference I "heard" was probably my imagination.

But how does this affect the point?  What constitutes evidence for what does not depend on my motives, after all.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 05:21:16
If the effect is greater with longer stimulus duration, then it may be that a brief exposure would make no detectable difference, but a longer exposure would create a difference in the way the passage sounds.


Who said ABX testing requires brief exposure?  Out of curiosity, are you talking "long exposure" like the duration of an album, or like decades?


Long enough so that the subject can't reliably compare the sound of the two passages.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 05:44:56
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't it enough that I have explained the conditions under which such a case would arise?  An effect that is undetectable in brief exposure but which increases with longer stimulus duration, say, one minute?  Isn't this something that would be hard to detect by having someone listen to 1-minute stretches, because, to repeat, she can't compare the sound of corresponding parts directly; but which also wouldn't show up in switching back and forth, because A and B would be affected equally by the prior exposure?  This is not a valid theoretical and methodological point?

I'm sorry, but is what I have been suggesting, and repeated in the previous paragraph, not clear?  Or are you saying that you know that it couldn't happen?
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-25 06:12:02
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't it enough that I have explained the conditions under which such a case would arise?  An effect that is undetectable in brief exposure but which increases with longer stimulus duration, say, one minute?  Isn't this something that would be hard to detect by having someone listen to 1-minute stretches, because, to repeat, she can't compare the sound of corresponding parts directly; but which also wouldn't show up in switching back and forth, because A and B would be affected equally by the prior exposure?  This is not a valid theoretical and methodological point?

I'm sorry, but is what I have been suggesting, and repeated in the previous paragraph, not clear?  Or are you saying that you know that it couldn't happen?


Ok, now, this is stuff and nonsense. There is no difference between whatever hypothetical, unproven effects would happen in an ABX test and that which would happen in any equal length non-ABX test. Period. ABX tests have no time limit, etc.  You've been told this several times, you've ducked, weaved, and evaded several points, and now you ask "isn't it enough".

No, it's not. I'm convinced you are simply trying to play games here, given the total bankruptcy of your own position, and your own failure to realize how your very silly objections (look, there has been a lot of study on what you hear and remember, and the relevant time periods appear to be 200 milliseconds and a few seconds, and then long-term memory.  This is not supposition, this is what people who have actually done work, investigated the subject, and attempted to learn more have actually discovered.) read on any other kind of listening test than ABX, including sighted.

You simply have nothing. Nothing.  I would suggest that you stop, if you really are sincere, which I sincerely doubt, and think a minute, and realize that the silliness, and I'm sorry, but based on the modern understanding it looks very, very silly, that you propose above would happen in any listening situation of the given length. There is nothing there about ABX tests or DBT's that is not the same for anything else.

I'm trying not to be rude, but either you're a total wind-up, or you simply aren't thinking through your own unsupported, untested ideas.

If you want to convince someone, figure out an appropriate test, run it, and submit it for publication in JASA or ASSP Transactions.

Your bit about 'forgetting what it sounded like' would be a problem in any test if there was anything to your idea, hence the idea isn't even well formed.

Finally, your introducing "priming" suggests to me that you don't know the difference between listener training and listener testing, and that you don't understand the duration of partial loudness memory. Perhaps a little literature review of the basics, and some reviewing of definitions such as "what is sound" in a technical sense might help you learn enough to make sense.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-25 06:41:58
Who said ABX testing requires brief exposure?  Out of curiosity, are you talking "long exposure" like the duration of an album, or like decades?


Long enough so that the subject can't reliably compare the sound of the two passages.

So if the subject can't reliably compare the sound, why does it matter anyway?

If there was a magical difference that will make her happier/sadder, she will reliably compare it over whatever long period of time is needed. If she can't, it's because it doesn't make her happy/sad (or ANY other secondary "effect"). If there's no reliable secondary effect, again, why does it matter?
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-25 10:07:20
Isn't it enough that I have explained the conditions under which such a case would arise?


No, just because you suffer from postmodern analytical philosophy doesn't mean you'll find mercy here (nor a cure). Most of the time that disciple does not comes further than a few hypothetical papers referencing each other to be obsoleted by the next analytical trend. The length of a book has become too much for many proponents. Discussing hot air is often preferred, as you have just imposingly demonstrated.

I'm sorry, but is what I have been suggesting, and repeated in the previous paragraph, not clear?  Or are you saying that you know that it couldn't happen?


The point is, as I have tried to explain many times: the set of comparable thin air hypotheses, what could and could not exist and whatnot, is infinite. To decide which one of those is worth delving into, it helps if there is at least a tiny bit of empirical substance (e. g. unexpected results in an experiment), to limit the risk of totally wasting your own time and here particularly that of others.

You have failed to deliver on that request up to this moment and that's why you are confronted with so much opposition. It is wasting people's time, but still they aren't willing to let your rhetorical hanky-panky raise the impression that the HA community was ignorant.

That's why I asked for a particular case. If there was one, the discussion could be more fruitful.
Title: What's the problem with double-blind testing?
Post by: DonP on 2010-01-25 11:53:04
But how does this affect the point?  What constitutes evidence for what does not depend on my motives, after all.


Well it does put it in the same category as tobacco company research showing that cigarettes are harmless, or a coal company treatise on global warming.


Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 13:30:30
He's just continuing to ignore posts that don't support his fantasy.


"Fantasy" = "Hypothesis" + "[Boo! Hiss!]" + "[I don't think that's very likely]"

Reason for the last part?

Would you have thought that unconscious priming could occur?
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 14:31:28
Could you please state a particular case, for which you expect the forum to accept the limits of ABX testing, but haven't found acceptance, yet? One might get the impression that you are withholding this, so that your fantasy - as greynol called it so nicely  - doesn't get harmed.


Isn't it enough that I have explained the conditions under which such a case would arise?  An effect that is undetectable in brief exposure but which increases with longer stimulus duration, say, one minute?  Isn't this something that would be hard to detect by having someone listen to 1-minute stretches, because, to repeat, she can't compare the sound of corresponding parts directly; but which also wouldn't show up in switching back and forth, because A and B would be affected equally by the prior exposure?  This is not a valid theoretical and methodological point?

I'm sorry, but is what I have been suggesting, and repeated in the previous paragraph, not clear?  Or are you saying that you know that it couldn't happen?


Ok, now, this is stuff and nonsense. There is no difference between whatever hypothetical, unproven effects would happen in an ABX test and that which would happen in any equal length non-ABX test. Period. ABX tests have no time limit, etc.  You've been told this several times, you've ducked, weaved, and evaded several points, and now you ask "isn't it enough".


If you are saying that the sorts of things that a conventional ABX listening test could discover includes all of the things that any psychological test in general could discover, that seems false.  To wit: unconscious priming.  The subject can't discriminate the unconscious stimuli in the usual manner, but the experimenter can detect the effects of the information.  For all we know, there are effects that would not be detected, or would be hard to detect, by any conventional ABX listening test.  Possibly psychologists could come up with other ways of detecting those effects.  So it is incorrect to say that there is "no difference."  (Not sure what you mean by "equal length.")

Quote
No, it's not. I'm convinced you are simply trying to play games here, given the total bankruptcy of your own position, and your own failure to realize how your very silly objections (look, there has been a lot of study on what you hear and remember, and the relevant time periods appear to be 200 milliseconds and a few seconds, and then long-term memory.  This is not supposition, this is what people who have actually done work, investigated the subject, and attempted to learn more have actually discovered.) read on any other kind of listening test than ABX, including sighted.

You simply have nothing. Nothing.  I would suggest that you stop, if you really are sincere, which I sincerely doubt, and think a minute, and realize that the silliness, and I'm sorry, but based on the modern understanding it looks very, very silly, that you propose above would happen in any listening situation of the given length. There is nothing there about ABX tests or DBT's that is not the same for anything else.

I'm trying not to be rude, but either you're a total wind-up, or you simply aren't thinking through your own unsupported, untested ideas.

If you want to convince someone, figure out an appropriate test, run it, and submit it for publication in JASA or ASSP Transactions.

I'm sorry if the point isn't clear.  It's not that I am saying that there are effects or phenomena of the kind hypothesized.  It's that the relevant tests don't show that there aren't.  The question is what those tests show
Quote
Your bit about 'forgetting what it sounded like' would be a problem in any test if there was anything to your idea, hence the idea isn't even well formed.


If you think that idea is not well formed, try the following.  Repeatedly compare two pairs of passages a minute in duration, where in each pair the passages are either identical or subtly different in timbre.  The task is to say whether they are the same or different.  In one version of the task, you listen straight through to each passage, and then you say whether you thought they were identical or not.  In another version, you switch back and forth between them.  Which do you think you are going to do better at?

Experience shows, I think, that the latter is going to be a more sensitive test. 

But, if there are the sorts of effects I am asking about, ones that build up over time, then they will not show up on the latter sort of test because A and B are affected equally by the prior exposure to A (the signal that causes the effect) that is occurring in the test. 

So we will be stuck with the longer, less reliable tests (if we remain within ABX).

Is what I have just described not clear?  Do you know that there cannot be such effects?  Do the studies you refer to show that there cannot be?  Or are you just going to say that it is "very, very silly"?

Quote
Finally, your introducing "priming" suggests to me that you don't know the difference between listener training and listener testing, and that you don't understand the duration of partial loudness memory. Perhaps a little literature review of the basics, and some reviewing of definitions such as "what is sound" in a technical sense might help you learn enough to make sense.


If there is a scientific explanation, in terms of the duration of partial loudness memory, or anything else, that would answer these questions, then that's what I've been requesting from the very beginning.  Please, go ahead, explain in scientific terms why what I'm hypothesizing couldn't occur.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 14:40:11
Who said ABX testing requires brief exposure?  Out of curiosity, are you talking "long exposure" like the duration of an album, or like decades?


Long enough so that the subject can't reliably compare the sound of the two passages.

So if the subject can't reliably compare the sound, why does it matter anyway?

If there was a magical difference that will make her happier/sadder, she will reliably compare it over whatever long period of time is needed. If she can't, it's because it doesn't make her happy/sad (or ANY other secondary "effect"). If there's no reliable secondary effect, again, why does it matter?


You are assuming that if a person is in a state that matters, then he can monitor that state and is very good at distinguishing when he is in that state from when he is not.  But consider: is it possible to be angry without knowing it?  A person might have an interest in not being in such states.  That might well matter to the person.  So what a person can reliably discriminate does not necessarily set the boundaries on what matters to him.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 14:45:11
But how does this affect the point?  What constitutes evidence for what does not depend on my motives, after all.


Well it does put it in the same category as tobacco company research showing that cigarettes are harmless, or a coal company treatise on global warming.


Of course this conversation is like one with the tobacco company.  It is like a person in the 1920's saying, "Will cigarette smoking harm my health?" and the tobacco company saying, "Define health."
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 15:06:03
Isn't it enough that I have explained the conditions under which such a case would arise?


No, just because you suffer from postmodern analytical philosophy doesn't mean you'll find mercy here (nor a cure). Most of the time that disciple does not comes further than a few hypothetical papers referencing each other to be obsoleted by the next analytical trend. The length of a book has become too much for many proponents. Discussing hot air is often preferred, as you have just imposingly demonstrated.

I'm sorry, but is what I have been suggesting, and repeated in the previous paragraph, not clear?  Or are you saying that you know that it couldn't happen?


The point is, as I have tried to explain many times: the set of comparable thin air hypotheses, what could and could not exist and whatnot, is infinite. To decide which one of those is worth delving into, it helps if there is at least a tiny bit of empirical substance (e. g. unexpected results in an experiment), to limit the risk of totally wasting your own time and here particularly that of others.

You have failed to deliver on that request up to this moment and that's why you are confronted with so much opposition. It is wasting people's time, but still they aren't willing to let your rhetorical hanky-panky raise the impression that the HA community was ignorant.

That's why I asked for a particular case. If there was one, the discussion could be more fruitful.


It would be failure only if my purpose were different. 

Yes, the space of possibilities is infinite.  But that doesn't mean that (as if it were legitimated by that abundance) we're entitled to latch onto some portion of it, and take an attitude of belief toward it, without a reason. 

As I said in another, recent post, doubt seems to me to be what is called for in this situation.  If you think that anything more than doubt is called for, I would be interested to know what your reason is.

Or are you saying that, unless I "deliver on [your] request," you are entitled to believe something without a reason?
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-25 15:33:48
But consider: is it possible to be angry without knowing it?

Not if you're asking yourself if you're angry. If you're not that self-aware, then, again, there's no point in speculating whether there's a difference.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-25 16:57:07
If you are saying that the sorts of things that a conventional ABX listening test could discover includes all of the things that any psychological test in general could discover, that seems false.

So.

Is there some reason you made up a straw man rather than actually responding ot what I said?
Quote
I'm sorry if the point isn't clear.  It's not that I am saying that there are effects or phenomena of the kind hypothesized.  It's that the relevant tests don't show that there aren't.  The question is what those tests show.

Perhaps then you need to look into the experimental design?
Quote
If you think that idea is not well formed, try the following.  Repeatedly compare two pairs of passages a minute in duration, where in each pair the passages are either identical or subtly different in timbre.  The task is to say whether they are the same or different.  In one version of the task, you listen straight through to each passage, and then you say whether you thought they were identical or not.  In another version, you switch back and forth between them.  Which do you think you are going to do better at?

Experience shows, I think, that the latter is going to be a more sensitive test.

Since you can do an ABX test either way, what does this have to do with the price of corn in Lapland, anyhow?
Quote
But, if there are the sorts of effects I am asking about, ones that build up over time, then they will not show up on the latter sort of test because A and B are affected equally by the prior exposure to A (the signal that causes the effect) that is occurring in the test.

If, if, if.  Why don't you simply show that these effects exist in some kind of test that passes peer review in a reputable journal?  The knowlege of present suggests that long-term tests are less sensitive, period.  Your idea of long-term issues is limited, I dare say, to the idea of listener training, and if you do see such an effect it's most likely the listener learning, in which case your listener training is what is indicted.
Quote
Is what I have just described not clear?  Do you know that there cannot be such effects?  Do the studies you refer to show that there cannot be?  Or are you just going to say that it is "very, very silly"?

First, you can't prove a negative, so that's outside the realm of science already. Then, I can also propose that the phase of the moon affects your hearing, or that the surface temperature at the north pole of Betelgeuse affects your hearing...

What you need is evidence, and evidence in a test with proper listener training.  You do understand that discovering small artifacts in a long-term detection experiment depends on the first integration time (well, physiologically it won't be that clean, but the point holds) time of the detector, yes?

And that, in hearing, is known to be under 200 milliseconds.

Ergo, anything you're detecting in this long-term experiment will be at a much less sensitive level, as it must be due to the known loss of information between partial loudnesses and auditory features.

This puts the effect, very likely, (as I said there is no proving a negative) into the catagory of learning. That's what you try to avoid by training listeners to start with.
Quote
If there is a scientific explanation, in terms of the duration of partial loudness memory, or anything else, that would answer these questions, then that's what I've been requesting from the very beginning.  Please, go ahead, explain in scientific terms why what I'm hypothesizing couldn't occur.


One can not prove a negative, one can simply show that unless the entire understanding of hearing is overturned, it's extremely unlikely.

What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc. Test after test that does sequential A/B, or worse, the "other" abx test in which it is sequential (one proposed as a straw man bad test in the 1950's, having nothing to do with the current ABX test methodology) have been demonstrated to be less sensitive, again and again, with material detect in an ABX or ABC/hr test.

In short, both the current understanding and the evidence argue otherwise. If you can muster up experimental support for your idea, then it becomes interesting. Otherwise, you're tilting at a windmill.

What's more, I still don't think you've addressed a very basic issue, that of how you AVOID having the same problem with any other test that you have with ABX, or ABC/hr, or whatever blind test. If the effect exists, it affects all tests.

More to the point, barging in with this hypothesis, with no evidence, running counter to the present knowlege in the subject, really looks like a pure wind-up. Much more of the trying to put words in my mouth and you'll have to go find yourself another expert to annoy.

Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-25 17:09:53
Yes, the space of possibilities is infinite.  But that doesn't mean that (as if it were legitimated by that abundance) we're entitled to latch onto some portion of it, and take an attitude of belief toward it, without a reason.


It does indeed sound like the fallacy that asserts if we don't have full information, we have none, doesn't it?
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 21:27:04
But consider: is it possible to be angry without knowing it?

Not if you're asking yourself if you're angry. If you're not that self-aware, then, again, there's no point in speculating whether there's a difference.


There is room for other views.  Freudian psychology, for example, suggests that people can have emotions or desires that are not always accessible simply by "asking yourself."  And it can matter to a person whether or not she is in such a state. 

You may think that there is no point in caring about such states, but others may not feel that way.  And with good reason.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-01-25 21:32:51
Can you tie this back to the original topic citing results to any relevant repeatable peer-reviewed studies?
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-25 21:34:25
Yes, the space of possibilities is infinite.  But that doesn't mean that (as if it were legitimated by that abundance) we're entitled to latch onto some portion of it, and take an attitude of belief toward it, without a reason.


It does indeed sound like the fallacy that asserts if we don't have full information, we have none, doesn't it?


No, why do you think it sounds like that?

Are you disagreeing with my second sentence, then?  Do you think that we ought to have beliefs, assert them, encourage others to believe them, etc., without having a good reason to believe them, i.e., without being justified in believing them?
Title: What's the problem with double-blind testing?
Post by: Ed Seedhouse on 2010-01-25 22:50:14
An effect that is undetectable in brief exposure but which increases with longer stimulus duration, say, one minute?


Would be detectable in an ABX test, given that there is no limit of duration in such a test.

You have been told several times that there is no time limit in an ABX test, which is a double blind test.  So continuing to use such examples is either disingenuous or fantastically ignorant.

Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-25 22:51:00
No, why do you think it sounds like that?

Are you disagreeing with my second sentence, then?  Do you think that we ought to have beliefs, assert them, encourage others to believe them, etc., without having a good reason to believe them, i.e., without being justified in believing them?



A classical use of appeal ad ignorantum here.  It ought to be in a book of rhetorical misconduct.

Dude, Mark, got evidence? Got any? If you do, out with it. If you don't, the moon ain't made of green cheese, neither.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-25 22:54:51
An effect that is undetectable in brief exposure but which increases with longer stimulus duration, say, one minute?


Would be detectable in an ABX test, given that there is no limit of duration in such a test.

You have been told several times that there is no time limit in an ABX test, which is a double blind test.  So continuing to use such examples is either disingenuous or fantastically ignorant.



I think it's a pure wind-up at this point. I've made this point several times, only to have it ignored with a 'but,but, but'. Then when he asks for real time constants in listening, and I tell him some, he ignores them, and, rather, objects to my pointing out his fundamental rhetorical fallacies.  Even his objection to that contains a straw man and an appeal ad ignorantum.

Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-25 23:18:34
I've made this point several times, only to have it ignored with a 'but,but, but'. Then when he asks for real time constants in listening, and I tell him some, he ignores them


Quote
Subjectivist Fallacies:
                                                                                        Subjectivism

In an argument of this sort, a subjective state--the mere fact that we have a belief or desire--is used as evidence for the truth of a proposition.

We can see what's wrong with this argument by identifying the implicit premise. To make this argument stronger, one would have to accept the premise that whatever I believe or want to be true is true. That is, subjectivism implicitly assumes that we are infallible. And of course we aren't.

Subjectivism is not only a way of adopting conclusions on subjective grounds, but also--and probably more often--a way of evading conclusions by refusing to believe in them. Some people have perfected the skill of simply not seeing what they don't want to see, and most of us indulge in this habit occasionally. If the habit were put into words, it would take the form, "I don't want to accept p; therefore, p isn't true." That's subjectivism.

Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-26 00:14:09
The point is, as I have tried to explain many times: the set of comparable thin air hypotheses, what could and could not exist and whatnot, is infinite. To decide which one of those is worth delving into, it helps if there is at least a tiny bit of empirical substance (e. g. unexpected results in an experiment), to limit the risk of totally wasting your own time and here particularly that of others.

You have failed to deliver on that request up to this moment and that's why you are confronted with so much opposition. It is wasting people's time, but still they aren't willing to let your rhetorical hanky-panky raise the impression that the HA community was ignorant.

That's why I asked for a particular case. If there was one, the discussion could be more fruitful.

I agree that if I were to provide empirical evidence that what I am suggesting is possible is not only possible but actual--which is what you seem to be requesting--that would help advance matters.  However, I have nothing to offer at present.

But lest anyone get "the impression that the HA community was ignorant" (your words, not mine), here is its opportunity to shine.  Do you have any scientific basis on which to say that the possibility in question--that stimuli that cannot be discriminated from one another could have relevantly different causal influences on perception--cannot obtain?

If not, I fail to see any "opposition" of substance, or any actual disagreement on your part with anything I've said.

And this dialogue just reinforces the observation I made at the outset, that I have never seen a convincing argument that such a possibility cannot obtain.  You certainly haven't provided one.  If you think the question is a waste of time, nobody's forcing you to engage with it.
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-26 00:19:50
But lest anyone get "the impression that the HA community was ignorant" (your words, not mine), here is its opportunity to shine.  Do you have any scientific basis on which to say that the possibility in question--that stimuli that cannot be discriminated from one another could have relevantly different causal influences on perception--cannot obtain?

You've shifted the question yet again. That one's easy.

Take three signals 1.5 dL's apart, to be clear, one a level x, one at level x+.75dL, and one at x+1.5dL.

Compare the two lower loudness ones. No difference.
Compare the two louder ones, no difference.
Compare the louder and softer. Detectable difference.

No news here.  No evidence for your assertion, either. Just more evasion of the substance on your part, while making the offensive accusation quoted below:
Quote
If not, I fail to see any "opposition" of substance, or any actual disagreement on your part with anything I've said.


I would suggest that you go back and re-read my comments on time constants in the auditory system.  There is more substance in that one post than you have offered to date, given that the data I give you dates back in its origins over 100 years.
Title: What's the problem with double-blind testing?
Post by: Soap on 2010-01-26 00:31:10
The thread is already moving in this general direction, and I beg it continue:

I propose that the only way to end this circle jerk is to limit the conversation to one voice at a time.  It is when there are four or five (or ten) responses to address that we see the most picking and choosing, bobbing and weaving.  If you feel, as I do, that this has gone on for days longer than necessary, I beg of you to refrain from addressing a point until that time the last point has been satisfactorily answered.
Title: What's the problem with double-blind testing?
Post by: Mark DeB on 2010-01-26 00:59:37
What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc.


Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?

Are the tests you refer to ABX?  If ABX tests aren't going to be good at revealing the existence of said effects, isn't this begging the question?
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-26 01:05:59
The thread is already moving in this general direction, and I beg it continue:

I propose that the only way to end this circle jerk is to limit the conversation to one voice at a time.  It is when there are four or five (or ten) responses to address that we see the most picking and choosing, bobbing and weaving.  If you feel, as I do, that this has gone on for days longer than necessary, I beg of you to refrain from addressing a point until that time the last point has been satisfactorily answered.

this is me agreeing
Title: What's the problem with double-blind testing?
Post by: Light-Fire on 2010-01-26 01:09:40
What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc.


Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?

Are the tests you refer to ABX?  If ABX tests aren't going to be good at revealing the existence of said effects, isn't this begging the question?


If your "context-dependent effects" interfere with double blind tests they would also interfere with a normal (not blind) comparison as well. So why are you picking on ABX?!
Title: What's the problem with double-blind testing?
Post by: rpp3po on 2010-01-26 01:23:06
Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?


Since you haven't actually suggested anything particular and even resist to do so, although you act like you had, let me fill in the formal blank:

I bought a new toilet paper brand last month, I didn't perceive any difference in everyday usage until today, when you brought it to my attention again while I was thinking about an example for your nonsense. I can't ABX the rolls, they look exactly identical to the old brand. The only difference is the outer packaging and that they come from different factories. Well, since 2 weeks of thorough long-term usage I cannot report any conscious difference. There is always a lot of peaceful emotion involved when I take a shit, but the time spans are to long to get a clear picture of any difference. Exactly as you want it.  Still the papers are different of course, but since I haven't perceived it, I suggest you install a MRI in my toilet to evaluate the highly probable chance that I had indeed "different experiences" but was just unable to become aware of it. Your "proof" might involve claiming that in 78% of rounds, when I used paper A, region xyz of my brain resulted in green color on the screen, which else was orange. Is that what you want?

If it is, I reject it and will call it uncorrelated noise at position xyz with no causal effect onto my mental state. How do you want to react, what's your truth condition? You might want to increase the number of rounds to 1000 and then claim rock hard evidence after running an automated analysis on the MRI data. Well if I really could not perceive a difference after so many trials, I could not care less. Maybe you just confused it with a dead salmon (http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf) (great paper!)? Do you have any truth condition other than things as brain scans, that you basically cannot even explain?

PS Soap, I don't believe that your proposal is going to work. As long as "the one" would answer with n>1 sentences Mark would be able to continue ignoring any n-1. So why not just take this with humor? The thread is a juggernaut already, anyway, and circling as you say.
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-26 02:27:29
However, I have nothing to offer at present.

Hurray!! We're finally there.

......wait.....are we?? 

Title: What's the problem with double-blind testing?
Post by: aclo on 2010-01-26 03:24:03
Maybe you just confused it with a dead salmon (http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf) (great paper!)? Do you have any truth condition other than things as brain scans, that you basically cannot even explain?


Wow, this makes my posters look lame by comparison! Brilliant!

Quote
PS Soap, I don't believe that your proposal is going to work. As long as "the one" would answer with n>1 sentences Mark would be able to continue ignoring any n-1. So why not just take this with humor? The thread is a juggernaut already, anyway, and circling as you say.


Right. I don't understand why anybody bothers to argue any more. This probably is some sort of show put up for his friends somewhere. Or he could be serious about it, I've met intelligent, articulate people who really did behave like this--a philosopher and two lawyers, as it happens. The two lawyers are good friends, and one is as smart as anybody I know. Didn't help her much: she claims that a ball, dropped in a (sealed) airplane, falls backwards. I tried all sorts of arguments: "which frame of reference does it fall back in?", "what force causes it to fall back?", "how does it know the thing is moving?", "you yourself could not tell whether the plane is moving or not if it's not accelerating, how could the ball?", "I am a physicist and I am telling you, it doesn't!!", to no avail. She still seems convinced that a ball dropped in a plane will move backwards (slightly, which is why we can't see it...). And no she's not winding me up.

So either we're dealing with a similar case, or our "skeptic" is in fact winding everybody up, and doing it suspiciously professionally (notice calm composure, mostly ignoring ridicule, apparently reasonable arguments, impressive looking references etc--all this looks very much designed to appeal to an outside audience and make it look like he does in fact have a point). I vote for the second.
Title: What's the problem with double-blind testing?
Post by: andy o on 2010-01-26 08:16:53
I have a friend who used to go to a medium (or card-reader or whatever it was) and it's been pretty much years now that I've slowly made her skeptical to it, but once in a while she still comes telling me about her sister's bad luck or some sort of bad energy somewhere. The way humans have evolved, we have to make a special effort not to fall for this stuff. I almost bought an ultrasonic roach repellent today!
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-26 11:25:00
I almost bought an ultrasonic roach repellent today!

You should have. If it didn't work on roaches, you could always sell it for $4k on Audiogon as an "audiophile" super tweeter. They can hear up into the Ghz...and will buy anything.

cheers,

AJ
Title: What's the problem with double-blind testing?
Post by: Woodinville on 2010-01-26 21:04:03
What's more, such issues have never been demonstrated, in a variety of tests that involve fast switching, slow switching, no switching, etc.


Has anyone ever tested for the sorts of longer-term, context-dependent effects I've suggested?

Are the tests you refer to ABX?  If ABX tests aren't going to be good at revealing the existence of said effects, isn't this begging the question?



WHAT effects that won't happen in any test, ABX or not. Please be specific.
Title: What's the problem with double-blind testing?
Post by: ajinfla on 2010-01-27 00:21:12
WHAT effects that won't happen in any test, ABX or not.

The correct term is WITCH effects (http://www.enjoythemusic.com/Magazine/BAS/1208/)
Title: What's the problem with double-blind testing?
Post by: djcombes on 2010-02-23 22:11:57
I have an idea as how to test what djcombes and Mark DeB are trying to get tested... There should be a reality show that lasted for a month or so and the contestants would have to listen to a lot of music and they would never know the source and they would always have to grade their experience level, aswell as guess if it came from original source or if it was a compressed sample.

All in all, I'm sure the results would show that there would be no difference in experience grades between original sources and compressed ones. I am sure that somedays the contestants would grade compressed samples better and then other days they would grade the originals better...

Anyway... such a prolonged test would be interresting and would definetly bring more insight and more evidence for this debate.
You've just described a long term ABX test. It's been done already and is not what was being suggested by duff. Duff was suggesting that there might be a difference in perception below the threshold of conscious perception. It's a bit angels on the head of a pin (if it can't be perceived without some convoluted test methodology does it matter?), but he might be correct. Since it involves something unconscious, the hypothesis is difficult to investigate. It's necessary to come up with a test method in which the response is not mediated by conscious thought (such as would be required in rating an experience). Previous experiments (not related to music or lossy compression) have demonstrated that under some circumstances a stimulus which is below the threshold of perception (in the ABX sense), can nonetheless have an effect on reaction time (for instance).

I can't work out how you set up a decent test to investigate. You need to be able to translate a listening experience into some kind of motor response without the need for conscious thought. Tricky! I'm guessing that duff hasn't come up with a sensible approach, or he has and found he's wrong - i.e. no evidence for sub-conscious perception of lossy compression . Either way he's not returned to comment.
Title: What's the problem with double-blind testing?
Post by: muaddib on 2010-02-24 11:55:49
You need to be able to translate a listening experience into some kind of motor response without the need for conscious thought. Tricky!

What about clicking those buttons in an ABX test app without conscious thoughts?
Title: What's the problem with double-blind testing?
Post by: djcombes on 2010-02-24 13:32:03
What about clicking those buttons in an ABX test app without conscious thoughts?
I think the act of comparison involves conscious thinking about it. You'd need something more along the lines of measuring foot tapping response to a catchy tune, and seeing if it was correlated with the level of compression. That's obviously a hopeless example, but I can't think of a good way to design an experiment to check for this.

Which does kind of relegate it to the level of how many angels can dance on the head of the pin.

ETA - I've just re-engaged my sarcasm detector. It was obviously switched off.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-02-24 15:35:17
Duff was suggesting that there might be a difference in perception below the threshold of conscious perception.
WRT audio, again, based on what evidence exactly?

From his last relevant post (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=56797&view=findpost&p=510877) on this forum, back in August of 2007:
Quote
That experiment is coming...
Guess what, we're still here waiting.
Title: What's the problem with double-blind testing?
Post by: djcombes on 2010-02-24 20:28:07
Duff was suggesting that there might be a difference in perception below the threshold of conscious perception.
WRT audio, again, based on what evidence exactly?

From his last relevant post (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=56797&view=findpost&p=510877) on this forum, back in August of 2007:
Quote
That experiment is coming...
Guess what, we're still here waiting.
WRT to audio specifically I seem to remember that there was some reference to an imperceptible change in masking noise level having a measurable effect on perception, even though the two noise levels couldn't be distinguished by ABX.

I agree - we are still waiting. In the absence of any information from duff I think it would be reasonable to infer that any such effect is very small and hard to detect, or that there is no such effect. Or that duff was just full of sh*t, and never had any intention of carrying out experiments into the matter.

;-)
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-02-24 20:30:09
Link please.
Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-02-24 21:44:17
Or that duff was just full of sh*t, and never had any intention of carrying out experiments into the matter.
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-02-24 21:51:31
I hope that wasn't an appeal to authority.
Title: What's the problem with double-blind testing?
Post by: djcombes on 2010-02-25 13:41:56
Link please.
There were some (read through the thread), but they're broken now.
Title: What's the problem with double-blind testing?
Post by: djcombes on 2010-02-25 13:47:27
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies

Assistant professor - what does that mean? Is that the lowest rank of the academic ladder in the US? What you'd call a lecturer in the UK? Does this mean that he is busy teaching and doesn't have time for much research these days?
Title: What's the problem with double-blind testing?
Post by: garym on 2010-02-25 22:55:00
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies

Assistant professor - what does that mean? Is that the lowest rank of the academic ladder in the US? What you'd call a lecturer in the UK? Does this mean that he is busy teaching and doesn't have time for much research these days?


Yes, similar to "lecturer" in the UK. In the US the progression is Assistant > Associate > "Full" professor. All depends on disciplines and universities, but at my "shop" the assistants do very little teaching and a lot of research.  We have to offer only one semester a year of teaching to attract the top candidates. But again, this varies greatly by discipline, even within the same university.

edit: and in US, the Asst/Assoc/Full ranks are typically associated with tenure track. We also have lecturer type ranks that are typically non-tenure earning.
Title: What's the problem with double-blind testing?
Post by: Axon on 2010-02-25 23:07:01
It's probably worth noting that the tenure track can be 25+ years long in some universities. I knew a guy who joined a dept as asst/associate professor (I forget which) in the mid 80s, and only got his tenure after I graduated in '02.

It's a really messed up system and I wouldn't belittle any professor who wasn't tenured.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-02-25 23:16:20
I certainly wouldn't belittle an associate professor either, though I do question whether this person should be taken any more seriously than anyone else regarding the subject material simply because he's an Associate Professor in the Department of Communication Studies.
Title: What's the problem with double-blind testing?
Post by: garym on 2010-02-25 23:18:52
It's a really messed up system and I wouldn't belittle any professor who wasn't tenured.


agree, and I certainly wasn't dissing anyone without tenure....just pointing out the title differences and what they typically mean in the U.S.  Heck, 99.99999% of all employees in the world do NOT have tenure.

edit: and yes, professors of any stripe may or may not know more about a subject than other people (even in their purported specialty field).
Title: What's the problem with double-blind testing?
Post by: krabapple on 2010-02-26 02:26:06
Or that duff was just full of sh*t, and never had any intention of carrying out experiments into the matter.
Or that duff is simply too busy with his work as Assistant Professor, Department of Communication Studies


“Duffman is thrusting in the direction of the problem!” *




(*Simpsons quote)
Title: What's the problem with double-blind testing?
Post by: krabapple on 2010-02-26 02:28:06
It's probably worth noting that the tenure track can be 25+ years long in some universities. I knew a guy who joined a dept as asst/associate professor (I forget which) in the mid 80s, and only got his tenure after I graduated in '02.

It's a really messed up system and I wouldn't belittle any professor who wasn't tenured.



Especially if they own a gun, these days.
Title: What's the problem with double-blind testing?
Post by: MLXXX on 2010-02-27 04:11:15
Some people might assume from reading the post below that if they placed a PAL DVD of a Hollywood movie onto a standard PAL DVD player it should play at correct speed and pitch, i.e. the speed and pitch at which the actors spoke, or the orchestral instruments were played.

This cannot be assumed.  The technology for motion interpolation to create a relatively artifact-free 25 video frames per second from a source of 24 video frames per second (the traditional speed at which movies are filmed) has only become available in recent years, and is still not perfect.

As an alternative approach, leaving the video as is [i.e. simply leaving each video frame intact and allowing the 24fps to play at 25fps] but processing the audio to reduce the apparent pitch back to original pitch, is complex and causes audible artifacts.  With a complex source, e.g. an orchestral sound track, it may be recommended to leave the pitch 0.7 semitones sharp.  In any event the tempo is still wrong with this approach, even if the pitch has been 'corrected'.

On a related matter, in Australia any number of sitcoms and movies produced in the USA are currently broadcast on digital TV with a simple 25/24 speedup, causing an increase in pitch of the sound.  Perhaps surprisingly, most people do not notice this speedup of just over 4%.

But in an ABX test with a short gap between listening to a correct speed  version and a version that is 0.7 of a semitone sharp, I am sure these same people (most of them anyway) would notice the set of differences.  [The set of differences includes tempo, timbre, and vowel formant frequencies.] 

This is a very interesting practical example of a difference in sound that typically goes unnoticed in real life; despite being quite a big difference in an ABX comparison.  [If anyone reading doubts how different the sound is, use Audacity to speed up an audio clip by 4% and compare the processed sound with the original clip.]

As a result of the many hours in my childhood spent watching movies on television with a 4% speedup, watching a movie at correct speed at the cinema, on an NTSC DVD, or on a Blu-ray DVD, makes the sound seem unusually solid and real.

I do get annoyed with the 4% speedup when watching TV, or my few PAL DVDs, but I am in the minority.  Most people remain blissfully unaware of it.


I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre).


Just because the FPS change, there's no necessity for the pitch or timing to change. People working with films and video have been managing this situation for several decades.

This article explains how FPS changes are have been managed for decades so that they don't affect the length or running speed of film or video:

http://www.zerocut.com/tech/pulldown.html (http://www.zerocut.com/tech/pulldown.html)

In the old days, the production paths for sound and video were separated, the video or film was altered using pull down techniques, and the audio was added back into the finished product with  no changes. Or, the sound was added back into the finished product with a different FPS from a more origional source.

These days, most video production software manages things like this automagically.  You just specify the FPS of the finished product, and it is produced in accordance with your spec under the covers, as it were.

In short, your example has a rather serious flaw.
Title: What's the problem with double-blind testing?
Post by: 2Bdecided on 2010-03-01 15:23:36
If you've heard the content before, the pitch shift is quite obvious. Even if it's weeks later that you hear it again.

If it's new to you, then far less so - e.g. if you don't know what the actor's voice really sounds like, or the music track really sounds like, then the pitch shift is meaningless to you, and won't be detected.

Cheers,
David.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2010-03-02 11:19:47
I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre).


Just because the FPS change, there's no necessity for the pitch or timing to change. People working with films and video have been managing this situation for several decades.

This article explains how FPS changes are have been managed for decades so that they don't affect the length or running speed of film or video:

http://www.zerocut.com/tech/pulldown.html (http://www.zerocut.com/tech/pulldown.html)

In the old days, the production paths for sound and video were separated, the video or film was altered using pull down techniques, and the audio was added back into the finished product with  no changes. Or, the sound was added back into the finished product with a different FPS from a more origional source.

These days, most video production software manages things like this automagically.  You just specify the FPS of the finished product, and it is produced in accordance with your spec under the covers, as it were.

In short, your example has a rather serious flaw.

It's in fact your example that has a serious flaw, because the method you posted a link to (2:3 pulldown) is for NTSC and not for PAL. All PAL movie DVDs I've encountered so far (old and new) have been speed-up. This means the tempo of the audio always has been speed-up (with or without pitch correction, personally I prefer without). From capturing I know that long ago movies in PAL were broadcoasted with fieldshifting which didn't require to change the audio tempo, but nowadays this technique isn't used anymore, at least not with the movies I have captured.

If you've heard the content before, the pitch shift is quite obvious. Even if it's weeks later that you hear it again.

Personally, I have to disagree with this. My tonal memory is pretty awful even though I'm playing piano and guitar since I was 8.
Title: What's the problem with double-blind testing?
Post by: MLXXX on 2010-03-02 13:55:35
This clip Samples combined - HOUSE MD (http://www.hydrogenaudio.org/forums/index.php?act=attach&type=post&id=5766) contains three samples of the opening music to the TV series HOUSE MD as broadcast in Australia on Digital TV (terrestrial):

1. Correct speed (as broadcast in 2008)
2. With PAL speedup (as broadcast in 2007)
3. Version 1 again

Although the difference is quite marked when the samples are contrasted with each other without delay (e.g. with an immediate ABX test), most people watching 50fps TV do not notice whether a particular TV episode is being broadcast with PAL speedup or not.

Here is a post I made in 2008:

[blockquote]I settled back to enjoy an episode of House-MD tonight, recorded earlier this week with Vista Media Centre.

Recalling that some of the episodes last year were 24p but broadcast at 25fps (50i), I chose to play the episode with ZoomPlayer, which is now permanently configured on my HTPC to use ReClock, and with a fixed set Reclock rate of 24fps.

I immediately noticed the video was not as smooth as it usually is with Reclock running. After a while I noticed that some of the actors had particularly deep voices that didn't seem quite natural. I then switched over to playing the episode in Vista Media Centre and it sounded natural. A quick check with VideoRedo revealed that the episode was interlaced.

It was clear that the episode this week was a correct speed 50i version, and not a sped-up 24p version.[/blockquote]
Similarly, most people do not notice the 25/24 speedup usually present in PAL DVD releases of Hollywood movies.  When I have raised the topic socially of "PAL speedup" with television and with PAL DVDs, not one person has expressed awareness of the issue.  And yet trying to play a PAL DVD and its soundtrack CD together reveals an obvious difference.  The PAL DVD races ahead, and is higher in pitch. This difference is not hard to notice!

If you've heard the content before, the pitch shift is quite obvious. Even if it's weeks later that you hear it again.

Personally, I have to disagree with this. My tonal memory is pretty awful even though I'm playing piano and guitar since I was 8.

With an isolated note,  0.7 of a semitone is too small a difference for long term memory, other than for the very small percentage of people who sense pitch absolutely and can name any musical note by ear ("perfect pitch").

With connected music [i.e. not just a single test tone] some of us can recall how a piece sounded and a 4% difference will be noticed because of its cumulative effect: tempo and timbre.  With the spoken voice, vowel formant frequencies tend to sound too light and high if the playback is 4% fast.

Some people can perceive the unnatural thinness and hurried quality of a 4% speedup without having heard the speech or music previously, and despite not possessing perfect pitch. I happen to fall into that category.

A verification method if you suspect audio has been subjected to 25/24 speedup, is to subject it to 24/25 slowdown (i.e. a 4.000% slowdown).  It will then either sound right, or too low in pitch and too slow.  Another method is to wait for music in the audio, sing a note of the melody and go to a piano.  The note you sing will lie in between two notes, about 0.7 sharp of one note, and 0.3 of a semitone flat with respect to the note above, if the music has been sped up by 25/24, i.e. by 4.166%.

The PAL speedup is frequently used to accommodate the fact that analogue and digital TV systems in PAL countries operate at 25fps or 50fps, rather than at 24fps or 48fps. Film is conventionally captured at 24fps.

In the United States, Japan and other NTSC countries, a 3:2 pulldown process enables 24fps film to run at 23.976fps, from which is derived "59.94fps interlaced" for broadcast, a standard that has been used since the introduction of color in the 1950s.  59.94fps interlaced is more commonly referred to as 60fps or 30fps for simplicity.  [The speed and pitch difference factor of 1000/1001 is practically negligible for human hearing.]
Title: What's the problem with double-blind testing?
Post by: 2Bdecided on 2010-03-02 14:05:18
With the spoken voice, vowel formant frequencies tend to sound too light and high if the playback is 4% fast.
Yes - or more simply, it sounds like a different person talking - sometimes a younger person - sometimes just plain wrong.

Quote
A verification method if you suspect audio has been subjected to 25/24 speedup, is to subject it to 24/25 slowdown (i.e. a 4.000% slowdown).  It will then either sound right, or too low in pitch and too slow.
True - because we're supposedly more sensitive to "slightly too slow/low" than we are to "slightly too fast/high".

Quote
Another method is to wait for music in the audio, sing a note of the melody and go to a piano.  The note you sing will lie in between the notes, about 0.7 sharp of one note, and 0.3 of a semitone flat the semitone above, if the music has been sped up by 25/24, i.e.  by 4.166%.
I don't think many people can sing this well!  Other things get in the way with this comparison though - instruments are not guaranteed to be "in tune" (your piano, and/or the ones on the recording) - music is not guaranteed to be played into the soundtrack at the correct speed. Also different releases of the same music (especially older music) sometimes have slightly different speed.

Cheers,
David.
Title: What's the problem with double-blind testing?
Post by: MLXXX on 2010-03-02 14:22:22
Other things get in the way with this comparison though - instruments are not guaranteed to be "in tune" (your piano, and/or the ones on the recording) - music is not guaranteed to be played into the soundtrack at the correct speed. Also different releases of the same music (especially older music) sometimes have slightly different speed.

True. But it is a very quick and useful method if listening to a tv series or movie produced in say the last 20 years, and despite the slighly different pitch for concert A used by some European orchestras. [My piano gets tuned once a year and drifts very little; nothing like 0.3 of a semitone. And many people these days have electronic keybords.]

Another method is to play along with a piano and try to find a matching note. A 0.3 of a semitone discrepancy will frustrate the attempt. 

Also some pieces of music are usually played in a particular key.  If the TV broadcast is about three-quarters of a semitone sharp of the usual key, this will be a strong indicator.

Unforunately PAL DVDs of movies include no warning about the speedup.  It is presumably considered too minor a matter to comment on ...
Title: What's the problem with double-blind testing?
Post by: probedb on 2010-03-02 17:26:34
I generally don't notice PAL speedup at all. The only times I notice it are when watching favourite TV episodes that I have been used to in PAL then getting the NTSC version.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2010-03-03 08:12:32
For TV shows it depends, because often they are shot in 30i and then converted via field blending which doesn't affect the audio.
Title: What's the problem with double-blind testing?
Post by: duff on 2010-03-04 06:29:25
Hi everybody -

Thanks to Kees for letting me know this thread came back to life - though I didn't know that my academic job was under discussion. Hah. Needless to say, I have been doing other stuff. My brother ("bryant" the developer of WavPack (http://www.wavpack.com)) and I still plan on doing the set of experiments we talked about here 3 years ago. Apparently, we like to think about these things for awhile! Truth is, we are just busy. I'm quite active in research but that work is not on psychoacoustic testing methods and auditory processing.

In the experiments we are devising, all we intend to demonstrate is that decision-based paradigms such as ABX might suggest perceptual equality between different stimuli at the level of conscious awareness, while implicit measures such as reaction time tasks might reveal consequential processing differences in those same stimuli. Additionally, long term exposures and subsequent judgments might reveal preference differences between stimuli that are also not perceptually different using ABX. If this is the case (and I suspect it very well could be), then short term and/or long-term processing differences might be at the root of some people's apparent discomfort associated with lossy codecs, and even 44.1/16 bit digitization. I have already explained the ideas quite thoroughly before, so check out the early pages of this thread. But my links were dead. I copied one post from Nov 2005 and fixed the links (I just learned you can't edit after 1 hour now!), so now those papers are available if you'd like to read them.

Keep in mind that this is not some ultimate rejection of ABX, and admittedly the effect (if there) will be small. But a small effect is all you need to provide some explanation for various audiophile claims. It's just an attempt at providing experimental support for the idea that conscious decision processes that ABX paradigms rely on might not always be the last word in auditory perception. A similar effect happens in visual perception. I explained the flicker rate studies (http://www.gregbryant.org/ha/flicker_rateCRT.pdf) before. Same idea: subjects can't detect the difference in flicker rates in two CRT presentation conditions, but in one condition the rate disrupted eye movements and caused subjects to re-read many characters. No awareness of the experimental manipulation, significant difference in reaction time tests, and we know flicker rates have measurable effects on people's comfort. The relevant information is below threshold, but has an impact that is only measurable using certain tasks.

Someone mentioned doing brain scans - our expectation would be that given the right measurement (tall order given the sophistication of such technology even now), one might find neural differences. As I described earlier, there are neural correlates to distinct sensory and decision processes. I think it is quite plausible that greater neural activity could be associated with processing compressed audio relative to uncompressed counterparts, and ABX decision tasks would not reflect it. Just saying it's possible. And if so, rather than calling all audiophile stupid nutcases, we would have at least one explanation that could help resolve this long standing debate.


I have compiled a number of articles that provide anybody here resources and empirical support for the various claims David and I have been making. I claimed that there is a distinction between sensory and decision processes. This recent Nature article (http://www.gregbryant.org/ha/decisionsensory.pdf) provides empirical support for this. Cognitive scientists have understood this distinction for decades, but here is recent evidence demonstrating the neurological basis for it. This is a fundamental issue in our criticism of the validity of ABX testing as the last word on ultimate differences in the auditory processing of lossy audio. Acoustic differences that could matter for a listener’s overall experience might not be audible.

Here are two articles that show the difference between decision processes linked to the discrimination of stimuli versus sensory processes that are affected when there is no discrimination. There is clear enhanced brain activity that occurs during the presentation of noisier stimuli even when it cannot be distinguished from a less noisy counterpart.

Effects of Low-Pass Noise Masking on Auditory Event-Related Potentials to Speech (http://www.gregbryant.org/ha/noisemasking2.pdf)
The Effects of Decreased Audibility Produced by High-Pass Noise (http://www.gregbryant.org/ha/noisemasking3.pdf)

Woodinside claimed that no filtering processes happen after the ear. This is definitely incorrect, as filtering processes do happen in the auditory cortex (http://www.gregbryant.org/ha/spatialauditory.pdf) (one of the many analogues between auditory and visual processing), but I’ll assume he meant that the relevant filtering for audio codecs involves processes that happen only in the ear. This is fine, as there’s no reason to distinguish peripheral (inner and outer ear) processes from central processes (auditory cortex) when talking about metabolic costs of neural effort. The neurons must work harder (wherever this happens) to resolve the signal and encode it for later processing. And this does not deal with the likely possibility that the resulting representations might differ depending on stimulus quality (not a necessary feature of our argument).

There has been quite a bit of skepticism regarding the claim that compressed or degraded stimuli require more effort to process. I could cite a list of papers a mile long showing reaction time increases as a function of stimulus complexity (which heavily imply that processing difficulty increases), but there is also work showing that neurons respond to increased attention demands (http://www.gregbryant.org/ha/attentionneural.pdf). Additionally, there seem to be specialized neural systems for resolving degraded signals (http://www.gregbryant.org/ha/learningdegraded.pdf), which in turn contribute to learning. These processes require effort, and have associated metabolic costs. This is exactly the sort of thing we have been arguing.



Title: What's the problem with double-blind testing?
Post by: MLXXX on 2010-03-04 09:11:28
I think it is quite plausible that greater neural activity could be associated with processing compressed audio relative to uncompressed counterparts, and ABX decision tasks would not reflect it. Just saying it's possible.

Consider also that the reverse might apply: lesser neural activity.

I find if listening to conventional Dolby 5.1 from a DVD that the sound is "neater" than an uncompressed version.  The compression removes less important sounds (potential distractions) and may thus help guide human hearing towards perception of the "significant and relevant" audio content.
Title: What's the problem with double-blind testing?
Post by: duff on 2010-03-05 08:35:29
I think it is quite plausible that greater neural activity could be associated with processing compressed audio relative to uncompressed counterparts, and ABX decision tasks would not reflect it. Just saying it's possible.

Consider also that the reverse might apply: lesser neural activity.

I find if listening to conventional Dolby 5.1 from a DVD that the sound is "neater" than an uncompressed version.  The compression removes less important sounds (potential distractions) and may thus help guide human hearing towards perception of the "significant and relevant" audio content.


You never know, but... the Dolby 5.1 differences are above threshold -- we're talking about differences you can't necessarily hear. Still, I agree that it's unclear what the relationship is between subjective sound judgment and brain activity.

Title: What's the problem with double-blind testing?
Post by: greynol on 2010-03-05 09:28:21
Assuming there actually is one?
Title: What's the problem with double-blind testing?
Post by: Kees de Visser on 2010-03-05 11:35:48
Assuming there actually is one?
For simplicity let's assume there is a brain
What puzzles me is that training is an essential aspect of perceptual DBT (audio, wine etc.) and can make a difference in the results. An athlete can train his/her body for (visibly) better performance, but what makes a trained ear ? We probably can't train to hear things beyond the physical limitations of our ear. I've always assumed that the trained ear has to do with brain activity. What else can it be ?
Title: What's the problem with double-blind testing?
Post by: Arnold B. Krueger on 2010-03-05 13:54:48
Assuming there actually is one?
For simplicity let's assume there is a brain
What puzzles me is that training is an essential aspect of perceptual DBT (audio, wine etc.) and can make a difference in the results. An athlete can train his/her body for (visibly) better performance, but what makes a trained ear ? We probably can't train to hear things beyond the physical limitations of our ear. I've always assumed that the trained ear has to do with brain activity. What else can it be ?


IME ear training is a misnomer. What needs training is the brain.
Title: What's the problem with double-blind testing?
Post by: Arnold B. Krueger on 2010-03-05 14:19:38
I finally found an example : in Europe, when we watch a movie in the theatre, then several monthes later on DVD, the sound track is speeded up, because the movie runs at 25 fps (video PAL) instead of 24 fps (theatre).


Just because the FPS change, there's no necessity for the pitch or timing to change. People working with films and video have been managing this situation for several decades.

This article explains how FPS changes are have been managed for decades so that they don't affect the length or running speed of film or video:

http://www.zerocut.com/tech/pulldown.html (http://www.zerocut.com/tech/pulldown.html)

In the old days, the production paths for sound and video were separated, the video or film was altered using pull down techniques, and the audio was added back into the finished product with  no changes. Or, the sound was added back into the finished product with a different FPS from a more origional source.

These days, most video production software manages things like this automagically.  You just specify the FPS of the finished product, and it is produced in accordance with your spec under the covers, as it were.

In short, your example has a rather serious flaw.

It's in fact your example that has a serious flaw, because the method you posted a link to (2:3 pulldown) is for NTSC and not for PAL. All PAL movie DVDs I've encountered so far (old and new) have been speed-up. This means the tempo of the audio always has been speed-up (with or without pitch correction, personally I prefer without). From capturing I know that long ago movies in PAL were broadcoasted with fieldshifting which didn't require to change the audio tempo, but nowadays this technique isn't used anymore, at least not with the movies I have captured.


You've missed my point. My point is that in 2010 there is no need to accept a pitch shift in the accompanying audio when changing the frame rate of A/V media. The general problem has been around for decades, and the means for solving it have only become more sophisticated, economical and undetectable over the years.  You have fastened your attention on a particular implementation (2:3 pulldown) of a particular technique (pulldown) and mistakenly invalidated the general comment because of some variations in impementations of that technique. There are also other techniques.

If perchance a piece of video actually has  its running time changed, there is still no need to change the pitch of the accompanying audio. For decades there have been means for changing the running time of audio without changing its pitch. Currently, this is a standard feature of a number of reasonably-priced audio editing programs. I personally do this sort of thing every once in a while.

So, if you are getting PAL media whose pitch is different from the corresponding NTSC media, the core problem is that the production people did not avail themselves of any number of different approaches that are available for a reasonable cost and in general use, at least by people who have a clue and have the resources to exercise that clue.

Furthermore if this problem really bugs you, you could fix it for yourself because the tools for doing so aren't all that expensive or hard to use.

This seems like a good time to shoot or at least beat up the messenger - the messenger being the video media distributor who is foisting these A/V kludges off on you. He's taking your money and feeding you crap. Tell him that you are madder than %$## and won't take it any more! ;-)
Title: What's the problem with double-blind testing?
Post by: Arnold B. Krueger on 2010-03-05 14:25:51
I think it is quite plausible that greater neural activity could be associated with processing compressed audio relative to uncompressed counterparts, and ABX decision tasks would not reflect it. Just saying it's possible.

Consider also that the reverse might apply: lesser neural activity.

I find if listening to conventional Dolby 5.1 from a DVD that the sound is "neater" than an uncompressed version.  The compression removes less important sounds (potential distractions) and may thus help guide human hearing towards perception of the "significant and relevant" audio content.


You never know, but... the Dolby 5.1 differences are above threshold -- we're talking about differences you can't necessarily hear. Still, I agree that it's unclear what the relationship is between subjective sound judgment and brain activity.


Right, Dolby 5.1 can cause audible degradation all by itself. It is not as sonically accurate as 48 KHz 16 bit PCM.

Dolby digital is itself a perceptual coding technique.  It is not necessarily even a outstandingly good perceptual coding technique. It is reliatively old, but apparently the coders in general use have been upgraded somewhat over the years. There are blind tests that show that audible degradation due to it can be heard, at least in some cases.

Cascading different percpetual coding techniques can lead to unexpectedly bad sound quality losses.
Title: What's the problem with double-blind testing?
Post by: 2Bdecided on 2010-03-05 15:17:26
So, if you are getting PAL media whose pitch is different from the corresponding NTSC media, the core problem is that the production people did not avail themselves of any number of different approaches that are available for a reasonable cost and in general use, at least by people who have a clue and have the resources to exercise that clue.
No, most mastering houses have at least one implementation - some have several.

It's just that sometimes none of them are judged to be good enough.

Also, for cheaper releases, a simple resample (changing time and pitch) is a known quantity - whereas an algorithm which changes time but maintains pitch may introduce content-specific artefacts. If there's no time to check this carefully, it's better to introduce a known flaw (wrong pitch), rather than introduce a potential serious unknown flaw (e.g. mangling the sound of some strings).

It's a bit like noise reduction - it is possible to remove hiss from older recordings - but sometimes it's better not to, given what's available. Just because someone chooses not to doesn't mean that they can't - not doing something can be a sign of great care, rather than great incompetence.


Anyway, many PC-based home theatre enthusiasts are thankful for 25p masters with a speed up - they can easily slow them down again and get the original sound - something which is impossible with pitch "corrected" versions!

Doesn't matter with BluRay - most are 24p even in "PAL" countries.

Cheers,
David.
Title: What's the problem with double-blind testing?
Post by: 2Bdecided on 2010-03-05 15:38:25
@duff,

The correlate of the flicker example you give would be finding a task which was impaired by lossy coding, despite the lossy coding being inaudible in an ABX test. e.g. "write down the words that are spoken", or "identify the instrument", or "sing the tune" etc.

I'm not convinced by the flicker example anyway - whether you see a difference between different flicker rates depends on whether the flickering device is in the centre or periphery of your field of view, whether your eyes are moving or stationary, and whether your eyes are following something (eye-tracked motion), or moving across something that isn't itself moving.

Hence if you can't ABX a difference when simply staring at a screen, but can perceive a difference when reading from that screen, it doesn't necessarily tell you anything about ABX vs reading - because you've introduced another variable (eye movement) which you would expect to interact with the flicker anyway.


Not sure about the ba/da examples you posted. I get the point, but it could be more complex than is suggested. After all, no one is claiming that the subjects can't consciously differentiate between the other variable in the test - the LPF of the noise - though I can't think why this should matter.

There are (possibly) four categories:
1. differences we can short-term ABX
2. differences we cannot short-term ABX directly, but which yield different results in some task
3. differences we cannot short-term ABX directly, which do not yield different results in any known task, but which reveal consistent differences in some kind of brain respond
4. differences which are undetectable on all three counts

The thing with (2) is that the task itself becomes the ABX test - you merely have to ask the correct question.
The thing with (3) is that, if there really is no question or task or anything else that can reveal any kind of conscious knowledge of the difference, over any time scale, why would any one care when listening to music? (It's nice research, but...!)
Obviously (4) self evidently doesn't matter.

Cheers,
David.
Title: What's the problem with double-blind testing?
Post by: greynol on 2010-03-05 16:47:53
For simplicity let's assume there is a brain

Right about now I'm wondering where mine was when I posted that.
Title: What's the problem with double-blind testing?
Post by: duff on 2010-03-05 20:34:20
@duff,

The correlate of the flicker example you give would be finding a task which was impaired by lossy coding, despite the lossy coding being inaudible in an ABX test. e.g. "write down the words that are spoken", or "identify the instrument", or "sing the tune" etc.

I'm not convinced by the flicker example anyway - whether you see a difference between different flicker rates depends on whether the flickering device is in the centre or periphery of your field of view, whether your eyes are moving or stationary, and whether your eyes are following something (eye-tracked motion), or moving across something that isn't itself moving.

Hence if you can't ABX a difference when simply staring at a screen, but can perceive a difference when reading from that screen, it doesn't necessarily tell you anything about ABX vs reading - because you've introduced another variable (eye movement) which you would expect to interact with the flicker anyway.

There are (possibly) four categories:
1. differences we can short-term ABX
2. differences we cannot short-term ABX directly, but which yield different results in some task
3. differences we cannot short-term ABX directly, which do not yield different results in any known task, but which reveal consistent differences in some kind of brain respond
4. differences which are undetectable on all three counts

The thing with (2) is that the task itself becomes the ABX test - you merely have to ask the correct question.
The thing with (3) is that, if there really is no question or task or anything else that can reveal any kind of conscious knowledge of the difference, over any time scale, why would any one care when listening to music? (It's nice research, but...!)
Obviously (4) self evidently doesn't matter.

Cheers,
David.


The point of the flicker example is just that a reaction time test (and analysis of eye movements) revealed processing differences that did not apparently affect conscious awareness - but conscious decision processes are what ABX tests rely on. Subjects' eyes were moving because they were reading - but regardless, the idea is just that they didn't notice differences even though there were measurable differences in their behavior - and those measurable differences can have long-term impacts such as increased fatigue and discomfort that short term tests won't capture.

Nevertheless, it is only analogous, and they didn't do the ABX test. It's just an example to give you an idea of how processing can have consequences that are only measurable through implicit means. The correlate of the flicker example you suggest is right along the lines of what we would like to try. A positive result in something like that would be an issue for those who claim ABX testing is the end all.

As for your categories, we are after #2. But I wouldn't quite say the task "becomes the ABX test" unless you mean the task becomes the means by which we can uncover any kind of processing difference (which is different from perceptual difference). The limitation of ABX in this sense is that it's reliant on conscious reports. ABX might fail to reveal a processing difference - and any processing differences might be contributing to various phenomena people associate with lossy codecs and 44.1/16 bit digitization.
Title: What's the problem with double-blind testing?
Post by: stephanV on 2010-03-05 23:28:48
You've missed my point. My point is that in 2010 there is no need to accept a pitch shift in the accompanying audio when changing the frame rate of A/V media. The general problem has been around for decades, and the means for solving it have only become more sophisticated, economical and undetectable over the years.  You have fastened your attention on a particular implementation (2:3 pulldown) of a particular technique (pulldown) and mistakenly invalidated the general comment because of some variations in impementations of that technique. There are also other techniques.

I'd rather accept a pitch shift than a ratty pitch correction, and this still does not solve tempo.

Quote
If perchance a piece of video actually has  its running time changed, there is still no need to change the pitch of the accompanying audio. For decades there have been means for changing the running time of audio without changing its pitch. Currently, this is a standard feature of a number of reasonably-priced audio editing programs. I personally do this sort of thing every once in a while.

So, if you are getting PAL media whose pitch is different from the corresponding NTSC media, the core problem is that the production people did not avail themselves of any number of different approaches that are available for a reasonable cost and in general use, at least by people who have a clue and have the resources to exercise that clue.

Sorry, but it just seems you are clueless about the PAL world. Sped-up DVDs are not an anomaly at all, they are the standard.
Title: What's the problem with double-blind testing?
Post by: MLXXX on 2010-03-06 00:07:36
This seems like a good time to shoot or at least beat up the messenger - the messenger being the video media distributor who is foisting these A/V kludges off on you. He's taking your money and feeding you crap. Tell him that you are madder than %$## and won't take it any more! ;-)

Unfortunately, it is standard industry practice to release PAL DVDs with a simple 25/24 speedup.  It is also standard practice in PAL countries to broadcast TV series and movies with 25/24 speedup.

For example the lush orchestral sound accompanying Harry Potter movies will simply be broadcast 0.7 semitone sharp, 4.16% fast; and the timbre of the instruments and "gravitas" of the performance are significantly affected.  Around 98% of the population will not notice.  But an immediate ABX test will reveal the marked differences.

If there's no time to check this carefully, it's better to introduce a known flaw (wrong pitch), rather than introduce a potential serious unknown flaw (e.g. mangling the sound of some strings).

It's a bit like noise reduction - it is possible to remove hiss from older recordings - but sometimes it's better not to, given what's available. Just because someone chooses not to doesn't mean that they can't - not doing something can be a sign of great care, rather than great incompetence.
Indeed.

Quote
Anyway, many PC-based home theatre enthusiasts are thankful for 25p masters with a speed up - they can easily slow them down again and get the original sound - something which is impossible with pitch "corrected" versions!

Doesn't matter with BluRay - most are 24p even in "PAL" countries.
Yes.  I can and do use Reclock to play back selected TV recordings I capture in Australia, and the occasional PAL DVDs, at correct speed, so am thankful that no attempt has been made to "correct the pitch", as I would then have to perhaps "re-correct the pitch".  Pitch correction without a corresponding speed change is a complex and imperfect exercise.  You may be able to get away with it when editing the sound of a solo instrument, or some dialogue, but it will compromise the sound of an orchestra.  In fact I prefer the sound to remain 0.7 of a semitone sharp, than to be "pitch corrected".
Title: What's the problem with double-blind testing?
Post by: MLXXX on 2010-03-06 01:16:56
Dolby digital is itself a perceptual coding technique.  It is not necessarily even a outstandingly good perceptual coding technique. It is reliatively old, but apparently the coders in general use have been upgraded somewhat over the years. There are blind tests that show that audible degradation due to it can be heard, at least in some cases.

I notice differences when comparing sound tracks on HD-DVD and Blu-ray discs [adjusted to match levels - though this matching can be quite difficult as lossy codecs alter the waveforms so drastically].  The difference between lossless and high bitrate lossy (e.g. DD+ 1536Kbps) is extremely subtle and I find with my hearing that I have to listen very intently in an ABX comparison.  There is very little in it.

It is interesting how well 448kbps Dolby 5.1 as found on an NTSC DVD discs performs, if not compared with another source.  It is quite pleasant, if a little bland.  Without an immediate ABX test available, many listeners would not notice the audible degradation relative to a lossless, or high definition lossy, version.