HydrogenAudio

Hydrogenaudio Forum => General Audio => Topic started by: Yahzi on 2012-12-24 16:34:30

Title: What we measure is what we hear
Post by: Yahzi on 2012-12-24 16:34:30
I've heard audiophiles claim time and time again that what we measure does not reflect what we hear in a listening evaluation. I think Floyd Tool is the pioneer in conducting the studies for this, but who else as in a position of authority has led research into this field? It can't be mere opinion. I feel measurements largely do correlate with subjective sound quality but I don't think *all* measurements do. Some are more relevant than others.

What is your position?
Title: What we measure is what we hear
Post by: slks on 2012-12-24 16:56:45
It depends on which measurements we're talking about. A volume change of 6 dB is certainly noticeable, while increasing top-end frequency response from 20 to 30 khz is almost certainly not noticeable.

I don't know of any prominent researchers in this area, but it's fairly simple to do your own studies with the wide variety of audio tools out there, and an ABX test. The only thing with doing it yourself is that you need to make sure you use the proper methodology, or your results will be invalid.
Title: What we measure is what we hear
Post by: Yahzi on 2012-12-24 17:22:48
So you are saying that measurements can explain what we hear assuming you are using the correct measurements, but not all measurements are appropriate for the task so it depends what exactly you are measuring for. Am I understanding this correctly?
Title: What we measure is what we hear
Post by: Ethan Winer on 2012-12-24 18:23:25
^^^ Yes, you understand correctly. I've done a lot of such testing using commonly available audio tools, and several articles on my web site explain the method and offer files you can download to assess audibility for yourself:

Converter Loop-Back Tests (http://www.ethanwiner.com/loop-back.htm)
Artifact Audibility Report (http://www.ethanwiner.com/audibility.html)
The Truth About Record Levels (http://www.ethanwiner.com/levels.htm)
Hearing Below the Noise Floor (http://www.ethanwiner.com/audibility.html#part2)
Dither Report (http://www.ethanwiner.com/dither.html)
Converter Comparison (http://www.ethanwiner.com/converters.html)
Perception - the Final Frontier (http://www.ethanwiner.com/perception.htm)

--Ethan
Title: What we measure is what we hear
Post by: AndyH-ha on 2012-12-24 20:59:25
To sum of one aspect of "what we measure is NOT what we hear"
Some aspects that can be measured, such as distortion, noise level, and frequency response, make a very significant contribution to what we hear -- up to a point. Beyond that point, improvements in the equipment of an order of magnitude, or two orders of magnitude, might be possible and measurable, but those improvements make no difference in what we hear. They are beyond human abilities to differentiate. Also, if the same level of improvement is not maintained throughout the signal chain, from the source to the output transducers and listening environment, they can be readily swamped, washed out, by those parts of the signal chain that can't meet the same standards.
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2012-12-24 22:40:32
I've heard audiophiles claim time and time again that what we measure does not reflect what we hear in a listening evaluation. I think Floyd Tool is the pioneer in conducting the studies for this, but who else as in a position of authority has led research into this field? It can't be mere opinion. I feel measurements largely do correlate with subjective sound quality but I don't think *all* measurements do. Some are more relevant than others.

What is your position?


My position is that if there is an audible change in an audio signal then there is a readily measurable change in that signal. But, just because there is a measurable change, there is no guarantee of an audible change. This is because modern audio measurements are far more sensitive than the human ear.

The idea that there are audible changes for which there is no corresponding measurable change has been tested carefully for over 30 years and it has been found that the idea that this is so is a result of people not doing proper bias controlled listening tests.

IOW, human bias can lead to the perception of an audible change for which there is no physical explanation.
Title: What we measure is what we hear
Post by: Porcus on 2012-12-24 22:56:40
Two totally different points:

(1) Those annoying spectrogram-comparisons of a lossy version against the original. Easy to quantify huge differences which are not heard, or at least not heard anywhere near what the measurements indicate. On the other hand, could hide audible artifacts.

What you measure is not what you hear – unless you find a much smarter algorithm. And such one has already been applied:

(2) Training a psy model. Correlating to actual listening tests. Predicting what actual humans hear. Actually implement it in a lossy codec. It measures what we actually hear; we hear something which sounds quite a bit like [lossy version].

The “problem” about the latter one, is that it does not directly return to you any “difference is this big” measure. That is not what it is designed for; it is designed to return the lossy signal.
Title: What we measure is what we hear
Post by: Woodinville on 2012-12-25 02:52:47
Many people have worked on measuring what you hear. No perceptual coder (MP3, AAC, AC3, etc) would work if many people had not only done that, but also done a good job of it.

Floyd has done some very good work on preference, which is a different (but good) thing. Work on this goes all the way back to the 1920's, articulation, critical bands, etc, of course.

Look at www.aes.org/sections/pnw/ppt.htm for the "heyser lecture" slide deck for some discussion on that subject.
Title: What we measure is what we hear
Post by: jkauff on 2012-12-29 14:18:22
Many people have worked on measuring what you hear. No perceptual coder (MP3, AAC, AC3, etc) would work if many people had not only done that, but also done a good job of it.

My own experience makes me wonder how well human hearing has been measured. I'm sure average human hearing has been well-studied, but how many outliers are there in the world population? My daughter, for example, can hear the high frequency tones used by cat-repelling devices. She's demonstrated this many times, but my wife and I can't hear them at all (nor can most people). I wonder if the hearing range of various human populations has been well-studied (such as African tribes with certain members who can hear low frequency tones emitted by elephants).

Is it fair to say that this good work has been done primarily based on averages of 1st World populations?
Title: What we measure is what we hear
Post by: dhromed on 2012-12-29 14:27:57
Your daughter is not an unmeasured outlier; she's just young.
Title: What we measure is what we hear
Post by: greynol on 2012-12-29 14:30:06
I doubt she is as well. What frequency is the cat repelling device?
Title: What we measure is what we hear
Post by: garym on 2012-12-29 15:14:02
Your daughter is not an unmeasured outlier; she's just young.


Yes of course. Recall the "ring tones" that only kids in school could hear but none of the teachers could hear it. More here:

http://www.nytimes.com/2006/06/12/technology/12ring.html (http://www.nytimes.com/2006/06/12/technology/12ring.html)
Title: What we measure is what we hear
Post by: greynol on 2012-12-29 18:21:47
We hear about people in their early 20s and younger who can hear up to 22kHz every once in a while.

Pure tone testing doesn't really tell the story, however.
Title: What we measure is what we hear
Post by: sld on 2012-12-29 18:27:56
Your daughter is not an unmeasured outlier; she's just young.


Yes of course. Recall the "ring tones" that only kids in school could hear but none of the teachers could hear it. More here:

http://www.nytimes.com/2006/06/12/technology/12ring.html (http://www.nytimes.com/2006/06/12/technology/12ring.html)

Or the udial.wav that used to circulate around here to test for resampling aliasing...
Title: What we measure is what we hear
Post by: Mach-X on 2012-12-29 19:55:43
Ouch.
Title: What we measure is what we hear
Post by: Yahzi on 2012-12-30 13:07:01
Correct me if I'm wrong but we hear using our brains, not our ears?
Title: What we measure is what we hear
Post by: Soap on 2012-12-30 14:05:30
Correct me if I'm wrong but we hear using our brains, not our ears?


Your point? 

If you're addressing one of the dozens of comments how about quoting the relevant one for some context?

Your brain can't hear more than what the ears are physically capable of processing, and demonstrably hears less.
Title: What we measure is what we hear
Post by: Yahzi on 2012-12-30 15:39:45
I just thought I'd ask the question and I thought instead of starting a new thread I might as well ask in this one. Besides the "we can't measure what we can hear" arguments there are also arguments that our ears are instruments and are the only things that we use in a listening evaluation. Hence the question I asked - "we hear using our brains, not our ears?" Given all the systematic biases that plague human hearing, sight affecting sound .. mood, just thought I might as well get reassurance on that point.



Title: What we measure is what we hear
Post by: Speedskater on 2012-12-30 16:52:02
If you restrict the use of the words "hear" or "see" to only the processes inside the brains of humans and animals, you have eliminated about 99.99 % of the common usage of these words.
Title: What we measure is what we hear
Post by: jkauff on 2012-12-30 22:05:21
I doubt she is as well. What frequency is the cat repelling device?

I don't know. Devices differ--some use ultrasound, some don't. Cat hearing range, however, is 45-64,000 Hz.
Title: What we measure is what we hear
Post by: Soap on 2012-12-30 22:49:42
Cat hearing range, however, is 45-64,000 Hz.


Is it fair to say that this good work has been done primarily based on averages of 1st world cat populations?
Title: What we measure is what we hear
Post by: pisymbol on 2012-12-31 01:05:29
I've heard audiophiles claim time and time again that what we measure does not reflect what we hear in a listening evaluation. I think Floyd Tool is the pioneer in conducting the studies for this, but who else as in a position of authority has led research into this field? It can't be mere opinion. I feel measurements largely do correlate with subjective sound quality but I don't think *all* measurements do. Some are more relevant than others.

What is your position?


I am more of a lurker on this board but I would like to contribute the following:

The problem with measurements is well the metrics! Performance is only relevant when you define your metrics. And here in lies the rub:

Without sounding too coy is that as far as I am aware of there are no measurements that describe "hearing" or rather more ambiguously "sound in which your brain interprets it." So truth be told, the audiophile is partially correct, citing THD, FR, spectrum analysis, etc. etc. all do not tell me a damn on how I actually perceive. They sure do give some obvious indicators and are invaluable tools to determine audibility in the first place (and what isn't audible but placebo). This is why many of the adaptive predictive models had to be fine tuned and then *listener* verified via sophisticated DBTs (e.g. MP3).

But frankly, audiophiles reviews and impressions still contain intrinsic value even if most of it "can't be trusted" since they are interpretations on how they "hear" which as far as I know is how everyone on this thread hears too...
Title: What we measure is what we hear
Post by: AndyH-ha on 2012-12-31 01:52:34
But frankly, audiophiles reviews and impressions still contain intrinsic value even if most of it "can't be trusted" since they are interpretations on how they "hear" which as far as I know is how everyone on this thread hears too...

Not really, perhaps depending somewhat on what you mean by "hear." These are often interpretations of what their expectations and beliefs create inside their heads. They very often do not "hear" the same thing if they don't know  which piece of equipment they are listening to and they will "hear" what they think they are listening to even when they are actually listening to something else.
Title: What we measure is what we hear
Post by: pisymbol on 2012-12-31 02:12:40
But frankly, audiophiles reviews and impressions still contain intrinsic value even if most of it "can't be trusted" since they are interpretations on how they "hear" which as far as I know is how everyone on this thread hears too...

Not really, perhaps depending somewhat on what you mean by "hear." These are often interpretations of what their expectations and beliefs create inside their heads. They very often do not "hear" the same thing if they don't know  which piece of equipment they are listening to and they will "hear" what they think they are listening to even when they are actually listening to something else.


Subjectivity can still provide valuable data points (especially negative impressions).
Title: What we measure is what we hear
Post by: db1989 on 2012-12-31 04:25:53
Subjectivity can still provide valuable data points (especially negative impressions).
Please explain how subjective impressions, negative or positive, are of any use to people who are interested in valid, i.e. objective data about perception.
Title: What we measure is what we hear
Post by: AndyH-ha on 2012-12-31 06:38:57
Subjective impressions are, after all, impressions relative to the subject, most generally the particular subject only, not relative to the data or the general population.

They might tell us something about the subject, but not much about the object. In some case, subjects even hear what they expect to hear when there is no signal (data/sound) at all. That alone implies the possibility that what they report hearing in the vague terms used in such reports do not relate to anything at all coming from the traducer (speaker/headphone/etc.)
Title: What we measure is what we hear
Post by: Woodinville on 2012-12-31 08:43:48
Subjective impressions are, after all, impressions relative to the subject, most generally the particular subject only, not relative to the data or the general population.

They might tell us something about the subject, but not much about the object. In some case, subjects even hear what they expect to hear when there is no signal (data/sound) at all. That alone implies the possibility that what they report hearing in the vague terms used in such reports do not relate to anything at all coming from the traducer (speaker/headphone/etc.)


A properly designed and operated subjective test IS a form of measurement.

Audiophile tests generally (almost always) need not apply, of course.

But, a subjective test, run as a DBT, using a proper design, with anchors and controls, is a measurement.  A measurement that is a true pain in the behind to do, perhaps, but it IS a measurement, and it most often even gives a useful noise level for the measurement.
Title: What we measure is what we hear
Post by: greynol on 2012-12-31 10:59:07
TL;DR:
MUSHRA and other ABC-style tests are subjective tests.  This is because the person taking the test assigns a score to each stimuli in order to declare a personal preference.
Title: What we measure is what we hear
Post by: Satellite_6 on 2012-12-31 14:32:17
So far the best measuring headphones I have tried sound the best and the best measuring amps sound the best so there is no contradiction between what sounds good and what measures well to me. It makes it easy.
Title: What we measure is what we hear
Post by: Martel on 2012-12-31 15:15:21
What's the "best measuring" frequency response of a headphone? I wonder...
Title: What we measure is what we hear
Post by: Nessuno on 2012-12-31 15:18:42
So far the best measuring headphones I have tried sound the best and the best measuring amps sound the best so there is no contradiction between what sounds good and what measures well to me. It makes it easy.

AKA "expectation bias"... it rarely misses a shot!

P.S. What you said is not wrong in principle, but fact is that human ear is not linear neither has infinite precision...
Title: What we measure is what we hear
Post by: Soap on 2012-12-31 15:42:41
What's the "best measuring" frequency response of a headphone? I wonder...


The one which best matches your ear shape, of course!


Title: What we measure is what we hear
Post by: Kees de Visser on 2012-12-31 16:08:48
A properly designed and operated subjective test IS a form of measurement.
I thought the food industry is doing this all the time:
http://www.ift.org/ (http://www.ift.org/)
Quote
Sensory evaluation has been defined as "a scientific discipline used to evoke, measure, analyze, and interpret reactions to those characteristics of foods and materials as they are perceived by the senses of sight, smell, taste, touch, and hearing" (IFT, 1975).

Perhaps we should be less scared to be subjective. All humans "suffer" from it. The real art is in finding out which variables are responsible.

Happy new (multisensory) year !
Title: What we measure is what we hear
Post by: greynol on 2012-12-31 16:43:36
I don't think this forum is scared of subjective opinions; we simply believe that expectation bias should not have a role at the table when it comes to gauging and expressing them.
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2012-12-31 17:14:06
Many people have worked on measuring what you hear. No perceptual coder (MP3, AAC, AC3, etc) would work if many people had not only done that, but also done a good job of it.

My own experience makes me wonder how well human hearing has been measured. I'm sure average human hearing has been well-studied, but how many outliers are there in the world population? My daughter, for example, can hear the high frequency tones used by cat-repelling devices. She's demonstrated this many times, but my wife and I can't hear them at all (nor can most people).


Since you haven't given the actual operating frequency of this device, your claim lacks global meaning.

It is well known that children generally have better and more extended HF hearing than adults. Human ears have geometry that determines how they work and when they grow, the geometry shifts towards lower frequencies.

Quote
I wonder if the hearing range of various human populations has been well-studied (such as African tribes with certain members who can hear low frequency tones emitted by elephants).


We don't need to jump down every rabbit hole to know where to find rabbits every time we need to. ;-)

Perhaps one of the most cogent comments on this thread is from JJ, who makes the point that since lossy coders provide substantial amounts of data reduction with minimal sound quality loss all things considered, and they work based on what we know about human hearing, we must know quite a bit of useful information about human hearing.

Quote
Is it fair to say that this good work has been done primarily based on averages of 1st World populations?


More to the point there has been no need to develop special lossy encoders for every country, continent or people group. This is particularly true when good unbiased listening tests are used to determine what people are and are not hearing.
Title: What we measure is what we hear
Post by: Ethan Winer on 2012-12-31 17:14:54
Subjectivity can still provide valuable data points (especially negative impressions).

The problem with subjective non-blind tests is that perception varies from moment to moment. Something that sounds bad now might sound excellent tomorrow or even five minutes later. This is the main reason most subjective audiophile reviews are useless, except when something is extremely bad.

--Ethan
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2012-12-31 17:25:40
Subjectivity can still provide valuable data points (especially negative impressions).


The above is like saying that radioactivity can lead to thermonuclear explosions.

Subjectivity can provide valuable data points, but the frequency of incidence of this happening can be disappointingly low.

Consider all of the high end audiophile publications. Thousands or tens of thousands of pages of subjective impressions per week or month that would simply cease to exist if proper listening test protocols were used.
Title: What we measure is what we hear
Post by: Nessuno on 2012-12-31 17:41:32
Subjectivity can still provide valuable data points (especially negative impressions).

The problem with subjective non-blind tests is that perception varies from moment to moment. Something that sounds bad now might sound excellent tomorrow or even five minutes later. This is the main reason most subjective audiophile reviews are useless, except when something is extremely bad.

--Ethan

This could be true to a certain limited extent, but if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail, expecially at borderline conditions.
What can really change and by large amounts are expectations in sighted tests, when the tester knows what he's listening to and could link to the listened device side effects of other precedent or contemporary experiences, different knowledge, true or supposed, even the mood of the moment...
Title: What we measure is what we hear
Post by: greynol on 2012-12-31 19:29:03
Maybe we should take a close look at the meaning of the word perception. From where I stand, and from having an opinion shaped by what I have read by many of the fine members here and elsewhere with demonstrated expertise in the area, I will add that expectation bias does in fact alter our the perception of what we hear.  The suggestion of a difference in X (breath/openness/dancability/PRAT but also non-BS qualities) can result in the actual and very real perception of that difference in X to the individual, even when the stimulus is held constant.

So no, changing perception does not have to affect DBT results necessarily.  In the case of having placebophiles with strong expectation bias we often see the reverse: DBT affects the perception, or maybe not until the results of the test are revealed. That's still no guarantee since some people think that getting 13 right out of 20 means they passed the test.  Still others will continue to cling to their faith in fairy dust, especially when the perceived differences once again return when the constraint of a DBT has been lifted.
Title: What we measure is what we hear
Post by: jkauff on 2012-12-31 20:28:32
Next, on to wine tasting.... 
Title: What we measure is what we hear
Post by: greynol on 2012-12-31 21:13:40
Feel free to start a new topic, replacing the word hear in the subject line with the word taste.  Be sure to post it in the off-topic forum.
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-01 02:06:39
I don't think this forum is scared of subjective opinions; we simply believe that expectation bias should not have a role at the table when it comes to gauging and expressing them.



Amen to that. Sorry, but it had to be said twice.
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-01 02:08:26
What's the "best measuring" frequency response of a headphone? I wonder...


On whose head, with what haircut, under what circumstances, with what ear canal volume and shape?

Martel has a real point here.
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-01 02:10:45
This could be true to a certain limited extent, but if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail, expecially at borderline conditions.


To some extent that's true.

Even something so simple as measuring one's hearing can be very different on day 1 vs. day 3 of getting off of the plane, or if you spent 8 hours on the show floor vs. ran through it with earplugs.

But this kind of thing can be managed, and that's also what controls are for.
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-01 02:12:33
Feel free to start a new topic, replacing the word hear in the subject line with the word taste.  Be sure to post it in the off-topic forum.


Actually, I know of at least 1 or 2 good books on sensory evaluation of wine that directly read on audio testing if you think even a little bit.
Title: What we measure is what we hear
Post by: greynol on 2013-01-01 02:25:10
You're right, I just don't know that we should drop comparing/correlating audio measurements and actual hearing which we hardly touched upon in order to move onto expectation bias in wine tasting.  If anyone should be suggesting the direction of the discussion it should be the original poster.
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-01 11:47:31
Maybe a little OT, but I recall a "rule of thumb" more than a woman told me they follow when they buy perfumes is they don't evaluate more than three or even two different fragrances a time, or they would not be able to discern real differences and end up taking home something they could even not like any more days after.
I guess this has everything to do with temporal masking and even perception system fatigue.

P.S. Happy new year full of good music to all HA fellows! 
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2013-01-01 14:47:03
Feel free to start a new topic, replacing the word hear in the subject line with the word taste.  Be sure to post it in the off-topic forum.


Actually, I know of at least 1 or 2 good books on sensory evaluation of wine that directly read on audio testing if you think even a little bit.


I'm reminded of Mielgard, et al  classic book about Sensory Evaluation.

http://books.google.com/books?id=F_A-YtWXF3gC (http://books.google.com/books?id=F_A-YtWXF3gC)
Title: What we measure is what we hear
Post by: Ethan Winer on 2013-01-01 18:09:51
but if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail, expecially at borderline conditions.


There are several different issues. One is that our perception changes from moment to moment. Another is we can't focus on everything all at once when complex music is playing. Another is that some things really are so obvious that they won't be missed whether sighted or blind. Another is expectation affecting our opinion when we know which [device] we're hearing. Another is that moving your head even one inch in any direction really does change the sound reaching your ears (speakers, not headphones). Another is that our sensitivity changes at, as you said, borderline conditions. A real change may be noticed one time but not another. Especially for an old man like me.

--Ethan
Title: What we measure is what we hear
Post by: Martel on 2013-01-01 18:54:34
... moving your head even one inch in any direction really does change the sound reaching your ears (speakers, not headphones)
Almost all headphones "suffer" from a similar problem - seating them on/in one's head/ear (or any headphone measurement device). Two people are not very likely to position the same headphone exactly the same way on their heads due to different head/ear shapes, preferences etc. This is going to change the perceived/measured sound and introduce additional variable(s) into the mix.
Title: What we measure is what we hear
Post by: Yahzi on 2013-01-01 19:49:32
It seems this discussion ain't over yet. Thanks to all those who replied to my original question, much appreciated. Please carry on with the interesting discussion... I'm learning as I go along and I appreciate the input.
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-01 22:08:03
Let alone those quite obvious issues which actually affect ear sensibility, most of which are temporary (except aging, of course ) and as Woodinville pointed out could be managed and taken care of in a controlled environment (and even at home: you'll never try to ABX a critical sample while having a cold or just coming back from a night spent on the dance floor! ), maybe, as said by Greynol, we should agree on the term perception, which by the way comes from latin to take (CAPERE) by mean of (PER).

Now, let me put my previous statement it in another, rather hyperbolical way: if someone spent the price of a sport car in a hi-end amplifier and this fact let him enjoy his music better than before, that's good. It nevertheless doesn't change the amount of information his ear takes from the air motion, even if the new amp measures "far better" than the one he had before and so something in the air motion has actually changed, because those changes are below the treshold of his ear perception, even he is in perfect healt, relaxed, in a good mood and willing to hear some improvements. If instead he had spent far less money in a new pair of speakers which measure "better but not that much" and so his expectations could be lower, still his perception threshold would have been actually triggered by the larger amount of changes a new transducer takes, even if his head was in the exact place relative to point sources etc etc...

Both times his brain has felt a change, but the first time was not a perceived one (not by mean of ears at least).

All that said, my vision of the issue at hand and interpretation of the terms used might as well be wrong because this is not my working field of interest so I don't read specific literature (and also I'm not a native english speaker).
Title: What we measure is what we hear
Post by: greynol on 2013-01-01 23:23:04
The point I was making is that the ears (outer or inner) don't perceive; rather it is the conscious mind does the perceiving.  Further, perceptions can be altered though expectation bias.  People who are persuaded into "hearing" (or "not hearing") something will actually experience it, regardless of the stimulus to the ear.  To them it is very real and is perceived as such.
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-01 23:41:03
The point I was making is that the ears (outer or inner) don't perceive; rather it is the conscious mind does the perceiving.  Further, perceptions can be altered though expectation bias.  People who are persuaded into "hearing" (or "not hearing") something will actually experience it, regardless of the stimulus to the ear.  To them it is very real and is perceived as such.



This is really the whole point. Perception is very much influenced by everything, including your internal thoughts, and pure randomness.

In order to make useful measurements, one has to establish understanding in the person doing the sensory evaluation first. This is called "training" in some contexts.
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-02 00:01:43
The point I was making is that the ears (outer or inner) don't perceive; rather it is the conscious mind does the perceiving.

Ok, but "perceptual" codecs are focused on discarding exactly those stimula that the ears (and/or lower brain functions) are not able to discriminate and blind tests are used just to put "consciousness" out of the equation.

But now it's me just quibbling , I think the sense of what has been said upon measure vs. hear is clear and also upon hearing abilities not actually changing by expectations, so I stop here...
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-02 00:12:02
In order to make useful measurements, one has to establish understanding in the person doing the sensory evaluation first. This is called "training" in some contexts.

So hi-end magazines are simply "training" audiophiles on behalf of expensive gears makers?
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-02 00:40:02
In order to make useful measurements, one has to establish understanding in the person doing the sensory evaluation first. This is called "training" in some contexts.

So hi-end magazines are simply "training" audiophiles on behalf of expensive gears makers?


No, training is much more than reading an article with imprecise, untestable preferences that are stated as facts.
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-02 00:44:00
The point I was making is that the ears (outer or inner) don't perceive; rather it is the conscious mind does the perceiving.

Ok, but "perceptual" codecs are focused on discarding exactly those stimula that the ears (and/or lower brain functions) are not able to discriminate and blind tests are used just to put "consciousness" out of the equation.


Perceptual codecs, by and large, use the auditory periphery sensitivity to throw out information.

If the periphery can not capture the information, it never reaches the cognative level.

You must separate cognative effects from peripheral effects, please. Cognative effects are as plastic as plastic gets (i.e. they can change, adapt, etc).  The periphery is very nearly fixed, with simple feedback mechanisms from the CNS.
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-02 11:10:34
You must separate cognative effects from peripheral effects, please. Cognative effects are as plastic as plastic gets (i.e. they can change, adapt, etc).  The periphery is very nearly fixed, with simple feedback mechanisms from the CNS.

You are augmenting the detail level (and I'm liking to go deeper in a field I don't know in finer details) but this doesn't change the whole picture.
Correct me if I'm wrong: we could see it, for sake of simplicity, like a two block system. The first block which houses the transducer (ear) and its low level driver logic (peripheral auditory system), the second block which houses "business logic" (cognitive functions like, say, construct vectors of stimuli and match them with previously known patterns).
The second block takes its inputs mainly from the first, but is able to change its functions if properly instructed and is also able to aggregate other inputs, from for example sight, to select different functions and give different interpretations to a same input from the first block.

Now, for example, spectral or temporal masking is a resolution limit of the first block, you cannot "train" someone to improve upon this aspect, but you can instruct the second block to overcome some of this limitation integrating its input with other stimuli: when an individual sees, for example, a certain cable between the amp and the speakers he is listening to and he was told those cables possess a "superior resolution power", he actually perceives (in the sense Greynol gave to this term) "firmer bass", "silky trebles", "deeper soundstage" etc etc... even if swapping cables haven't changed anything at first block level.

Relating to what you call "training", correct me once more if I'm missing the point, is actually like improving functions of second block to better recognize patters in stimuli, but using only input from the first block.

So, to go back to the topic of this thread, better measures of a device doesn't directly translate in better perception, as long as their influence is limited to the first block (and the improvement in measurable quantity is beyond its resolving power).
Title: What we measure is what we hear
Post by: Martel on 2013-01-02 12:19:58
It's nearly impossible to measure subjective feelings of an individual (even sticking electrodes into one's brain doesn't help much). They need to describe/tell you what they hear/feel (until we finally invent the telepathy). Unfortunately, words do not carry nearly as much information as is contained within the feelings they're trying to describe. Hence all those audiophile metaphors (which translate very poorly between individuals and carry almost no useful information as a result).

When I was younger (teen-age) I used to bask in the placebo effect coming from comparing MP3 encoders through a spectrogram. Cut-off at 18kHz was "not good enough for me and sounded like crap", even though the headphone I was using at that time (HD 490 live) had a significant treble roll-off (roughly comparable to Porta Pro) - I just put them on again recently and the treble was virtually non-existent.  I ended up using some weird SW called SCMPX, which went all the way to 22kHz on 192kbit MP3 (and likely made worse MP3s overall).
Yet, the associated feeling of superiority (due to extended frequency range that I saw) was really there, it made me enjoy my MP3s much more. While I try to spend as little money as possible to get my equipment, I understand the audiophiles spending fortune on their equipment - for them, just spending big money on equipment is enough to boost their satisfaction level. Due to the physical limits of the human auditory apparatus, this seems to be one of the few ways to improve the perceived sound beyond some objective supremum.

I know I enjoy some fast black/death metal much more when I'm drunk. Some of my friends told me they enjoy techno more when high on weed. Now measure that. 
(I'm sorry if this is not appropriate - moderation remove this last sentence in case you find it too offensive)
Title: What we measure is what we hear
Post by: heatfire on 2013-01-02 14:53:11
Can only measure that we hear it not how or what we hear;)
What we experience in the sound is just subjective.
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2013-01-02 16:43:12
...if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail, especially at borderline conditions.


No argument. Stuff happens.

How do we explain ABX tests with other than perfectly consistent results?

They happen all the time - do they demand an explanation?

This from JJ looks to me like as good of an explanation as any:

"Perception is very much influenced by everything, including your internal thoughts, and pure randomness."

Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2013-01-02 16:47:07
It's nearly impossible to measure subjective feelings of an individual (even sticking electrodes into one's brain doesn't help much). They need to describe/tell you what they hear/feel (until we finally invent the telepathy).

I know I enjoy some fast black/death metal much more when I'm drunk. Some of my friends told me they enjoy techno more when high on weed. Now measure that.


I'm told that modern brain imaging can be an effective means for measuring the subjective feelings of an individual, especially pleasure.

Google on fmri
Title: What we measure is what we hear
Post by: pdq on 2013-01-02 16:50:07
if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail

Which is exactly why a failed ABX test proves nothing.
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-02 18:16:50
if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail

Which is exactly why a failed ABX test proves nothing.

A failed ABX proves nothing with respect to the codec, or better it doesn't prove that there is no difference between source and encoded sample, but proves that should a difference exists, the individual is not able to discern it, that is the sample is transparent to him.
Title: What we measure is what we hear
Post by: heatfire on 2013-01-02 18:49:42
There is one measurement some "audiophiles" use that really really is not what we hear.. and that is price.. so a 100dollar amp cant play 1000dollar headphones, might not.. or might do it, money aint the answer to the question.
Not even sure they are audiophiles though, more salesmen then reviewers.
Title: What we measure is what we hear
Post by: pdq on 2013-01-02 19:28:43
if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail

Which is exactly why a failed ABX test proves nothing.

A failed ABX proves nothing with respect to the codec, or better it doesn't prove that there is no difference between source and encoded sample, but proves that should a difference exists, the individual is not able to discern it, that is the sample is transparent to him.

I agree with the first part of your statement. However, a failed ABX test does not prove that the sample was transparent to the tester, only that (s)he was unable to show, statistically speaking, that it was not. The failure could have been due to reasons other than transparency or lack thereof.
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-02 20:12:58
However, a failed ABX test does not prove that the sample was transparent to the tester, only that (s)he was unable to show, statistically speaking, that it was not.

Right, by definition transparency cannot be proved in absolute term, only statistically. But if a tester fail in a statistically relevant number of runs, then well, let's say there is a low probability that tomorrow he could as well succeed in a statistically relevant number of runs, all conditions being equals. May we not, then, draw the conclusion that this sample is transparent to him/her?

Quote
The failure could have been due to reasons other than transparency or lack thereof.

As, for example? Please, keep in mind in precedent posts we've already ruled out obvious temporary impairing conditions (illness, fatigue from previous long listening tests, exposition to very noisy environment, very bad mood etc…)
Title: What we measure is what we hear
Post by: Kohlrabi on 2013-01-03 00:13:50
However, a failed ABX test does not prove that the sample was transparent to the tester, only that (s)he was unable to show, statistically speaking, that it was not.

Right, by definition transparency cannot be proved in absolute term, only statistically. But if a tester fail in a statistically relevant number of runs, then well, let's say there is a low probability that tomorrow he could as well succeed in a statistically relevant number of runs, all conditions being equals. May we not, then, draw the conclusion that this sample is transparent to him/her?
To say what pdq said in other words: The reason a failed ABX test doesn't prove anything is because that is not the intention of the ABX test in the first place. You conduct the test to reject the null hypothesis that two samples sound the same, and this can only be achieved by a successful test.
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2013-01-03 00:20:20
if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail

Which is exactly why a failed ABX test proves nothing.


If you are going to be that absolutist about it, then nothing proves anything.

Of course, in Science, proof is not of the essence.

All findings of Science are provisional, and only relevant until something more relevant is found.

So even though a failed ABX test proves nothing, it means something.
Title: What we measure is what we hear
Post by: Kohlrabi on 2013-01-03 00:23:19
So even though a failed ABX test proves nothing, it means something.
Sure, a lot of failed tests with lots of different listeners and circumstances means that it is very, very likely that the samples sound the same and the null hypothesis holds. Still, that's not a proof in the meaning of the word, but likely good enough for any real life considerations.

Science is, after all, the aim to find the most likely explanations for things occurring in nature. To accept that our modern life (or science in general) is based on not-yet-rejected hypotheses doesn't give me sleepless nights, nor do I think about it every waking hour.

It might feel a bit annoying to split hairs over these semantics, but to me understanding the purpose of the ABX test really helps to understand the meaning of its results and their interpretation.
Title: What we measure is what we hear
Post by: pdq on 2013-01-03 01:07:18
Quote
The failure could have been due to reasons other than transparency or lack thereof.

As, for example? Please, keep in mind in precedent posts we've already ruled out obvious temporary impairing conditions (illness, fatigue from previous long listening tests, exposition to very noisy environment, very bad mood etc…)

For example, the tester performed too few trials to be statistically significant, but a greater number of trials would have shown that he was able to hear a difference.

In any case, I am not fundamentally in disagreement with anything that has been said here on the subject, so I will stop nit picking. 
Title: What we measure is what we hear
Post by: Nessuno on 2013-01-03 01:11:15
However, a failed ABX test does not prove that the sample was transparent to the tester, only that (s)he was unable to show, statistically speaking, that it was not.

Right, by definition transparency cannot be proved in absolute term, only statistically. But if a tester fail in a statistically relevant number of runs, then well, let's say there is a low probability that tomorrow he could as well succeed in a statistically relevant number of runs, all conditions being equals. May we not, then, draw the conclusion that this sample is transparent to him/her?
To say what pdq said in other words: The reason a failed ABX test doesn't prove anything is because that is not the intention of the ABX test in the first place. You conduct the test to reject the null hypothesis that two samples sound the same, and this can only be achieved by a successful test.

Stricto sensu, neither a successful test proves anything: a tester can, in theory, guess every time the correct answer without even listening. Of course statistics as well as real life experience lead us to consider this a very unlikely thing to happen, so we accept a successful test as a proof. This is also the reason why a single run cannot prove anything either way.
Now, if a tester fails in a statistically relevant number of runs etc etc... (as per above) what could it possibly mean, according to statistics and real life experience? Will you still accept the hypothesis that the two samples sound different to him?

Edit: yep, I think the whole sense of what has been said is clear, so I stop nitpicking too...
Title: What we measure is what we hear
Post by: Woodinville on 2013-01-03 10:15:41
However, a failed ABX test does not prove that the sample was transparent to the tester, only that (s)he was unable to show, statistically speaking, that it was not.

Right, by definition transparency cannot be proved in absolute term, only statistically. But if a tester fail in a statistically relevant number of runs, then well, let's say there is a low probability that tomorrow he could as well succeed in a statistically relevant number of runs, all conditions being equals. May we not, then, draw the conclusion that this sample is transparent to him/her?
To say what pdq said in other words: The reason a failed ABX test doesn't prove anything is because that is not the intention of the ABX test in the first place. You conduct the test to reject the null hypothesis that two samples sound the same, and this can only be achieved by a successful test.

Stricto sensu, neither a successful test proves anything: a tester can, in theory, guess every time the correct answer without even listening. Of course statistics as well as real life experience lead us to consider this a very unlikely thing to happen, so we accept a successful test as a proof. This is also the reason why a single run cannot prove anything either way.
Now, if a tester fails in a statistically relevant number of runs etc etc... (as per above) what could it possibly mean, according to statistics and real life experience? Will you still accept the hypothesis that the two samples sound different to him?

Edit: yep, I think the whole sense of what has been said is clear, so I stop nitpicking too...


When, at the same time, the subject performs well beyond chance in the positive control stimulus, and randomly in the negative control stimulus.
Title: What we measure is what we hear
Post by: knutinh on 2013-01-03 12:02:54
Good work carried out by Toole and others aside:
Are there good measurement methods available that can prove that the spatial characteristics of two audio events will be indistinguishable in a blind test?

I think that measuring the behaviour of loudspeakers in a room (or microphones) in a perceptually meaningfull manner is quite hard. It is easy to find _some_ difference, it is hard to tag the relevant differences while discarding the irrelevant differences.

People tends to move their heads, and HRTFs can vary quite a bit from person to person. Sticking a single omni B&K microphone at "sweet spot" does not tell us all there is to know about a loudspeaker/room system. Toole advocates empirical weighting of on-axis/off-axis loudspeaker measurements done in anechoic chambers, together with physical analysis of room geometry, but I dont think that approach is necessarily "perfect"?

-k
Title: What we measure is what we hear
Post by: Arnold B. Krueger on 2013-01-03 12:34:01
Good work carried out by Toole and others aside:
Are there good measurement methods available that can prove that the spatial characteristics of two audio events will be indistinguishable in a blind test?


Read the last 10 posts or so again, and ask a different question! ;-)

All we can do in the blind test or any test is reject the hypothesis that there is an audible or other difference. We can't ever prove that any two sounds are absolutely indistinguishable.

The sensitivity of technical tests is such that we can always find a difference between two signals, particularly if they originate in acoustical events.

The sensitivity of really well-done DBTs is such that we can generally find a difference between two signals, particularly if they originate in different ways in an acoustical sense, even at the identical same event.  I haven't done any formal sensitivity studies, but I know that it is not always easy to find two microphones that match each other so well that they can't be distinguished by ear, even when they are as coincident as we can make them.

As has been pointed out by others, my experiences tell me that moving my head a foot, or changing its orientation by a dozen or two degrees makes things sound different. I know that if I'm doing a DBT involving small differences I have to hold my head in a consistent manner in order to obtain reliable detection near the threshold of audibility.

Quote
I think that measuring the behaviour of loudspeakers in a room (or microphones) in a perceptually meaningful manner is quite hard.


That depends on which perceptual meaning that you are investigating.

Quote
It is easy to find _some_ difference, it is hard to tag the relevant differences while discarding the irrelevant differences.


Obviously, the better job you do of reducing or eliminating the irrelevant differences, the easier it is to reliably detect the relevant ones. That's why the golden ears look so silly to people who have done a lot of good listening tests. Their so-called tests control so few things that their experiences are chock full of audible differences. Out of this thicket of audible differences they pick one subtle or non-existent thing such as the SQ of a DAC or amplifier, and write several pages of poetry about it.

Quote
People tends to move their heads, and HRTFs can vary quite a bit from person to person.


So when the golden ears fail to ever talk about the difficulties that are common in listening tests for these reasons, we have evidence about the reliablity of their conclusions.

Quote
Sticking a single omni B&K microphone at "sweet spot" does not tell us all there is to know about a loudspeaker/room system.


That would hardly be news!

Quote
Toole advocates empirical weighting of on-axis/off-axis loudspeaker measurements done in anechoic chambers, together with physical analysis of room geometry, but I dont think that approach is necessarily "perfect"?


You have not yet noticed that nothing in the real world is perfect or ever will be?

(Yoda-like laughter...)  ;-)
Title: What we measure is what we hear
Post by: krabapple on 2013-01-03 17:12:38
if perception really varied from moment to moment, we could as well argue that one day an individual could positively pass an ABX test and the other day fail

Which is exactly why a failed ABX test proves nothing.


By that token, a lone "successful" ABX  on Day 1 is not evidence enough to predict  that the subject would 'really' hear the difference again on Day 2.

Context matters.  Woodinville has outlined positive and negative controls that provide context for drawing more confident conclusions from an ABX text.

That said, when an 'audiophile' natters on about how exquisitely good their audio discrimination is, and then fails a simple ABX of the very abilities they claim to already have (e.g., "mp3s/standard resolution/mass market DACS/plain speaker cable sounds like crap to me.  I can tell every time.") ,  I don't worry too much about whether they were trained enough.  I just enjoy the show.