Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: what value listening tests (Read 41615 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

what value listening tests

There is much discussion that revolves around a/b type listening tests, mostly concering the results of course, but also a large amount concerning the details of the tests and how they are carried out. However so far I have never seen anyone express concern about the fundamental premise of listening tests, and to whether they can ever produce really meaningful results. It seems to be just accepted that the idea of listening tests is a good one, its just a matter of figuring out how to best carry them out.

There is, I believe,  such a huge and very fundamental difference between listening to music under normal circumstances and listening to music during a listening test. Normally when we listen to music its not a purely concious process, rather its more or less an automatic process - we simply let the music kind of wash over us, and leave it up to our subconcious mind to do the work, to  invoke an emotional response, and to provide us with a pleasurable sensation (hopefully).

This is very different to the very concious process during a listening test, where people sit in moods of deep concentration and furrowed brows, focussing their attention on bits of the music, comparing, making notes, remembering, judging, etc.

I dont believe the two processes are comparable enough to draw any conclusions that might apply equally to both scenarios, apart from very crude and broad ones. Normal listening and listening tests are as different as chalk and cheese. If I can invoke a slightly rude example, it is like the difference between a couple having passionate, romantic sex in the privacy of their own bedroom versus doing it in a laboratory in front of technicians and cameras.

Its only when you are really listening to music, as opposed to examining it under a microscope, that you can fully appreciate all the nuances of it - the nuances that will go right over you head and out the window in a listening test. You might think you can hear them, but your ability to pick them up will be skewed by the mood and state of mind you have to be in to do the test.

My advice to anyone trying to evaluate a codec is to convert a large portion or indeed all of your music to the codecs you want to test, load each one successively onto your player, listen to it during the course of a day (and i mean listen normally, not examine), and note down at the end of the day how much enjoyment and pleasure you got from the experience. Take particular note of the times you felt really enthusiastic about the music, when you enjoyed even some of your least favourite tracks, and the other days when you found you kept wanting to skip past tracks, or found your attention wandering from the music.

This is I think much the best way to really properly judge a codec, or a piece of equipment. I found that when I conducted a/b type listening tests with mp3 and wma codecs at high bit rates, I was unable to pick large differences, and wound up thinking that mp3 sounded quite ok. However, subsequent actual lsitening experience revealed that mp3 was very unsatisfying, and during the days I had mp3's on my player, I almost completely lost interest in listening to music. When I put wma versions of the same music on my player, I was suddenly interested once again, and enjoying the music once again. I actually performed this test by accident, accidentally replacing the wma with mp3, which i discovered later -  a perfect example of unbiased testing.

You normally listen to music with your subconcious mind, but compare and judge it with your concious mind. Make no mistake the two are very different.

My advice is by all means conduct listening tests, but take them with a pinch of salt. The best way to judge music is when you are listening normally, not when you are in judgement mode.

Anyway, thats my two cents worth (or less?) for now.

what value listening tests

Reply #1
I'll let someone else handle this. Did anyone bring popcorn?

what value listening tests

Reply #2
I don't know if this topic is gonna be deleted, cause the "argument" has been said a million times before.

Quote
Its only when you are really listening to music, as opposed to examining it under a microscope, that you can fully appreciate all the nuances of it - the nuances that will go right over you head and out the window in a listening test.


Actually, I don't know about you, but in high school biology class, we could see even more nuances when we put the onion skin under a microscope. Maybe you need to focus your microscope a bit, or turn on its light.

what value listening tests

Reply #3
Nobody here cares how you evaluate what you are listening to as long as you meet one critical requirement. You must not know which version you are listening to. Beyond that there are many many valid approaches to "listening tests".

what value listening tests

Reply #4
He has got a point. Sort of anyway. If he thinks it sounds better it does sound better. To him.

Provided he doesn't try to convince anyone else it really does sounds better simply on the basis of his own opinion I cannot see any harm done.

If my wife buys a new handbag and it makes her happy then I'm happy. I wouldn't dream of telling her I cannot detect any functional difference between the new version and all the others.

what value listening tests

Reply #5
Would you step in and correct her if she exaggerated some of its virtues or made unsubstantiated claims while chatting with her friends?

what value listening tests

Reply #6
There is much discussion that revolves around a/b type listening tests, mostly concering the results of course,

This is all I've read of the OP so far, but he's already wrong.

All objections to A/B testing are on the basis of methodology. Results are of no consequence when the methodology is obviously stupid.
elevatorladylevitateme

what value listening tests

Reply #7
Is the neurophysiology for 'listening to music' materially different to that of 'using music to evaluate something'? That's not a facetious question, BTW.






what value listening tests

Reply #8
Quote
Would you step in and correct her if she exaggerated some of its virtues or made unsubstantiated claims while chatting with her friends?


lol. How mad do you think I am? On second thoughts please don't answer that.

I know what you mean though. Just attempting to maintain an unreverential tone in a conversation we have had rather too often and which may well be a troll in any case.

 


what value listening tests

Reply #9
You are wise be cool on both of these fronts.

what value listening tests

Reply #10
The OP may be a troll, or an audiophile, or an honest guy who has an argument with some face plausibility.
Let me assume the latter for the moment.

It's true that, for many people, listening to music carefully for the sake of seeking out differences is different than most experiences of listening to music.
However, it is likely that, in careful listening when you are trying to hear differences, you will notice more nuance, not less. Comparable to how many details you notice in a picture when you're doing one of those "find all the differences between these two pictures" games.

However, it is at least possible that careful concerted listening for the sake of comparison and differentiation puts the listener in a different enough psychological state from normal listening, that he/she will no longer be able to notice some things.

For example, let's say the listener's emotion is sparked by "highs" in the music which are perceived at a subconscious level. Let's say, furthermore, that these "highs" are some of the data lost in the lossy encoding process. The subconscious perception --> emotional reaction link isn't engaged very well by blind testing with careful listening for differences.
But to people who pose this sort of example, I say: set up a different sort of blind listening test. Where you're listening to music as you otherwise would. And see if you experience the "highs". Maybe your wife can set up the test for you. Play the lossless or lossy version of an album, without telling you which one.

Now if the listener is held back from open emotive enjoyment by the awareness that "this music may or may not be the original"... well, sorry, you've lost to placebo already.
God kills a kitten every time you encode with CBR 320

what value listening tests

Reply #11
It's true that, for many people, listening to music carefully for the sake of seeking out differences is different than most experiences of listening to music.
However, it is likely that, in careful listening when you are trying to hear differences, you will notice more nuance, not less. Comparable to how many details you notice in a picture when you're doing one of those "find all the differences between these two pictures" games.


I don't agree.

There's a strong possibility that the psychological (or even neurological) conditions that apply when a person is enjoying a piece of art are materially different to those that apply when the same person is studying that artwork for authenticity. The second condition is not necessarily an amplified version of the first.

If that is the case, there might be different perception pathways between a person listening to a news report and the same person listening a piece of poetry, for example. So, does does music 'appreciation' differ from music 'evaluation' and using music as an evaluative tool in audio development?

what value listening tests

Reply #12
Let us not overlook that the arguments presented by the OP are typical of someone who believes he hears a difference in a sighted test, but is unable to prove it in a proper double-blind test.

The bottom line is that if you design the test in a way that you think will prove that you hear a difference, and as long as it is a proper double-blind test and the results are statistically significant, then we will accept that you can indeed hear a difference. Anything less will not be accepted.

what value listening tests

Reply #13
Now if the listener is held back from open emotive enjoyment by the awareness that "this music may or may not be the original"... well, sorry, you've lost to placebo already.

A problem with ABX testing is that it discounts the placebo effect.  Tests show that "expensive" wine tastes better than "cheap" wine (even if both samples are the same), placebo effects in drugs are real (many people taking placebos experience physiological changes), etc., etc.

For some percentage of the population, 128kbs will never sound as good as 320kbs, whatever a well designed test shows.  These people will have a better music experience with the higher bit rate.  That the experience can be simulated with a lower bit rate but telling them it's a higher bit rate is not really relevant to their enjoyment of the music.

Otherwise, the OPs objection can be solved by better designed listening tests, as others have noted.

what value listening tests

Reply #14
Yawn!  Well there is not one single thing in magneticrabbit's post that hasn't already been covered to death in this forum, but I'll indulge for a few minutes of my time...

My advice to anyone trying to evaluate a codec is to convert a large portion or indeed all of your music to the codecs you want to test, load each one successively onto your player, listen to it during the course of a day (and i mean listen normally, not examine), and note down at the end of the day how much enjoyment and pleasure you got from the experience.
There is no reason why this cannot be done and there is nothing in ABX methodology prohibiting this from being done, though you will need to decide ahead of time how many days you will need to conduct such an experiment.  I'm thinking you're going to need a minimum of 10 or 12 and that you need to be right on each and every day in order to demonstrate any statistical significance.

When I put wma versions of the same music on my player, I was suddenly interested once again, and enjoying the music once again. I actually performed this test by accident, accidentally replacing the wma with mp3, which i discovered later -  a perfect example of unbiased testing.
I flipped a coin once and called it heads; it landed heads.  That certainly must mean I can always predict the outcome when flipping a coin.

Is this an admission that since this was just an accident and that all the rest of your so-called "a/b" testing was sighted?

Provided you're not just a troll, you need to read this before continuing this discussion:
http://www.hydrogenaudio.org/forums/index....showtopic=16295

My advice is by all means conduct listening tests, but take them with a pinch of salt.
Just as you should take any claims that people make about sound quality from either a sighted test or a test with only a few trials with a pinch bag of salt.

what value listening tests

Reply #15
There is nothing inherent in the results of a listening test which guarantees a correlation with subjective listening perception. And subjective listening perception is how humans listen to music.

At the same time, there is nothing inherent in the results of a subjective listening test which guarantees correlated a subjective response with other people. That's the definition of "subjective", after all: it fundamentally varies from person to person, because human perception is not a logical process. (Otherwise, all listening tests would be objective.) Disagreements are plainly found to exist between two people listening to the same component. And even when there is complete agreement, sources of error remain with subjective listening which are strongly proven to reduce the ability to discriminate fine differences in sound.

Neither of those points in isolation matter - what makes them important is communication. And clear and meaningful communication about audio quality demands blind listening tests. When people speak, they generally want to be listened to. In the context of audio, the speech should be meaningful. A subjective evaluation be absolutely meaningful to the speaker - the evaluation correlates exactly with his/her subjective experience. But it might be completely meaningless to anybody else, for any number of reasons. Look at how often flamewars happen on a relatively unregulated subjectively-oriented audio forum over this sort of thing. Either they are extremely common - reflecting the mutual meaninglessness of the posters' subjective opinions - or they are not, BUT the board will almost always have a policy in place to prohibit attacks on posters' listening abilities or experiences - again, reflecting the mutual meaninglessness of the posters' subjective opinions.

In contrast, a well-run listening test may not have a direct correlation with the subjective evaluations of the testers involved, but the results of the test are universally meaningful. The results may be challenged based on specific merits but otherwise are not questioned once they are released. When a positive result is found, it is virtually universally accepted. When a negative result is found, in many cases, the result can still be meaningfully applied to one's listening experience.

In summary: Controlled listening tests are in some sense a sociological phenomenon, reflecting our need to communicate with greater meaning. I myself can have subjectively influenced perceptions which might vary wildly from what is demonstrated from listening tests. But I don't relate them to others, and I never feel like I should relate them to others, because they are meaningless to everybody else but me.

what value listening tests

Reply #16
I don't agree.
There's a strong possibility that the psychological (or even neurological) conditions that apply when a person is enjoying a piece of art are materially different to those that apply when the same person is studying that artwork for authenticity. The second condition is not necessarily an amplified version of the first.

If that is the case, there might be different perception pathways between a person listening to a news report and the same person listening a piece of poetry, for example. So, does does music 'appreciation' differ from music 'evaluation' and using music as an evaluative tool in audio development?

entirely plausible. but you'll notice that the third paragraph and beyond of my post are devoted precisely to detailing the sort of case you're raising here, noting how there can be a difference in things perceived with diff types of listening, and also arguing that you could set up a blind test to replicate this.
As others have argued since then in this thread.
God kills a kitten every time you encode with CBR 320

what value listening tests

Reply #17
entirely plausible. but you'll notice that the third paragraph and beyond of my post are devoted precisely to detailing the sort of case you're raising here, noting how there can be a difference in things perceived with diff types of listening, and also arguing that you could set up a blind test to replicate this.
As others have argued since then in this thread.


Yes, but my point is that AFAIK, no-one's performed such a test.


what value listening tests

Reply #18
Probably because the people who fabricate these silly excuses about the subconscious aren't inclined to perform them to begin with; they're likely just grasping at straws.

what value listening tests

Reply #19
... when we listen to music its not a purely concious process, rather its more or less an automatic process - we simply let the music kind of wash over us, and leave it up to our subconcious mind to do the work, to  invoke an emotional response, and to provide us with a pleasurable sensation (hopefully).


Don't worry. Your subconscious will present in a double blind test too.

what value listening tests

Reply #20
Quote
Tests show that "expensive" wine tastes better than "cheap" wine (even if both samples are the same),


I'm afraid i have to say that is a poor analogy.

If I played the same sample but told my listeners A was a .WAV and B was an MP3 many would say that A sounded 'better'. That's suggestion. Not a ABX.

In my experience most if not all people can tell two wines taste different and will generally, but not always, prefer the taste of the more expensive.

They generally don't have the experience or shared vocabulary to determine the grape variety, style or age but they do know they differ and which one they like the most.

So it's a totally different situation.


what value listening tests

Reply #21

Yes, but my point is that AFAIK, no-one's performed such a test.

Probably because the people who fabricate these silly excuses about the subconscious aren't inclined to perform them to begin with; they're likely just grasping at straws.


... when we listen to music its not a purely concious process, rather its more or less an automatic process - we simply let the music kind of wash over us, and leave it up to our subconcious mind to do the work, to  invoke an emotional response, and to provide us with a pleasurable sensation (hopefully).

Don't worry. Your subconscious will present in a double blind test too.


To both of these I say: awesome. thanks for the well-framed comments, greynol and Light-Fire
God kills a kitten every time you encode with CBR 320

what value listening tests

Reply #22
Now if the listener is held back from open emotive enjoyment by the awareness that "this music may or may not be the original"... well, sorry, you've lost to placebo already.

A problem with ABX testing is that it discounts the placebo effect.  Tests show that "expensive" wine tastes better than "cheap" wine (even if both samples are the same), placebo effects in drugs are real (many people taking placebos experience physiological changes), etc., etc.


It doesn't "discount'  placebo effects.  It reveals them.  In the case of audio, ABX indicates whether an effect is related to a real difference, or an imaginary one.  If the 'effect' is not correlated to whether A or B is really playing -- which is the raw data that ABX gives you -- then one can begin to conclude that it's the subject's biases/beliefs, and not the state of A or B, that are causing it.

No one says imaginary differences can't have SUBJECTIVE effects (like, 'this one sounds better').  The points, that's all they are: subjective.

Quote
For some percentage of the population, 128kbs will never sound as good as 320kbs, whatever a well designed test shows.These people will have a better music experience with the higher bit rate.  That the experience can be simulated with a lower bit rate but telling them it's a higher bit rate is not really relevant to their enjoyment of the music.


Sorry, what are you trying to say here? That some people can really tell 128kbps from 320kbs?  This is not news.  Or is it that when people BELIEVE they are hearing 128kbps, and have been told that this is bad quality, they will tend to hear bad quality sound -- even if it's 320kbps?  This is not news either.

Am I missing your point?

what value listening tests

Reply #23
entirely plausible. but you'll notice that the third paragraph and beyond of my post are devoted precisely to detailing the sort of case you're raising here, noting how there can be a difference in things perceived with diff types of listening, and also arguing that you could set up a blind test to replicate this.
As others have argued since then in this thread.


Yes, but my point is that AFAIK, no-one's performed such a test.



Blind tests where the subjective *preference* or emotional response rather than difference is evaluated, are such tests: MUSHRA, etc. 


This is my problem with armchair 'skeptics' in many fields.  They seem to think no scientist in the field could have considered their point before.


(OTOH, if you're going to complain that no one has performed EXACTLY a certain test you might have in mind -- e.g., in THIS PARTICULAR living room with THIS PARTICULAR piece of gear/music -- please don't bother.)




what value listening tests

Reply #24
It doesn't "discount'  placebo effects.  It reveals them.  In the case of audio, ABX indicates whether an effect is related to a real difference, or an imaginary one.  If the 'effect' is not correlated to whether A or B is really playing -- which is the raw data that ABX gives you -- then one can begin to conclude that it's the subject's biases/beliefs, and not the state of A or B, that are causing it.

Perhaps a bad choice of words on my part.  Agreed

Quote
Am I missing your point?

I was trying to draw an analogy to the placebo effect seen in drugs. 

A pill without any active ingredient can nonetheless cause a physiological change in a person.  Bitrate information can have an effect on listeners, even without a basis in reality.  Discussions of ABX testing seem to dismiss this effect too quickly.

Perhaps a strained analogy and not anything new.