Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Should we care about ABX test results? (Read 38457 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Should we care about ABX test results?

Reply #50
Many have made very detailed, well structured, lengthy arguments here.  I think the issue is quite simple:  the argument made by the OP misses on a very basic premise - the goal of an ABX test is NOT to determine which is BETTER, it is to tell if there is any DIFFERENCE that can be perceived by the LISTENER.  This is a given and isn't up for debate.  "Good" had no place in an ABX test, no matter its definition.  *THE* goal of a lossy codec is to make it such that it cannot be differentiated by the listener from the original source.

The way I see it, the argument made by the OP does not agree with this fact, therefor the entire argument invalidated.  It is more of a statement about whether lossy audio can be BETTER in some way.  But this is NOT a goal of codec developers, and should not be.  Though this could be debated (I suppose), but it is an entirely different question.

Rather than take the validity of ABX testing into question, perhaps the OP should argue what the goals of a lossy codec should be.


Where a listener's percpetion of the audio being "good" or "bad" is valid is in evaluating codecs for telephony applications.  This is because the goal of these codecs is different from hifi audio codecs.  Here, it is a given that the audio from one of these codecs sounds different from the original source.  This is due to the bandwidth of telephone connections.  In evaluating these codecs, the listener's perception of "good" and "bad" are taken into account and are well defined for the listeners.  It was over 15 years ago that I was involved with these tests, so I'm fuzzy on the specific details. 

Certainly the hifi audio we discuss here is in much different realm than voice codecs for telephony applications :-).



Should we care about ABX test results?

Reply #51
Yes, telephony applications or, more generally, low-bit-rate coding, as I already mentioned in post 22. Higher bandwidth (sometimes even 16 kHz or so), but still far from transparency. Has anyone here listened to codecs at bit rates below, say, 96kbps stereo? HE-AAC, for example?

Tomasz, regarding memory-based testing and instant switching:

Such things are determined by the test interface, not necessarily the test procedure. In my opinion, an interface which does not allow the listener to define playback loops of arbitrary length (down to 1 sec or so) or to switch between codecs during playback (maybe with some short crossfading or fade-out and fade-in to prevent clicks) is a bad interface. Regardless of whether it's an ABX, ABX/HR, or MUSHRA test. For me, personally, loops of 2 - 3 sec reduce bias due to my memory to a minimum. Indeed, if I make the loops longer, I start to forget the "sound details" of the beginning of the loop when I reach its end, and that makes it difficult to compare two loops. I usually don't go lower than 2 sec, though, because then the looped segment loses stationarity, and I run into the risk of focusing more on loop effects than on the recording itself. But these numbers vary somewhat from person to person.

Chris
If I don't reply to your reply, it means I agree with you.

Should we care about ABX test results?

Reply #52
Not sure if you've read my post carefully.


Unfortunately, we read your stuff way too carefully for your credibiility to stand.

Quote
It is a pretty indisputable fact that your memory of an experience contains less details than the experience itself. And you are always comparing one sample with the memory of another sample. It's all I need to show that ABX tests cannot give us the certainty that the two experiences are equivalent.


You're changing your story Tszyn. Your old story was that only ABX tests had problems.

Are you beginning to see the light - that *all* valid listening tests have similar or indentical problems such as this one?

Are you beginning to understand that while ABX, ABC/hr, and MUSHRA have their (mostly managable) problems, they are worlds better than the common alternatives?

Quote
There can be no objective measurement of experience...


Actually there can be all kinds of objective measurements of human experience. For example, we can objectively measure the experience of running by measuring the time it takes for a certain runner to experience running a certain kind of course.

It appears that you are whipsawing us Tszyn, first with overly-narrow criticisms of ABX, now with overly broad indictments of just about *everything*.  Thing is Tszyn, we're not fooled, the problem is not too many unsolved problems, the problem is that you haven't done your homework and found out what solutions already exist.

Quote
Instant switching does not help, because then you are comparing different parts of the sample (the first part of A with the second part of B, for example).


Again, you haven't done your homework, Tszyn. We address that problem by comparing short segments. If A and B are short, you can compare all of A with all of B, and do it all within the limited time window of human perception of small differences.

Should we care about ABX test results?

Reply #53
Quote
The codec's task is perceptual fidelity to the original (measured objectively).


I am questioning the statement that the purpose of a codec is fidelity. In your response, you repeated the statement that I am questioning.

Perhaps this will help: If studies show that 90% of listeners prefer a particular kind of artefact, then do you think it should be left in or eliminated in the name of fidelity?

Should we care about ABX test results?

Reply #54
Perhaps this will help: If studies show that 90% of listeners prefer a particular kind of artefact, then do you think it should be left in or eliminated in the name of fidelity?

I think that was addressed to me.
The artifact should be eliminated.
Well let's assume you leave it in. You then have a codec that will affect 100% of music with that artifact, so those 10% who don't like it are stuffed and furthermore 90% may like it with their Madonna, but the 10% don't like it with their Bach.
If listeners like it then they should install some DSP that recreates it, or sound engineers might start using it as an effect.
But you've misunderstood the purpose of lossy codecs (in terms of music). 

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

Should we care about ABX test results?

Reply #55
Ron,

Quote
Such a phenomenon would have nothing to do with audio and is thus not a desirable component of any ABX test. Some other test might be devised to measure this "thinking drain", but we wouldn't want any of that interfering with an ABX test -- and it wouldn't.


I agree.

Quote
You're using this potential for the modification of subconscious "information" as a springboard to question the usefulness of ABX testing, which is, as you admit, a conscious act.


I never said ABX tests are useless. I just said that their usefulness is limited and that other types of tests should be done as well. I think you'll agree.

Quote
There are infinite possibilities for what listening to MP3s could potentially affect (perhaps, by some strange quantum force, listening to an MP3 causes a kitten's heart to stop in another dimension), but are those possibilities relevant to audio?


The mere existence of possibilities is of course no reason for any decisions. But scientific research starts from (sometimes wild) hypotheses. I think it is helpful for audio scientists to realize that there can be effects which do not make it to the listener's consciousness. That's really what I meant to say in the first part of my article.

Should we care about ABX test results?

Reply #56
Quote
The goal of ABX is precisely prevent the subject from using subjective means to choose, and instead tries to force him/her to find a true difference.


Not really. I can use ANY means I want in an ABX test. If A seems more enjoyable than B, then I can use that fact to tell between A and B.

Quote
You won't find friends in this board if you want to give too much importance to subjective opinions.


I give importance to subjective opinions, as long as they are backed by a blind test.

Audiophiles typically claim they HEAR a difference (i.e. they are conscious of it). ABX tests are a great tool to verify such claims.
My article was about something else -- differences that cannot be heard.

Quote
If someone brings up evidence that lossy encoding affect other aspects that could matter to us as a person (for example, if lossy encoding makes cat's life shorter), then we could debate and try to verify it. But in fact, some sort of blind test could be used in this case even. Just not an audio test. (Guess you know that blind tests are being used for medical purposes).


I agree.

Should we care about ABX test results?

Reply #57
Quote
You'll need to explain how something is even possible in the first place before you start wildly speculating.


First of all, I hope (for your sake) that you don't think I believe mp3's cause IQ loss.
Secondly, I agree that some plausible mechanism of action should generally be required before systematic research is done. However, one should remember the brain is a poorly understood organ and it's not easy to draw a flowchart of what affects what.

Should we care about ABX test results?

Reply #58
Audiophiles typically claim they HEAR a difference (i.e. they are conscious of it). ABX tests are a great tool to verify such claims.


Really?

Quote
My article was about something else -- differences that cannot be heard.


How do you know that a difference cannot be heard if your tests are inherently flawed?

Should we care about ABX test results?

Reply #59
Perhaps this will help: If studies show that 90% of listeners prefer a particular kind of artefact, then do you think it should be left in or eliminated in the name of fidelity?
The artifact should be eliminated.

Agree. We can procure this type of artefact during production, if we wish, as an artistic choice. The goal of quality-oriented lossy compression should be fidelity, without any deviation from that goal.

I think it is helpful for audio scientists to realize that there can be effects which do not make it to the listener's consciousness. That's really what I meant to say in the first part of my article.

I see.

Should we care about ABX test results?

Reply #60
Your article was a very confusing read...

What is your point? There seem to be one but as I read further, it seem to spiral into nowhere.. the strange analogy didn't help me either.

Should we care about ABX test results?

Reply #61
Quote
The goal of ABX is precisely prevent the subject from using subjective means to choose, and instead tries to force him/her to find a true difference.


Not really. I can use ANY means I want in an ABX test. If A seems more enjoyable than B, then I can use that fact to tell between A and B.
 
Are you saying you can distinguish A as being more enjoyable, without hearing a difference between the two?

(BTW 'subjective means' here refers to non-auditory input, as far as I understand JAZ's argument.)

Should we care about ABX test results?

Reply #62
Quote
The goal of ABX is precisely prevent the subject from using subjective means to choose, and instead tries to force him/her to find a true difference.


Not really. I can use ANY means I want in an ABX test. If A seems more enjoyable than B, then I can use that fact to tell between A and B.



You can do what you want, and people can look at what you do and reach whatever conclusion they want to. Or, you can use tools for what they are designed for in an effective way, and serious honest people will give you credit for the good that you do.

ABX tests are tests for differences, not preferences. It stands to reason that if you prefer one thing over another, then they have to be different in a relevant way. All ABX can do is tell you that they are different. If an ABX does not tell you that they are different, then any preference you may have, is questionable to say the least.

Should we care about ABX test results?

Reply #63
Quote
The goal of ABX is precisely prevent the subject from using subjective means to choose, and instead tries to force him/her to find a true difference.


Not really. I can use ANY means I want in an ABX test. If A seems more enjoyable than B, then I can use that fact to tell between A and B.



You can do what you want, and people can look at what you do and reach whatever conclusion they want to. Or, you can use tools for what they are designed for in an effective way, and serious honest people will give you credit for the good that you do.

ABX tests are tests for differences, not preferences. It stands to reason that if you prefer one thing over another, then they have to be different in a relevant way. All ABX can do is tell you that they are different. If an ABX does not tell you that they are different, then any preference you may have, is questionable to say the least.


What if someone were to claim that listening to compressed audio made their nose itch after a few minutes of listening? Couldn't ABX testing be used to test that claim?

Should we care about ABX test results?

Reply #64
Yes, why not? I think ABX would be very useful in this case. Provided, of course, that the lossy encoded sample is sonically transparent (to avoid bias) and of sufficient length for the claimed nose itch to develop.

In this case the difference under test is the alleged nose itch inducing capabilities of the lossy sample.

Edit: It reminds me of certain audiophile 'listening fatigue' arguments against lossy codecs. Perfectly testable.

Should we care about ABX test results?

Reply #65
Quote
Well let's assume you leave it in. You then have a codec that will affect 100% of music with that artifact, so those 10% who don't like it are stuffed and furthermore 90% may like it with their Madonna, but the 10% don't like it with their Bach.
If listeners like it then they should install some DSP that recreates it, or sound engineers might start using it as an effect.
But you've misunderstood the purpose of lossy codecs (in terms of music).


So you are essentially saying that if a sound engineer uses an effect then that's alright, but if a codec engineer uses the same effect, then that's wrong. In other words, only sound engineers or end users have the right to make music more enjoyable.

I'm sorry, but that sounds overly dogmatic to me. It's a bit like saying that a taxi driver may not crack jokes because his/her job is to take you from A to B, not to entertain you while you are in the car. If you want entertainment, you can listen to a comedy podcast on your iPod. Of course, you know my opinion: if a taxi driver can drive me to my destination AND make me smile at the same time, more power to him/her.

Sure, 10% of the passengers may not like the taxi driver's jokes, but should we sacrifice the enjoyment of 90% to stop 10% from getting annoyed?

It seems what you are arguing for is choice and I can sympathize with that. DSP filters give me that choice (I can either use them or not), but so would options in a codec. What's the difference?

(There is one argument against euphonicity in codecs that I consider valid: If compressed music sounded better than original music, it would be impossible to perform ABX tests, so it would be harder to test codecs properly. The only available tests would be preference tests, but once A and B are distinguishable, it's harder to eliminate bias, because the listener can figure out which sample is compressed and adjust their responses to make the compressed sample look bad. Of course, this argument does not apply if the euphonicity is optional.)

Should we care about ABX test results?

Reply #66
Quote
So you are essentially saying that if a sound engineer uses an effect then that's alright, but if a codec engineer uses the same effect, then that's wrong.

lossy codecs don't use "effects" (except lowpass filter, of course).

Quote
If compressed music sounded better than original music, it would be impossible to perform ABX tests

No, it would be possible and easy, because compressed music differs from original.

Should we care about ABX test results?

Reply #67
There is one argument against euphonicity in codecs that I consider valid ...

Everyone else, me included seems to disagree. As I said:

Quote
you've misunderstood the purpose of lossy codecs (in terms of music).


From the HA wiki: http://wiki.hydrogenaudio.org/index.php?title=MP3
Quote
MPEG-1 Audio Layer 3, more commonly referred to as MP3, is a popular digital audio encoding and lossy compression format, designed to greatly reduce the amount of data required to represent audio, yet still sound like a faithful reproduction of the original uncompressed audio to most listeners.

Faithful, NOT "better than".


From Wiki: http://en.wikipedia.org/wiki/Transparency_(data_compression)
Quote
In data compression or psychoacoustics, transparency is the ideal result of lossy data compression. If a lossily compressed result is perceptually indistinguishable from the uncompressed input, then the compression can be declared to be transparent. In other words, transparency is the situation where compression artifacts are nonexistent or imperceptible.

"Perceptually indistinguishable" with NO artifacts, NOT "better than".


As it's been said many many times, you're after your personal and highly subjective "better" which has nothing to do with the remit (above) of lossy compression. End of story.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

Should we care about ABX test results?

Reply #68
So you are essentially saying that if a sound engineer uses an effect then that's alright, but if a codec engineer uses the same effect, then that's wrong. In other words, only sound engineers or end users have the right to make music more enjoyable.

I'm sorry, but that sounds overly dogmatic to me.


No, it is absolutely justified, and not at all dogmatic. The sound engineer can use effects because it is part of the process of making the recording, and should be part of the artistic decision-making shared amongst performer(s), producer(s), composer(s) if alive and present, and maybe the studio cat.

The end user (AKA audience) can use effects because s/he knows what s/he wants, in general or in listening to specific music, and no one else is affected.

The codec writer should not use effects because they will be applied blindly to every piece of music encoded with that codec, whether it's death metal, baroque harpsichord solos, or the traditional music of the Cook Islands. The function of the codec, like all the rest of the channel between music makers and music listeners, is to be as transparent as possible. No questions, no ifs, no buts, no different perspectives.

Should we care about ABX test results?

Reply #69
Are you saying you can distinguish A as being more enjoyable, without hearing a difference between the two?


Yes. Check out e.g. this paper on the hypersonic effect: http://jn.physiology.org/cgi/content/full/83/6/3548

The test subjects were able to identify the hypersonic sample as more comfortable to the ears and more nuanced, even though they were not necessarily aware of the difference in a conscious manner. The outcome depended on the question asked.

Also see the burning house example in my blog post for proof that people can see no difference consciously while being able to subconsciously make the right decision if you ask them the right question.

Should we care about ABX test results?

Reply #70
Quote
you've misunderstood the purpose of lossy codecs (in terms of music).


WTF? I am questioning the very purpose of lossy codecs and there you go again, simply restating your point that the purpose of lossy codecs is X. This is no way to carry on a discussion. Do you think the purpose of lossy codecs is something that needs no justification? Are we not allowed to ask questions about what the purpose of lossy codecs should be?

Should we care about ABX test results?

Reply #71
Why shouldn't a car be a boat? You can question the very purpose of cars if you like. Why can't we discuss that? Perhaps because it would be stupid, since boats are already boats, and cars make pretty good cars.
Let's make lossy codecs into digital effects processors, when DSPs already do that.
Furthermore AFAIK foobar can already do what you want, it can apply a variety of DSPs as part of its encoding process, outputting a "better" sounding lossy file.

But fixing a DSP into a codec is as stupid as creating a microphone with a flanger fixed into it.
Why do you think effects pedals come seperate from the instrument? Because a guitar is a guitar and an effect pedal is an effects pedal - they do different jobs.

C.

EDIT: minor amendments to 1st para
PC = TAK + LossyWAV  ::  Portable = Opus (130)

Should we care about ABX test results?

Reply #72
Quote
The codec writer should not use effects because they will be applied blindly to every piece of music encoded with that codec, whether it's death metal, baroque harpsichord solos, or the traditional music of the Cook Islands.


This assumes that a codec cannot tell between different types of music and apply effects accordingly. That may be true of current codecs, but it is not true in principle.

Also, I believe (correct me if I'm wrong) that applying DSP to compressed audio produces inferior results than applying it to the original stream. This would be a technical argument in favor of providing euphonic options in codecs.

By the way, notice I said "options". A codec effect that improved music 90% of the time for 90% of the people should be turned on by default, but people who care (like you and me) should have the option of turning it off. Still, if providing choice is not feasible, we should do whatever is best statistically.

In other words:
If I could make a codec that improved music 90% of the time for 90% of the people (and made it worse to an equal degree in 10% of the cases) and could not provide it as an option for whatever reason, I would go for it. Why? Because the alternative is absurd. The alternative amounts to putting the needs of 10% above the needs of 90%.

If you're going to ignore the needs of 90% in the name of some "purpose" (such as transparency), you'd better be ready to show how that purpose will improve everyone's well-being in the long run. The moment you lose sight of that is the moment you've become an ideologist, i.e. someone who puts abstract ideas above the welfare of the people.

Those of you who feel the need to lecture me about "the purpose of codecs" again, this is the place where you need to justify why it is good for everyone that codecs have a narrowly defined purpose that excludes enjoyment from consideration. There is no Magical Purpose From The Sky.

Should we care about ABX test results?

Reply #73
Why shouldn't a car be a boat?


If 90% of the people preferred a car that doubled as a boat (we're discussing the case where 90% of the people prefer a particular change), then there is no reason why that should not be built. It's only a question of how often people need to sail in it and how much worse of a car it would be.

Saying "a car is not a boat" is an argument only to people who believe there is some God-given template of car that we should not diverge from. Conservatives say: "Homosexual marriage is wrong because a marriage is a union between a man and a woman". That's the end of the discussion for them. Liberals say: "That's not enough. We can make marriage whatever we want it to be. The only issue is whether it causes actual harm.".

Similarly:
If 90% of the people want to make phone calls on their PCs, are you going to say "a PC is not a telephone, we already have phones for that?"

On the other hand, there are some things that should not be combined. For example, a mobile phone that doubles as a sandwich would be a poor idea. Why? Not because "a mobile phone is a mobile phone". Because the resulting concoction would be useless as a phone and useless as a sandwich.


Quote
Why do you think effects pedals come seperate from the instrument?


So you wouldn't buy a guitar that came bundled with a couple effects pedals?

Should we care about ABX test results?

Reply #74
So you wouldn't buy a guitar that came bundled with a couple effects pedals?
Not if they could not be switched off or removed, I wouldn't.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-