Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How do we establish "better/best"? (Read 45156 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How do we establish "better/best"?

Reply #25 –
Lemme put it a different way:

I show all of you 3 photos.  You have never before seen the 3 photos.  Each photo is of the same scene, a high resolution image taken recently of the International Space Station with Earth below.  The difference of the photos is that one is 16 colors, one is 256 colors, and one is 32 bit color.

I find it difficult to grasp there would be no concensus (if not absolute concensus) that the 32 bit color image is *best*.  What I'm hypothesizing is that there is some genetic logic in all of us, which means preference is not necessary arbitrary nor infinitely diverse.  Who doesn't like a cold cup of water or a nicely cooked potato?

I think people get too caught up in music taste to understand there are some common mechanics which *can* be stated as fact, like "all other things being equal, CD playback is better than FM radio broadcast".

How do we establish "better/best"?

Reply #26 –
Quote from: gaillard on
Quote from: cabbagerat on

Quote from: gaillard on

but how much resolution? this is hard because its really the brain... not our ears as they are not digital.  its like records.  its analog, it can capture things so minut that digital can never match that, but if our brain has a limit how minut it can tell the difference between then it could.
Please stop trolling. If you really believe this then you need to do some background reading (maybe learn what the terms SNR and quantization noise mean, for a start) before you start a debate on it.
Quote
an anolog system of anything has infinite resolution compared to any non quantum computing. Take for example a needle on a pivot. One end of the needle is pushing on some deformable material, clay for instance. if you where to run that clay surface by the needle at a set rate and then wiggled the other end of the needle sporadically it would get a bunch of impressions. now in digital how do you store those impression levels? you would need an infinite number of values.
I'll give you the benefit of the doubt and assume that you are merely ignorant and not a troll.

Analog electronic systems do not have infinite resolution. Small changes in voltage are swamped by noise generated in all the conductors and semiconductors in the system. This noise is measured with a Signal to Noise Ratio (SNR) - when this is low enough the signal is effectively lost. Noise is the limiting factor in both analog and digital audio systems.

Quote
this is why calculus was invented. to be able to do math with infinite things. but you can not use calculus do things machanical that are infinite, because in a given amount of time how are you going to run an infinite number of calculations?
Eh? What exactly are you suggesting here?




Did i say anything about an electrical system? i am merely talking about how to store something that is analog, just how to represent it.  I don't appreciate all the insults.


The issue here is that you are confusing the theory of an anlog signal (which cannot exist in reality) with actual analog signals, which are only rough approximatations of this ideal (much in the sense that PCM is an approximation of the ideal digital signal with infinate bit depth).  True analog depends on having infinate levels to choose from.  Basically it assumes that all of reality is continuous.  Clearly this is not the case because air is made of molecules, energy comes in quanta, and electric fields have discrete electrons in them.

Actually both PCM and analog have the same fundimental limiation, and thats that neither can store infinately much information in a finite space due to those pesky laws of thermodynamics.  The difference is in the approach they take to addressing this problem.  Since PCM is generally more robust and easier to work with, its more popular these days, although the two are largely equivilent.

This limitation is even more fundimental then this however, It comes up in a lot of places.  For instance, if analog signals like you're thinking of actually existed, we could build Hyper Turing Machines and use them to invalidate the Church Turing Thesis, which is one of the underling premises of computer science.  Unfortunately,  so far, it appears that our universe cannot accomidate such things.

How do we establish "better/best"?

Reply #27 –
Quote
I'm pointing out that having a 'reference' in a pure preference test may be a mistake.

how can have a mistake in what we prefer? scientific proove against the preference?
it takes me again to what i wrote about preference(taste) :
Quote
taste is personal(individual)!
is easy to deduce but don't means that what is "better" is the "best" for my(our) taste.
as "best" is individual taste and "better" is the scientific result,we trash the scientific result or  change the mind as(if) "best/better" are differents?
worse: if we change the mind means that we don't have personal taste...

....really worse: what is the advantage of abandon(left) the preference to the scientific proove if they are differents?
i'm sure that we all choose for the preference.....right?

@ hellokeith

great post! 

@ Mike Giacomelli
i'm still reading your post again and again because
Quote
The issue here is that you are confusing the theory of an anlog signal (which cannot exist in reality)
but we can ...lol
what's the issue if we commit mistakes and confusions in what don't exist?

How do we establish "better/best"?

Reply #28 –
Quote from: Mike Giacomelli on
This limitation is even more fundimental then this however, It comes up in a lot of places.  For instance, if analog signals like you're thinking of actually existed, we could build Hyper Turing Machines and use them to invalidate the Church Turing Thesis, which is one of the underling premises of computer science.  Unfortunately,  so far, it appears that our universe cannot accomidate such things.
This conjecture, which seems to be a good candidate for being a fundamental law of the universe, comes up in all sorts of strange places. Computability theory is extremely interesting - with profound implications for the future of technology. For example, were it not for relativity and it's effects, it would be possible to construct a mechanical oracle, which would calculate the exact value of the halting probability in finite time. This (or the analog hypercomputer you describe) would really mix things up in physics, computer science and mathematics.

How do we establish "better/best"?

Reply #29 –
Quote from: jlt on
Quote
I'm pointing out that having a 'reference' in a pure preference test may be a mistake.

how can have a mistake in what we prefer? scientific proove against the preference?


ABC/HR is not a test for just preference. Is for testing similarity to the reference, and just that. It can be a test for preference just in case your absolute goal and preference is always for fidelity to the reference, no matter the reference. But then we are establishing in advance what has to be considered better. In case of lossy codecs this is usually the goal, but this can't be said to be always the case. What if you test a DSP that enhances a poor sounding reference? A non-processed clip would sound exactly as the reference, so would get the highest rating in the test, but the processed clip would probably sound better to your ears, and should get a lower rating. But sounds better. So ABC/HR would just complicate things.

In a test for just personal preference, without any other considerations, you would just need two clips. You would listen to them, and then say which one you prefer. Without any conditions of what has to be considered better. Just your taste.

How do we establish "better/best"?

Reply #30 –
Quote
ABC/HR is not a test for just preference.Is for testing similarity to the reference, and just that. It can be a test for preference just in case your absolute goal and preference is always for fidelity to the reference, no matter the reference.
So ABC/HR would just complicate things.
humm..seems confuse but i understood.
ABC/HR can be used for tests if the preference is always for fidelity but would just complicate things.
ok.

Quote
In a test for just personal preference, without any other considerations, you would just need two clips. You would listen to them, and then say which one you prefer. Without any conditions of what has to be considered better. Just your taste.
all right,is clever.

my pleasure "meet you" in the forum,thanks for all clarifications.

How do we establish "better/best"?

Reply #31 –
Quote from: jlt on
Quote
and I have 95% confidence that less than 5% of the general population prefer that kind of color combination...
ah!ah!ah!... lol
you're too much.

you wrote the word explaining the meaning of "paradox": is conflict! (lol,my poor english really bore me because now came one new paradox(or conflict?):what is "better" and what is "best"?! conflict or paradox?!    )

thanks pepoluan,your last 'diplomatic' post show how the 'statistics' works.
I'm not really good at statistics too because the "ghost of paradox"(or conflict?) back "alive" sometimes. lol

Um, I know what the definition of "paradox" is; what I want to know in my posting was, which part of your statement does your use of "paradox" applies to?

Which is why I asked, "What paradox?" Or in a more formal wording: "You stated that there was I a paradox; I did not see where. Could you point out where is the paradox?"

Quote from: Woodinville on
Quote from: pepoluan on
I was only pointing out the tool by which one can determine which is "best" for him/her.
And, I'm pointing out that having a 'reference' in a pure preference test may be a mistake.
Gee, I see you are also ignorant of the "/HR" part. It stands for "Hidden Reference", which means that the reference is hidden, i.e. the listener has no visual cue whatsoever which one is called the reference; in this case, the ABC/HR tool will only perform as comparator and subjective scorer, allowing you to give score to both the "Left" sample and the "Right" sample (i.e. based on their location on the computer screen).

Quote from: KikeG on
ABC/HR is not a test for just preference. Is for testing similarity to the reference, and just that.
That is what happens when you launch a full-fledged ABC/HR session with 1 reference sample and > 1 test samples.

When you use only 2 samples, assign one as the pseudo-reference, and the other as the test; when the test run, there will be only 2 columns, labeled "Left" and "Right". You have no way to determine, before the test ends, which one is which, unless you do a memory dump and analyze the internal variables of the tool.


How I wish people before posting will at least check out how the ABC/HR tool works; I am not debating the principle of the tool.


Put it this way: The ideal tool to punch a nail into a wood is arguably a hammer. But in lack of a hammer, I wouldn't hesitate to use a biggish, solid rock, despite the obvious difference in the functions of a hammer and a rock.

How do we establish "better/best"?

Reply #32 –
Quote from: pepoluan on
Quote from: KikeG on
ABC/HR is not a test for just preference. Is for testing similarity to the reference, and just that.
That is what happens when you launch a full-fledged ABC/HR session with 1 reference sample and > 1 test samples.

When you use only 2 samples, assign one as the pseudo-reference, and the other as the test; when the test run, there will be only 2 columns, labeled "Left" and "Right". You have no way to determine, before the test ends, which one is which, unless you do a memory dump and analyze the internal variables of the tool.


How I wish people before posting will at least check out how the ABC/HR tool works; I am not debating the principle of the tool.




I'm fairly certain you didn't read Kikeg's post.  You seemed to repeat everything he said and then accuse him of not saying it.

How do we establish "better/best"?

Reply #33 –
In case of just comparing two samples, both ABXY and ABC/HR could be employed to do a test for preference, though they were not designed for that precise task. You would have to avoid listening to A&B in ABXY or to the reference in ABC/HR.

In case of testing several clips ABC/HR would make things quite complicated. For that task, the old tool WavRate from Ivan could be handy. It's similar to ABC/HR but without needing to rate a hidden reference for every clip at test. It has a single reference that can be easily avoided.

How do we establish "better/best"?

Reply #35 –
Quote from: Mike Giacomelli on
The issue here is that you are confusing the theory of an anlog signal (which cannot exist in reality) with actual analog signals, which are only rough approximatations of this ideal (much in the sense that PCM is an approximation of the ideal digital signal with infinate bit depth).  True analog depends on having infinate levels to choose from.  Basically it assumes that all of reality is continuous.  Clearly this is not the case because air is made of molecules, energy comes in quanta, and electric fields have discrete electrons in them.

Actually both PCM and analog have the same fundimental limiation, and thats that neither can store infinately much information in a finite space due to those pesky laws of thermodynamics.  The difference is in the approach they take to addressing this problem.  Since PCM is generally more robust and easier to work with, its more popular these days, although the two are largely equivilent.

This limitation is even more fundimental then this however, It comes up in a lot of places.  For instance, if analog signals like you're thinking of actually existed, we could build Hyper Turing Machines and use them to invalidate the Church Turing Thesis, which is one of the underling premises of computer science.  Unfortunately,  so far, it appears that our universe cannot accomidate such things.



Which is smaller, molecules in the air of some wave or 16 bits of the same wave?  Not to mention molecules are not just static points they change and influence their neighbors and such, how can one just ignore all of that?

How do we establish "better/best"?

Reply #36 –
Quote from: gaillard on
Quote from: Mike Giacomelli on
The issue here is that you are confusing the theory of an anlog signal (which cannot exist in reality) with actual analog signals, which are only rough approximatations of this ideal (much in the sense that PCM is an approximation of the ideal digital signal with infinate bit depth).  True analog depends on having infinate levels to choose from.  Basically it assumes that all of reality is continuous.  Clearly this is not the case because air is made of molecules, energy comes in quanta, and electric fields have discrete electrons in them.
Which is smaller, molecules in the air of some wave or 16 bits of the same wave?  Not to mention molecules are not just static points they change and influence their neighbors and such, how can one just ignore all of that?
Unfortunately all this pointless debate about analog waves finally have to pass the highly-variable neuro-path between the ear and the brain... which IIRC is a roughly-quantized sampler.

How do we establish "better/best"?

Reply #37 –
Quote from: gaillard on
Which is smaller, molecules in the air of some wave or 16 bits of the same wave?  Not to mention molecules are not just static points they change and influence their neighbors and such, how can one just ignore all of that?
Huh? What are you trying to imply?

Bits don't really have a size (but there is a limit on how small they can be), and sound waves have length, but their physical size depends on the surface that radiated them and other factors. I don't really see the point of your argument.

How do we establish "better/best"?

Reply #38 –
@ pepoluan
 
take it easy 
read with atention please:
Quote
you wrote the word explaining the meaning of "paradox": is conflict!

Quote
Um, I know what the definition of "paradox" is; what I want to know in my posting was, which part of your statement does your use of "paradox" applies to?
i wrote that paradox means conflict are equivalent words and i not accuse something in your post,i don't apply the word to your post but only about the meaning.
i was clear and i'm trying to be clear here again.
few more lines later i thank you...seems that you really misunderstood what i wrote:
Quote
thanks pepoluan,your last 'diplomatic' post show how the 'statistics' works


wait....now i need to read the remainder and stop to    cry before answer. (lol)

...
Quote
Gee, I see you are also ignorant of the "/HR" part.
for sure and not only about "/HR" part.
i don't know "everything about all".my mind is not blocked with false proud and remains opened to learn.
who think that "knows to much" or always have "wise feelings" have mind blocked...and can be sick.it's not my case.
i learn more doing questions than trying to show more than i know( hey,this last phrase can be used in rock or metal...repeat 3 times...)

..the remainder of your post is another lesson and i thank you again.
remeber,
forum is to change informations, (i)we are here to learn and when possible,take what we know as gift to the others.

regards.

How do we establish "better/best"?

Reply #39 –
Quote from: gaillard on
Which is smaller, molecules in the air of some wave or 16 bits of the same wave?  Not to mention molecules are not just static points they change and influence their neighbors and such, how can one just ignore all of that?


Perhaps you weren't watching, but it's easily shown that the atmospheric noise over the area of an eardrum is roughly a white noise level in the 20Hz to 20kHz range on the order of 6db SPL (standard psychoacoustic reference).

Now, what you mean by "16 bits of the same wave" is not clear to me, since you have not specified a relationship of quantizer level to reproduction level. Likewise, what are you talking about "are not just static points", we're talking measured noise level here. Isn't something like "measured noise level" the way to verify a calculation? (n.b. the calculation works out very nicely in the real world)

The level of atmospheric noise is not under a great deal of dispute among those who have looked at the mechanism and studied what comes out of a low-noise microphone. Have you some data to offer us here?

Quote from: pepoluan on
Unfortunately all this pointless debate about analog waves finally have to pass the highly-variable neuro-path between the ear and the brain... which IIRC is a roughly-quantized sampler.



It would be better to call it a fairly rougly quantized pulse-position-modulation system with many parallel channels.

But your point does hold. Inside any given critical bandwidth at a given signal level, something on the order of 30dB SNR is about what you get.

Quote from: pepoluan on
Gee, I see you are also ignorant of the "/HR" part. It stands for "Hidden Reference", which means that the reference is hidden, i.e. the listener has no visual cue whatsoever which one is called the reference; in this case, the ABC/HR tool will only perform as comparator and subjective scorer, allowing you to give score to both the "Left" sample and the "Right" sample (i.e. based on their location on the computer screen).


Let's you and I understand something, I am one of the people who DESIGNED the ABC/hr test.

A is a reference. It is always the reference. It is nothing but the reference. The fact that there is ALSO a hidden reference is irrelevant.  The reference is always, and I repeat, always known to the listener. All the time, always.  If it's not, then it's not an ABC/hr test.

You can not get away from the simple fact that A is always the reference. The reference is always available. In a pure test of preference this is wrong.

Consider:

I make a 24 bit signal that sounds good. Then I reduce the level by 10dB, and put a Q=20 peak 10dB high at 2kHz in this signal.

I make that the reference.

Now, I run a test with A being that, one of B or C being that, and the other one being a refiltered signal that has a Q=20 ZERO of -10dB directly over the peak, followed by a 10dB gain. In short, except for losing about 2 bits at the bottom end of the signal, we're back to the good-sounding signal.

We run this in ABC/hr. What happens? The FIXED signal, the one that everone would prefer, is now "very different" and will always score lower than the broken signal.

In other words, you do not test preference with ABC/hr.

It won't work right.

That's not what it's for.
-----
J. D. (jj) Johnston

How do we establish "better/best"?

Reply #40 –
There's something designed for something, yet it is usable for something else. Like my analogy of hammer vs stone...

Ah, but I digress. I'm tired of this pointless debate. 

Anyways, to return to the crux of the matter: The only way to determine what's best between 2 alternatives... is to blindly compare & score them with whatever tool you have that does allow blind comparison & scoring, and accept the score as truth based on your subjective preference. If your ABC/HR tool allows you to not listen to the reference... then don't listen to the reference and listen to the L & R samples only. Discipline yourself to not listen to the reference.

There, that's my final statement.

Barring any personal attack  I don't think I have more to add here

Stay cool, people

/me = over & out.

How do we establish "better/best"?

Reply #41 –
Quote from: Woodinville on
Let's you and I understand something, I am one of the people who DESIGNED the ABC/hr test.


I wonder if I've "talked" with you in another venue?  Anyway, everyone seems to be saying the same thing here, that ABC/hr was not designed as a pure preference test.

The book I consult for subjective testing says that the duo-trio test, as they term it, is a difference, not a preference test, so strictly speaking the grading scale should not be there.

But it seems to work with the grading scale regardless.

ff123

How do we establish "better/best"?

Reply #42 –
Quote from: pepoluan on
Discipline yourself to not listen to the reference.


Why not just use direct (double-blind)  A/B comparison? Goodness!


Quote from: ff123 on
The book I consult for subjective testing says that the duo-trio test, as they term it, is a difference, not a preference test, so strictly speaking the grading scale should not be there.

But it seems to work with the grading scale regardless.

ff123


Uh, but the grading scale is not a preference scale, it is a difference scale. While I don't have the reference in my hand, there are quite some work that shows that difference tends to be more stable than preference, including scaling the difference.

Consider the example of the 10dB bump/removed bump I gave above. With a difference scale, the subject has no conflict, it sounds different. Yes, its' better, so it won't be annoying, or maybe the user will decide that it is annoyingly different than the reference,  but it IS different, and clearly so.

Consider now if we used an MOS scale for both. The listener would face a nasty dichotomy.
-----
J. D. (jj) Johnston

How do we establish "better/best"?

Reply #43 –
Quote from: Woodinville on
Quote from: pepoluan on
Discipline yourself to not listen to the reference.

Why not just use direct (double-blind)  A/B comparison? Goodness!


Because the (free) software is not readily available, probably.  There has been some demand though for a MUSHRA-style format, where the comparisons don't force you to distinguish from the reference individually.  Unfortunately, I won't be the one to write any other software in my limited free time.

Quote
Uh, but the grading scale is not a preference scale, it is a difference scale. While I don't have the reference in my hand, there are quite some work that shows that difference tends to be more stable than preference, including scaling the difference.

Consider the example of the 10dB bump/removed bump I gave above. With a difference scale, the subject has no conflict, it sounds different. Yes, its' better, so it won't be annoying, or maybe the user will decide that it is annoyingly different than the reference,  but it IS different, and clearly so.

Consider now if we used an MOS scale for both. The listener would face a nasty dichotomy.


I see your point.  The text I consulted did not indicate that a grading scale could be used.  I'll have to go back and see why that is.  Could be a hole in what is otherwise a fairly comprehensive book.

ff123

How do we establish "better/best"?

Reply #44 –
There are likely only 2 scientific methods to use here, given the type of data involved.

One is to simply measure signals as in the "days of old" - e.g., analog hi-fi days - and determine which one is statistically closer to the original. This can be presented as criteria of % distortion, noise, phase, etc. For many years this was a chief marketing tool of audio manufacturers, arguing that an amplifier with 0.001% distortion was inherently "better" than one with 0.01% distortion.

While technically valid, this does say anything about the wildly sensitive, selective, non-linear "customer" for these products, the human ear. Ergo ABX or ABC/HR tests. These do not judge quality, but differences and quite possibly preferences of the "customer".

The ear/brain system is very fickle and varies widely from one person to another - the "hardware" may not matter as much as the "software", or ultimate perceiver of the signals in question. Here our science boils down to statistics and nothing more. If I test 100 individuals and find that there is no statistical difference they can perceive between 128kbps MP3s and FLACs of the same material, is 128k MP3 "just as good"? Yes, for these statistical purposes, it is. Does this mean that the participants have damaged hardware? Probably not. It is probably the software.

In this case, only tests from the first method will reliably show that one signal is "better" than the other.

As an engineer (and musician/producer), it is my belief that sound reproduction for music is hampered by many issues that are not at all addressed by these types of tests and measurements. It is something of an "elephant in the room" point of view, I suppose; I assert that the entire method of recording and delivering 2 channel audio is so flawed that "realism" is out of the realm of possibility for the great majority of recordings (surely a topic in itself!). I do not buy any arguments about expensive cables or amplifier "burn-in", I believe the problems are far from subtle and that esoteric playback systems are generally a waste of money thrown at the wrong area. There, now you know my bias 

Does this mean that listening the recorded music isn't enjoyable? Of course not - I am listening right now.  But convinced that the performance is "real"? No way. From that point of view, the goal of various audio delivery systems in the current environment is to provide an aesthetically enjoyable experience, not a faithful one per se.

So the short answer: we can't establish "better/best" in most cases except 1) in extreme cases, and 2) statistically.

How do we establish "better/best"?

Reply #45 –
Quote
Does this mean that listening the recorded music isn't enjoyable? Of course not - I am listening right now. But convinced that the performance is "real"? No way. From that point of view, the goal of various audio delivery systems in the current environment is to provide an aesthetically enjoyable experience, not a faithful one per se.
BINGO!

Quote
So the short answer: we can't establish "better/best" in most cases except 1) in extreme cases, and 2) statistically.
perfect.
i'm not an engineer (and musician/producer) like you, i'm only technician/home musician and this is exactly what i was trying to post(but was affraid) and my crap english can't express the right words like yours.
you broke the "paradox" that i was "talking" (i think that you understand me after read my posts)

my best regards!

How do we establish "better/best"?

Reply #46 –
wow guys, you can take some things and run with them thats for sure.

I am not talking of the eardrum or any darn electronics here.  just simple (or not simple) pattern in nature verse it exactly REPRESENTED digitally.  its not possible without infinite resolution (resolution has a broader definition too...)

thus my original example...

How do we establish "better/best"?

Reply #47 –
Quote from: gaillard on
I am not talking of the eardrum or any darn electronics here.  just simple (or not simple) pattern in nature verse it exactly REPRESENTED digitally.  its not possible without infinite resolution (resolution has a broader definition too...)

Are you sure quantization of reality requires infinite resolution?

Since you're considering a broader definition of resolution, shouldn't you also consider your definition of reality?

How do we establish "better/best"?

Reply #48 –
Quote from: BradPDX on
From that point of view, the goal of various audio delivery systems in the current environment is to provide an aesthetically enjoyable experience, not a faithful one per se.

So the short answer: we can't establish "better/best" in most cases except 1) in extreme cases, and 2) statistically.


Very nicely layed out post, Brad.

Regarding fidelity in reproduction, since very few musical pieces are performed by all members of a band/group simultaneously, there really isn't an analagous way to reproduce that.  And some might even argue that "surround sound" or three dimensional sound is not realistic, since few audience members ever sit in the middle of the stage while the group is performing live.

What I would like to see, is a multichannel audio format which allows us to mix the musical components as we please, and save that mixing in the form of metadata.  Then there could be whole new genre's by digital dj's, without the piracy concerns inherent in our current limited digital audio formats.

How do we establish "better/best"?

Reply #49 –
Quote from: gaillard on
I am not talking of the eardrum or any darn electronics here.  just simple (or not simple) pattern in nature verse it exactly REPRESENTED digitally.  its not possible without infinite resolution (resolution has a broader definition too...)

thus my original example...


So, if you're not willing to consider the eardrum, you're not willing to accept the pattern in nature.

There is, regardless of your belief system, no such thing as "exact" representation in any kind of analog, be it sampled and quantized, or quasi-continuous in time and amplitude.

In the real world there is no exact.
-----
J. D. (jj) Johnston