Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: RMAA Test on different Codec (Read 11851 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

RMAA Test on different Codec

Reply #25 –
Quote
Hmm.... so what is the best measurement tool here?

Blind listening tests.

RMAA Test on different Codec

Reply #26 –
Quote
Blind listening tests.
[a href="index.php?act=findpost&pid=231074"][{POST_SNAPBACK}][/a]

But again, isn't that subjective?


[Edited] Remove unnecessary quoting.
Break The Rules!!!

RMAA Test on different Codec

Reply #27 –
Quote
Quote
Blind listening tests.
[a href="index.php?act=findpost&pid=231074"][{POST_SNAPBACK}][/a]

But again, isn't that subjective?
[a href="index.php?act=findpost&pid=231076"][{POST_SNAPBACK}][/a]

Inter-subjective (collective) listening test is the answer to the problem.

RMAA Test on different Codec

Reply #28 –
Quote
Inter-subjective (collective) listening test is the answer to the problem.
[a href="index.php?act=findpost&pid=231077"][{POST_SNAPBACK}][/a]

Reading your post makes me think twice. You did have a valid points here: Collective. Hmm....


[Edited] Missing word in bold. Sigh.... why oh why I need to edit every of my posts...? 
Break The Rules!!!

RMAA Test on different Codec

Reply #29 –
Quote
Ok, after reading the link, and some discussion with my friend, more or less I get the picture of it. Thanks again.  

Now, the "inaudible" noise you mentioned here, from what I think, is truly subjective. Reason? Everyone has different ears.


...but the masking curves of most people’s ears are fairly similar. If you code to please the best, then everyone else will be happy too.


Quote
When our ears are different from one to another, the ability to capture the quantization noise from the product of psychoacoustic model is also different.


Slightly - not greatly. Anyway, it's kind of irrelevant. Like every other piece of psychometric data, you work with averages and percentiles. The data is only for simple sounds (pure tones, bandpass noise) so extrapolating this to complex time-varying musical signals is difficult. The only way to judge if the codec is doing this properly is to listen to the result. If you try to second guess the human ear, then you may get close - but your test will be as much a test of the accuracy of your ear model as it is a test of the accuracy of the codec.


Quote
Hmm.... so what is the best measurement tool here? While AP cannot measure how well the quantization noise handling by the codec, can it paint a overall behaviour picture by doing a frequency sweep?


The frequency sweep show you which frequencies are passed by the codec when encoding a very easy signal. That reveals any fixed low and high pass filtering, but that's about all.

If you try a constant pure tone, you can usually see the shape of the band pass filters within the codec, because they'll be noise 20-40dB down from the tone, spread in the shape of the filter and the psychoacoustic model's masking prediction.

These are interesting features of the codec, but they don't tell you what it sounds like.


Quote
Open for discussion, truly appreciate to those reply to me and clear my doubts.
[{POST_SNAPBACK}][/a]


I think you need to go back and read those links again - I still don't think you've "got it" - sorry!

Cheers,
David.

P.S. the chapter of my thesis which discussed this is here:
[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=24632]http://www.hydrogenaudio.org/forums/index....showtopic=24632[/url]

RMAA Test on different Codec

Reply #30 –
Quote
Hmm.... so what is the best measurement tool here? While AP cannot measure how well the quantization noise handling by the codec, can it paint a overall behaviour picture by doing a frequency sweep?

Open for discussion, truly appreciate to those reply to me and clear my doubts.


The best measurement tool for this community are the listening tests ... a group of users compares different codecs with different music styles by double blind testing (with cheating being made impossible) ... a statistical analysis of the results is being performed afterwards to determine the overall codec ranking.
The name was Plex The Ripper, not Jack The Ripper

RMAA Test on different Codec

Reply #31 –
RMAA does have a use with codecs though - to test if a decoder was working correctly (when compared against other decoders of the same type) - it will not tell you if one decoder is sounding better than another, but rather if one decoder was way off and not working correctly (the iPod on high bit rate mp3s showed this).

RMAA Test on different Codec

Reply #32 –
Your basic assumption is that a human hearing is a linear measurement device (i.e. like a measurement instrument, such Audio Precision Cacade One, except less accurate) and that audibility/audible quality can be assessed through means of objective technical measurements.

However, human hearing is not a linear and "well-behaved" measurement instrument.

Human hearing is inherently non-linear (on signal to excitation level). This means that what you can measure doesn't always correlate well with what you can hear (under the best conditions) or vice versa.

What is even more problematic for making comparisons between non-linearities of human hearing and measurements is that there are known 100% accurate mapping between these two domains (i.e. measuring with tools and hearing with your "ears").

That is, there is no 100% (or anywhere near 99.95%) accurate way to map a measurement into a listening result: can you hear something or can you not. Please bear now in mind that I'm completely  (for the sake of discussion) ignoring whether the listener "likes" or "prefers" something. I merely compare detection (detected by a measurement tool vs detected by a human hearing apparatus).

Because of these non-linearities and relatively rough mapping between measurements and hearing results, a lot of research has gone into trying to find how humans hear.

This field is a study of psychoacoustics and it measures sound in two domains: both as an acoustic signal (e.g. 1 kHz simple sinusoid at 40 dB) and as a human hearing event (i.e. 1000 mels).

The first measurement (acoustics) can be done with technological measurement instruments.

The second can only be done by listening with the human hearing apparatus.

Then it's a matter of combining these two slowly, by doing very low level small tests with hearing (first with simple clicks, tones and longer sounds and then further with complex sounds, multiple sound sources, etc).

However this process of combining the two (what is a sound as an acoustic event and what is that which the human hears) measurements is "only" roughly  130 years old as a scientific discipline.

As such, still much remains unknown about what we hear (compared to what we measure) and when do we hear it (under what acoustic conditions).

So, to make a long story short: what people have written about using your ear as a tool to measure "audible performance" (as opposed to signal quality measured performance) is true.

While there do exist some attempts at "objective measure" of sound quality (i.e. mimicking how humans hear through the use of algorithms), these are still very much at rudimentary stages and do not perform anywhere like human hearing for more complex signals (multisource complex tones or more interestingly, music and normal acoustic recordings).

I hope this illustrates the problem of hearing vs. measuring a little bit more.

Best regards,
Halcyon

PS A lot of what is true for hearing is true for vision as well. We may think that our sense portray an objective truth about the world we live in or that what we can see or not see, is easily measured. However, this is not the case. Seeing, as much as hearing, has things like masking, constancy, temporal and other effects which makes our vision very much a badly known and ill-behaving measurement device (even on very low level tasks which do not curry the burden of higher level semantic reasoning, such as meaning making about objects we build up in our visual array).

RMAA Test on different Codec

Reply #33 –
Just a quick question, when you all conducting a listening test, did you include the original wave file as reference?

I will going through David's document before I make any reply. 

But one thing to clarify, Halcyon, I didn't make such assumption where a human hearing is a linear measurement device. If I do, kindly point out for me.
Break The Rules!!!

RMAA Test on different Codec

Reply #34 –
the whole thing can be explained without alot of mathematical terms:

lossy codecs are for listening - not for watching. It is about how the music sounds to you - not how it looks like.

So, mid-summary: it is unimportant how the music looks like - its only important how it sounds like.

That point being made, how to test how "transparent" a lossy audiofile >sounds< compared to the original source? Well, the most accurate tool to measure "perceived quality" are your ears. At the same time, your ears are the "target audience" - so its the direct way to measure perceived quality.

However, your ears can be tricked quite easily. The brain can make you imagine (and hear) things which aren't really there (placebo-effect) - so we need a counter-tool to exclude this error-source. The tool to do this is statistics and blind-listening tests.

So, you rank the transparency of a lossy codec with your ears - and minimize the possible danger of placebo-effects with a blind-listening tests. And by doing this multiple times you gain some statistics and therefore the "reliability" of the results increase.

Yes, a blind-listening test isn't 100% accurate and secure - but then again, nothing in the world is 100% for sure - "proven" in scientific terms only means "with such a high propability, that the risk of a false asumption is neglible".

- Lyx
I am arrogant and I can afford it because I deliver.

RMAA Test on different Codec

Reply #35 –
Quote
lossy codecs are for listening - not for watching. It is about how the music sounds to you - not how it looks like.
[a href="index.php?act=findpost&pid=231292"][{POST_SNAPBACK}][/a]


...but if the two were correlated (as they were to some extent with conventional audio equipment) then the tests would be worth doing, and could tell you something. It's just that, with audio codecs, because of their design, the measurements aren't correlated with what you hear.

To suggest that measurements are useless because we listen to audio isn't fair or correct.  Measurements can be useless for audio codecs because the codec is playing a trick on the human ear, and the measurement is measuring the wrong thing.

You could argue that every audio device is trying to trick the human ear, but only codecs try to give good subjective results and bad objective results. Without DSP to force the two to be different, they're actually closely linked! You can argue that the two aren't perfectly linked, but you know that a loudspeaker with a poor measured frequency response will sound like it subjectively has a poor frequency response. Compare a telephone speaker to your hi-fi - the measurement tells it like it is!


"We can't measure, we must only listen" is true with audio codecs. When you start to apply it to other audio equipment, you go down the rout of audiophile madness.

Cheers,
David.

RMAA Test on different Codec

Reply #36 –
Actually even the best of audio components have to be discerned with listening (that is, the relative superiority of the best of equipment).

Why?

Because we don't know what are all the objective technical measurements we should do in order to distinguish what is the best objectively.

Let me give you an example from headphones.

There exists no set of measurement data for headphones that tells how a perfect headphone should measure (if you do think it exists, please provide a reference, I'd be glad to revise my view on this).

In fact, listening tests have shown that headphones with less than ideal measurements from a set of professionally done dummy-head measurements can sound very good (to a set of trained analytical ears trying to assess accuracy, not personal sound preference).

Then again, a pair that measures much better according to some objective design ideals can sound really off, unnatural and even annoying [Ref: Finnish Hi-fi magazine headphone reviws 2002 (with dummy head measurements done the way they should be done)]

As such, listening is something we have to do even with audio equipment, although I do agree that completely forgetting measurements can quickly lead to absurdity.

regards,
Halcyon

RMAA Test on different Codec

Reply #37 –
Sorry for the late reply.

And David, I download your document but no time to print out for study. I am sorry for that. Thus, no comment by now.

And halycon, I understand where you come from, and I do agree with some of your opinion, but not all. Take an example, you bringing two high end set of amplifier, be it Mark Levinson or Krell, definately some will prefer ML and some prefer Krell, coz listening is subjective to preference. But in term of music reproduction, which simply means to reproduce the signal as close as original source, measurement still the way to go.
Break The Rules!!!

RMAA Test on different Codec

Reply #38 –
Quote
Take an example, you bringing two high end set of amplifier, be it Mark Levinson or Krell, definately some will prefer ML and some prefer Krell, coz listening is subjective to preference. But in term of music reproduction, which simply means to reproduce the signal as close as original source, measurement still the way to go.
[a href="index.php?act=findpost&pid=233067"][{POST_SNAPBACK}][/a]


The problem with comparing audio codecs with audio electronics is that people judge electronics on many factors besides sound quality. I suspect that people who buy audio electronics rarely do a double-blind level-matched ABX test in their listening room before making a purchase. They're choosing Krell over Mark Levinson or vice versa as much based on their feelings about the gear's reputation, looks, or price as its sound.

A measurement graph of a lossy codec can't tell you how it sounds to your ears. What looks like the most accurate reproduction of the original file isn't necessarily the one that sounds the most accurate. The goal is to produce the most accurate sounding (to human ears) reproduction of the source possible for a given file size limitation, regardless of what a graph might indicate.

As an example, a sound that is far beyond the range of human hearing can be easily measured and displayed on a graph, yet its presence in a recording wouldn't affect how it sounds to you.
 

RMAA Test on different Codec

Reply #39 –
Quote
The problem with comparing audio codecs with audio electronics is that people judge electronics on many factors besides sound quality. I suspect that people who buy audio electronics rarely do a double-blind level-matched ABX test in their listening room before making a purchase. They're choosing Krell over Mark Levinson or vice versa as much based on their feelings about the gear's reputation, looks, or price as its sound.

A measurement graph of a lossy codec can't tell you how it sounds to your ears. What looks like the most accurate reproduction of the original file isn't necessarily the one that sounds the most accurate. The goal is to produce the most accurate sounding (to human ears) reproduction of the source possible for a given file size limitation, regardless of what a graph might indicate.

As an example, a sound that is far beyond the range of human hearing can be easily measured and displayed on a graph, yet its presence in a recording wouldn't affect how it sounds to you.
[a href="index.php?act=findpost&pid=233087"][{POST_SNAPBACK}][/a]
I am sorry that the example I made is to reply Halycon's latest post.

And yes, I know well what you say and I agree with you. And no, my example is not related to the codec but halycon's comment.

Hope that clear your doubt and any misunderstanding.


[Edited] Add last three words.
Break The Rules!!!