HydrogenAudio

Hydrogenaudio Forum => Listening Tests => Topic started by: salvator on 2004-02-16 16:03:24

Title: Alternative/Supplement to listening test
Post by: salvator on 2004-02-16 16:03:24
I'm new to this forum and I spent a bunch of time reading posts yesterday and today trying to absorb what I could from past discussions.  In light of the comments by other newbies about their difficulties hearing artifacts in some of the listening tests, I was thinking maybe there is a way to quantitatively compare files. 

This may already have been done and discussed to death.  If so, if someone could direct me to such discussions, I would appreciate it. 

Otherwise, this is how I imagine such a system working.  You start with some reference wave file.  You encode that into whatever format you are interested in and then decode it back into a wave file.  The comparison program you've written then compares the waveforms of the two files and creates some kind of measure for the deviation of the encoded file from the reference. 

The first and simplest measure that comes to mind is simply some kind of mean squared deviation per sample.  In this case, you read in the value of both waveforms at each sample and subtract one from the other and square this difference to avoid positive and negative deviations from summing to zero.  You sum all these differences over the length of your file and at the end you divide it by the number of samples in the file you used. 

At this point, it's not clear to me that this would be a meaningful metric.  There may be more meaningful, though more complicated ways to measure the deviation of the encoded file from the reference.  The ideal would obviously be if a metric could be defined such that there was a clear correlation between numerical deviation from the reference and listener-perceived deviation from the reference.

I know that ultimately, what we are looking for from a codec is not necessarily complete faithfulness in terms of reproduction of reference waveform, but rather psychoacoustic equivalence: we just want the encoded file to sound the same as the reference.  So, this type of analysis is not a replacement for a listening test, but it could be an interesting additional piece of information to have.

I don't feel like I have the expertise or the time to write such a program myself (at least not right now) but please let me know your thoughts on this.
Title: Alternative/Supplement to listening test
Post by: Garf on 2004-02-16 16:11:58
To provide a simple summary:

it's not possible (so far) to create an algorithm to determine how good a file sounds compared to the original, and does so reliably and as well as a human listener.

If better algorithms than what we have now were invented, they'd be used in the codecs, and your measure would be useless again.

A simple example is that there exists a PEQUAL (sp?) utility that uses psychoacoustic methods to determine how similar two files sound. But the psychoacoustics used in the encoders are much more advanced than this model, so the measure becomes useless because the codecs are smarter than the evaluation software.

I'm sure this has been discussed here before, too.
Title: Alternative/Supplement to listening test
Post by: rjamorim on 2004-02-16 16:18:34
Quote
PEQUAL (sp?)

The utility is called EAQUAL. And the algorithm is called PEAQ

http://rarewares.hydrogenaudio.org/others.html (http://rarewares.hydrogenaudio.org/others.html)
Title: Alternative/Supplement to listening test
Post by: Pio2001 on 2004-02-16 21:32:28
Yes, this question is in the MP3 FAQ. However, Garf's answer being very synthetic and clear, I'm replacing the old FAQ link with this one.
You can still read more explanations here : http://www.hydrogenaudio.org/forums/index....t=ST&f=1&t=5838 (http://www.hydrogenaudio.org/forums/index.php?act=ST&f=1&t=5838)
Title: Alternative/Supplement to listening test
Post by: atici on 2004-02-16 22:46:36
And this (http://www.hydrogenaudio.org/show.php/showtopic/11286) discussion might be what you're looking for.
Title: Alternative/Supplement to listening test
Post by: salvator on 2004-02-17 02:12:32
Thanks for the links to previous discussions.  I'm glad that other people have thought about this stuff before and that I'm not way off base for thinking this way.  I actually mentioned to a friend yesterday that I would be interested in hearing the difference between a wave and what results after encoding and decoding.  Maybe I'll do this for fun anyway even though it's been done before.