Re: Objective difference measurements to predict listening test results?
Reply #40 – 2016-09-22 18:42:22
That is not disturbing, not a surprise at all. It is exactly what xnor has said, and what you inadvertently (to you) asserted: You can only get a generic prediction like: "if the difference is below -20dB", the score is likely to be above 3". And that's all. Since you are not making an acoustic test (i.e. you are not judging multiple relevant aspects of the audio), you cannot get an absolute position, but instead a likely position. Concretely, the problem is as follows: The higher the difference it is, the more relevant this difference is against other audible artifacts, so it gets a higher importance when dedicing a result. The lower it is, the less likely it is to be the main problem, so you can get a broader range of results when judging it by ear. All audible artifacts happen ONLY because initial waveform has changed. There are no artifacts without changes in waveform. This is obvious I think. So, every artifact has its “tracks” in waveform, and any such “tracks” can be measured with Df. The real problem is that in general case there is no simple dependency between magnitude of the audible artifacts and magnitude of their “tracks” in waveform. In other words, smaller “tracks” can cause greater audible artifacts and vice versa. This is because not all “tracks” are equal or similar. And here I contend that if the “tracks” are similar then there is a simple dependency between magnitude of artifacts (quality scores) and the size of their “tracks” (Df). Example of such dependency can be seen in Fig.18. It is not a “likely position” at all, it is very clear relationship, the more samples used the clearer. Taking all this into account your explanation of the “disturbing surprise” seems to me unconvincing. I think quite the opposite – the lower Df (closer to -Inf) the shorter the range of results when judging it by ear, just because the artifacts become hard to be noticed at low difference. Anyway, the research is aimed exactly to test my hypothesis and if you are right we will see more such disturbing occurrences. But 3 cases are really insufficient to make any conclusions. BTW, from the results of Public MP3 Listening Test @ 128 kbps (October 2008) you participated in it. May be you still have somewhere the samples from that test. I found 11 of them and still missing 3. Sebastian told me that sfbay is San Francisco Bay Blues by Eric Clapton from Unplugged. So I know all the three tracks but it's necessary to find the parts used during the test. Probably I can figure out those problem parts by listening but it would be better to know for sure.