Listening tests sample selection
Reply #1 – 2005-08-10 14:02:35
comments on Answer A: But, by design also, it should sound worse on 'non-complex' parts of the music. That's not true - not always at least. Low bitrate doesn't mean lower quality. By experience, I know that parts which are encoded with VBR at higher bitrate (than average) are often worse than parts encoded at lower bitrate (than average). Why? Probably because complex parts are often too complex , and therefore additional bitrate allocated by the encoder is not enough to avoid distortions and artefacts. But again, this is not a law. Some VBR encoders have problem with VBR and they tend to lower bitrate with bad impact on audible quality with musical parts which, according to low bitrate, shouldn't be "difficult".Now, if you compare the second minute of the song only, you effectively compare lame@170 vs lame@128. Guess who will win? VBR. I suppose so. Fortunately yes, VBR should be better otherwise VBR would be pointless.Now, if you decide to compare the first minute of the song, you do compare lame@80 vs lame@128. Guess who will win? CBR. You can't say that without testing it before. Some parts are so cool for encoders that they could be transparent, even at 80 kbps, and even for MP3. A good VBR implementation will allocate 80 kbps frame only if it doesn't harm.This result can even be proved within the test linked above. The sample called Debussy clearly show that all VBR codecs considered it 'not complex'. .No, this test could only prove that MP3 and MPC don't have a perfect VBR model, but not that VBR mean low quality with low bitrate. In the same test, you have Vorbis which allocate 110 kbps only on the sample named ItCouldBeSweet. And guess what: Vorbis obtained the best note (4,90) despite of lower bitrate. comments on Answer B:B) 'on a larger sample, these parameters would give an average of 128kbps' How relevant the average bitrate on another sample is to the test at hand? If you tune your VBR settings to reach an average of 128kbps on a sample and then do your test on another sample, what was the point of tuning it in the first place? It's not correct. The VBR settings used in this test was not tuned to reach 128 kbps on average with the tested sample. The VBR commandline produces on average ~128 kbps with a large number of discs. That's why you can use this command line with every samples, even if average bitrate of the small sample gallery is close to 100 kbps or superior to 180 kbps.1. VBR codecs will react differently on a song, based on a criteria that we will define as "complexity". Therefore it sounds natural to me that a 'general purpoose' listening test would include a representative sample of the typical variations in this criteria. If you compare only complex songs, you will favor one behavior of a VBR codec, while dismissing the other. Here is the real problem. But if "complex" samples were used, it was to lower the difficulty of the test. Try to organise a collective test with non-complex samples, and you won't obtain enough results. Not at 128 kbps at least... However, the problem is still here: the previous listening test doesn't tell us how will react all VBR encoders with "non-complex" parts.2. The average bitrate of ALL codecs should reach the same bitrate on the test samples Ideally, probably. But you're forced to allow a margin of tolerence. Roberto had fixed this margin to 10% IIRC, which is an acceptable one, and in practice the deviation is even inferior to this (128 -> 136 kbps i.e. 6,25%). Few kbps won't significantly change the results: difference is very subtle, at least at mid/high bitrate. N.B. atrac3 doesn't offer 128 kbps encoding. 132 kbps is the closest one.