Flawed conclusions on WMA
Reply #11 – 2007-04-17 09:35:02
It's a general fair comparison problem with VBR mode. But there is also a more general problem of which encoder setting to use. It is not a fair comparision problem at all, because of the scenario to which we want to extrapolate such tests. Or in other words: the reason why we actually test. We do not test because of killer samples. We dont even test because of small samples in general. Who cares about that? What we use lossy codecs for is listening to music. We want to extrapolate the testresults (after accounting for the increased difficulty of the test-samples) to real-world scenarios. In the real world, users dont encode small killer-samples for listening. In the real world, users encode entire music collections. The target for extrapolation is entire music collections - or in other words: music overally. Our hypothetical target-scenario looks like this: - we have a 100gb hard-drive and N music albums. - we want to encode them all to a lossy format, filling up all the available space - now we want to know which codec will give us the most bang for the buck, quality/space wise Even though, this scenario is a bit contrieved, it is valid, because it is the only reasonable (and thus fair) way to test lossy VBR-codecs. VBR codecs dont target a specific bitrate. They target a certain quality. In an utopian world, tests would work the other way around: we would already have the perceptual quality of each codec and would just have to do the maths to see which codec can achieve a given quality over an entire collection, with the least diskspace used. Unfortunatelly, the real world works a bit different. However, it is also clear, that VBR is a very efficient and reasonable concept, and that it is therefore illogical and stupid to make them all use the same average bitrate for the testsamples - that would beat the purpose of VBR - part of what makes a VBR codec better than another VBR codec, is that it should recognize difficult to encode parts, and increase the bitrate to maintain a constant quality. So, our only choice left is a sort of compromise: take a music collection as large and varied enough as possible, then make it a testrule that all encoders used, must encode this collection at the same overall bitrate. Even though this is against what VBR is about, it leaves the encoder a lot of freedom - enough freedom, to for example recognize difficult to encode music genres and easy to encode genres. It also leaves them alot of freedom to do what they are best at with the difficult testsamples: adjust their bitrate. Comparing all testsamples at the same overall bitrate, is what would be really stupid and illogical. First it would fail to test how good a VBR codec is at being VBR. Second the results of the test would be totally meaningless, because it would have no relevance to the real world anymore. This has been discussed to death in the past. Get educated or STFU, damnit. - Lyx