New Public Multiformat Listening Test (Jan 2014)
Reply #355 – 2013-12-28 20:46:52
1) Collecting large number of hard-to-encode (but musical) samples. 2) Grouping them by some attributes. For example by signal type (pure tone/transient/stereo), then in each group we may include representatives of each music genre to make testing even more objective. Also we must include into each group the samples with and without vocal, and so on. 3) After grouping we make a random selection of samples from each group (subgroup). This way we will get not so objective results as in first case (with analyzing of percentage ratio for each genre), but much more informative. After testing we can present not only average results for all samples, but also results for each group of samples, so we'll be able to evaluate behavior of each codec with each group (for example to compare which codec is better for transients). That's why we need to increase number of samples - because we must have at least few samples in each subgroup. I like this idea too. Maintaining a properly segmented bank of sound samples would be helpful for many audio researchers and enthusiasts. Along with killer-samples such bank can contain "ordinary" ones of different types - genres, voices, noisy/dirty, clipped ... . Some system of tags could be sufficient for the purpose. Then depending on the goal of the test samples with appropriate tags can be randomly selected. The use of ordinary (usual) sound samples for listening tests is a common practice especially for testing of low bit-rate encoders. For tests @96+ usual sound material is helpless. And this is another reason for not using in 96+ tests sample sets that are representative to some population of music - this ends up with large set of usual samples which are very hard to test @96+. On the other hand this approach could be interesting for testing @24-64.