Once more: Time for a new lossless codec comparision?
Reply #32 – 2012-05-11 00:09:18
Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music. My problem with that approach is that "kind of music" is so vague you can have completely different bitrate ranges for different tracks that will be grouped together as of one kind, which is the main indicator of the differences the encoder works with. What's even worse, another person would group them differently. Music genres is one of the worst conversation topics as everybody has their own unique set of opinions on particular tracks, let alone albums and artists as a whole. I think choosing a lossless encoder for a certain "kind" of music will never have enough factual ground to substantiate the preference due to this inherently faulty methodology. And there goes perhaps the most horrible question: what if I—or anybody else—like music of more than one kind? :P Basically, what I'm doing at home to solve this problem is (re)compressing all lossless material that comes my way with FLACCL -11 and WavPacj -hx6; I take whichever ends up smaller. WavPack usually wins by 5–10 kbps per album, but sometimes loses to FLACCL (more often than not on highly tonal ambient and drone ambient), and sometimes it wins by over 20 kbps on a single album! All of that music is of "my kind", but what makes the two codecs perform so differently? That's the interesting bit. (I chose those two because they have the best compression/decoding speed ratio among codecs supported by Rockbox; I wish it were possible to use TAK there as well!)Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total. I can assemble a more specific list; just tell me how many tracks would be sufficient for a test. Cutting up samples would work too I suppose, that'll even make the experiment purer. Would, say, 20–30 thirty-second samples per group suffice? I expect in this case other people could help find a lot more representative examples, too. As for covering more use cases, yes, it's quite possible that my list isn't comprehensive and other groups can be formed. For instance, I haven't even touched live recordings and 24-bit material, and those have unique traits of their own.P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development If the "usual" compression is determined by genre, especially if the genre is inherited from the whole CD, such a system will never work for the reasons described in the first paragraph of this post.