I think separating music by very arbitrary and vaguely defined genres is misleading and pointless, because the codecs don't know or care how you define the material they're compressing. In fact they don't know what music is at all, they only know about basic sound components and characteristics such as frequencies, dynamics, amplitudes, and so on. You could as well just lump every genre together, because many of the artists listed thus far have tracks of different genres on the same album, dirtying the experiment and rendering the results void.
Mastering is a very important factor, especially since the begin of the loudness war. If you only use highly dynamically compressed sources, you really don't have to care much about the specific details listed above!
Because mastering of newly released music has become so important, i usually don't add whole cds to my test collection, but choose 3 songs, either randomly or by properties. Better more cds with possibly different masterings.Some codecs apply very special techniques, which can result in quite extreme compression improvements, but only for relatively few cds. But if you happen to have overly many in your test set, this will severily reduce the represenetativity of your test. Another point for a many-cds-few-songs-approach.
It would be very nice, if you could include this (quite small) set into your tests.
+ speech please.
I don't have much. I guess there aren't many lectures broadcast lossless to use?
SOUNDSHOCK: FM FUNK MADDNESS!! (Ubiktune: http://ubiktune.org/releases/ubi020-variou...m-funk-maddness )[…]I list SOUNDSHOCK additionally because it's so utterly unlike pretty much any other style listed. Compositionally-focused jazz-funk rendered entirely with FM instrumentation? It's like Zappa's Jazz From Hell, only actually enjoyable. It has the added bonus of being completely free.
Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music.
Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total.
P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development
SOUNDSHOCK [...]like Zappa's Jazz From Hell, only actually enjoyable.
why would you focus on improving compression for, say, 'clicky'-music in particular?
This is nothing short of a huge bitrate advantage for FLACCL, and it really makes WP looks bad in comparison even though WP compresses better in general. Maybe it has worse temporal resolution, maybe there's something else.