HydrogenAudio

Lossless Audio Compression => Lossless / Other Codecs => Topic started by: ktf on 2012-04-27 13:30:50

Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-04-27 13:30:50
Hi all,

The recent activity on the FLAC-dev mailing list (it's alive again!) has reminded me of something I announced some time ago here but didn't really finish (http://www.hydrogenaudio.org/forums/index.php?showtopic=71612): make a new comparison of lossless codec performance, because most tests are pretty outdated or not very comprehensive. While there isn't much development at the moment (which I pity) especially TAK has developed and I hope FLAC maintenance development will soon turn to improvement.  As I still have all code of last time I made a comparison (I got it cleaned up and running in about 2 hours) the hardest thing to get right is the selection of music. That's why I set up this thread.

Test structure
I will make several comparisons, as last time it became clear that codecs perform very different when given different material. This asks for several tests for different genres. Moreover, the overall test has to be balanced too. While my taste in music is quite diverse, I hope you can help me cover the voids. I've set up some categories, Heavy (mostly metal), Rock, Electronic, Pop, Jazz, World Music, Orchestral, (Classical) Chamber music, Single-instrument/voice and pre-mix material. The starred items will be included in the main test, I'll try to take 5 out of every 'main' genre. Every item (that is, every album) is equally weighted in the final comparison.

Your opinion is important!
So, first of all, which 'main' genres are missing and which categories would you like to be included in the test? If you suggest a genre of category, please recommend me some CD's to buy which more or less 'span' the whole genre. Second, do you think these lists 'span' the whole genre depicted? Please keep in mind it is of course impossible to include every sub-genre, max 5 per genre would be nice.

Heavy
*Mercenary - 11 Dreams
*Dream Theater - Octavarium
Nightwish - Oceanborn
Linkin Park - Meteora
*System of a Down - Mezmerize
*Apocalyptica - Inquisition Symphony
*Metallica - Death Magnetic
Slipknot - All Hope is Gone
Blind Guardian - A Twist in the Myth

Rock (a.k.a less heavy)
*Paramore - Brand new Eyes
*Red Hot Chili Peppers - Stadium Arcadium
*Foo Fighters - Wasting Light
*Nickelback - Black Horse (Onder heavy?)
*Guns and Roses - Chinese Democracy
30 seconds to Mars - This is War
Muse - H.A.A.R.P. (live from Wembley)

Electronic
*Daft Punk - TRON Legacy R3CONF1GUR3D
*Daft Punk - Discovery (which is quite a different subgenre compared to previous entry)
*Tiësto - In Search of Sunrise 7: Asia
*The Prodigy - Invaders must Die (?)
*NIN - The Slip (Is dat Electronic?)

Pop
*5CD's of various popsongs
*Duffy - Endlessly
*Coldplay - Viva la vida or death and all his friends
*Katie Melua - The Katie Melua Collection
*Avril Lavigne - Let Go
BLØF - Umoja

Jazz, Soul
*The Rosenberg Trio - Djangologists (sample: http://www.youtube.com/watch?v=uOC-xZOWFnc) (http://www.youtube.com/watch?v=uOC-xZOWFnc))
*John Coltrane - Kind of Coltrane (Live at the half note '63, Vol 1 + 2)
*Joss Stone - Mind, Body and Soul
(I need some more here)

World music/Folk (very hard to define indeed  )
*Various - World Music Instruments, Single Reed Instruments (Clarinet)
*Fanfare Ciocarlia - Baro Biao (gypsy brass band)
(please suggest additions)

Orchestral
*Verdi Requiem
*Carneval des Animeaux
Le Sacre du printemps / Petrouchka
*Bach, Magnificat + Mahler, Symfonie nr. 1
Holst - The Planets
*The Lord of the Rings - Two Towers (complete recordings)

Chamber Music
*2 CD's worth of chamber music
*Some unaccompanied chamber choir recording

Single instrument
*EBU Sound Quality Assessment Material (SQAM) CD
Some of my chamber music recordings (hints for myself: kamermuziekexamen + KaMu-blend)
Excerpts from Yann Tiersens Good Bye Lenin and Amelie Poulain
*Ehren Starks - Lines Build Walls

Not categorized (only to balance the overall test)
*BLØF - April (to add some slow, balad-like music)
*Halo: Reach soundtrack

*Some pre-mix, 24-bit material (chamber-music, orchestral music, jazz)

Finally, please don't get upset about the choices I made making this list, just tell what you think is wrong. My musical taste is mainly in classical, rock and metal, so I tried to do the others as well as I could. Music and it's divions into genres is subject of much debate, but that's not the point of this list.

Thanks in advance!
Title: Once more: Time for a new lossless codec comparision?
Post by: Haubi on 2012-04-29 15:56:09
Great idea! It's really time to make a new comparison to see the power of TAK against other lossless codecs and maybe to push the evaluation of FLAC and others!

I think, if we get more than 5 CDs per category, it shouldn't be bad for the test because the more stuff the merrier the comparison!
My opinion so far:

Heavy
-- Apocalyptica (very special kind of metal, that doesn't fit in this category)
+ Pantera - A Vulgar Display of Power (well known)
+ Nightwish (very famous)
+ Children of Bodom - Hate Crew Deathroll

Rock
+ Nirvana - Nevermind (rock reference)

Elektronic
+ The Prodigy - Invaders Must Die (One of them is a MUST)
+ The Prodigy - The Fat Of The Land (One of them is a MUST)
+ Skrillex - Scary Monsters & Nice Sprites


Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-04-29 18:34:52
I think, if we get more than 5 CDs per category, it shouldn't be bad for the test because the more stuff the merrier the comparison!

Problem is the run-length of this test. 40CD's (5 CDs of 8 genres) would have a run-length of ~ 40 hours and some of these codecs run at only 2 to 3 times realtime (it was ~ 1.4x on my older laptop, it will probably be about 2.5x with my new one) so running such a test would then take several days... of course, this can be split up in several runs, but if this test grows to, say, 100 albums, this test might take over a week. 

Quote
Heavy
-- Apocalyptica (very special kind of metal, that doesn't fit in this category)

Right, I'll put that one under the not categorized 'balancing' group.

Thanks for the additions. I'll take them into consideration.
Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-04-29 20:16:52
Noise? Like Merzbow? Hardly much compressible ... I can report some figures when I get home to my collection.

In the opposite direction: Bobby "Don't Worry, Be Happy" McFerrin has a live album called The Voice, which consists only of his voice a cappella.  An interesting one.


(Myself I have enough music to do a comparison off my own collection. What stops me, is that I am missing a test suite which can to the job automatically and reliably ... (the Good Thing to use, I guess, would be some Linux live distribution running from RAM, and which does not try to go online for updates and download them ...) -- plus the fact that I don't really think people care that much anymore.)
Title: Once more: Time for a new lossless codec comparision?
Post by: kornchild2002 on 2012-04-29 21:50:08
In terms of electronic music, I would look at throwing a dubstep band/artist in there like the previously suggest Skrillex.  I think Nero should be thrown in there with the album Welcome Reality.  To me, it spans various electronic sub-genres and kind of has them all covered.  As for NIN, I don't know if I would consider them electronic.  They have released some electronic sounding albums (Ghosts comes to mind) but they also have some industrial and metal elements.  I would also take Meteora out of the heavy category since it is more hard rock.  There are other entries featuring chug-chug-chug riffs.  Otep had a new album that came out a few months ago.  Marilyn Manson even has an album coming out which is supposed to be more of a return to his older style.

Lastly, I wouldn't put Death Magnetic on there and instead focus on a more traditional metal album such as Megadeth - Thirteen or Anthrax - Worship Music.  My main argument against Death Magnetic is that it is mastered pretty poorly and I don't think it is a good representation of what you would find in the genre.  Had Metallica not blown it out of proportion, I would keep it on there but I think a different album should be used that better represents the genre.

You may even want to think about throwing some country on there.  My experience with the genre is that it is very similar to pop but you may want to get all of your bases covered.
Title: Once more: Time for a new lossless codec comparision?
Post by: saratoga on 2012-04-29 22:07:33
(Myself I have enough music to do a comparison off my own collection. What stops me, is that I am missing a test suite which can to the job automatically and reliably ... (the Good Thing to use, I guess, would be some Linux live distribution running from RAM, and which does not try to go online for updates and download them ...) -- plus the fact that I don't really think people care that much anymore.)


Or just a perl/python/whatever script that depended on having the codecs installed on the command line.  Then you could install an interpreter and test stuff from Windows too. 

We used to have something like that for rockbox test files, where you'd just download one flac and a script would convert it into a half dozen different formats for testing.
Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-04-30 21:35:49
Lastly, I wouldn't put Death Magnetic on there and instead focus on a more traditional metal album such as Megadeth - Thirteen or Anthrax - Worship Music.  My main argument against Death Magnetic is that it is mastered pretty poorly


It might be interesting to see how the formats/encoders do fare on these brickwalled recordings. Suggestion: make a selection of those and label them separately as 'prisoners of loudness war' as if that were a searate genre. (Eat that, Lars Ulrich.)
Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-04-30 21:47:20
Or just a perl/python/whatever script that depended on having the codecs installed on the command line.  Then you could install an interpreter and test stuff from Windows too.


Sure, for size. For reliable results when it comes to speed, you depend on your OS not starting any CPU- or hard drive-intensive processes in the meantime.
Title: Once more: Time for a new lossless codec comparision?
Post by: IgorC on 2012-04-30 23:02:47
ktf,

it might worth to look for the most used lossless codecs here 2012 ripping/encoding poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=92660)
Title: Once more: Time for a new lossless codec comparision?
Post by: GeSomeone on 2012-05-01 16:47:32
Apocalyptica is orchestral, isn't it?
If you want to cover the more traditional "Electronic" you should consider (any) Kraftwerk, Klaus Schulze or (not any) Vangelis.
As for Jazz, maybe a "classic" like Miles Davis - Kind of Blue.
If you want to add traditional Country, any album from Alison Krauss+Union Station would do.
For chamber music how about Vivaldi's the four seasons (performance of your choice).

I'm not trying to come up with rare albums, rather the more common ones. It adds up quickly and, if necessary, you could bring it down again to a fewer, most specific for the genre selection.
You have already 2 live albums on the list, those might have as an added characteristic crowd noise/applause. That's OK, I would only suggest, not too many of those.
Title: Once more: Time for a new lossless codec comparision?
Post by: icstm on 2012-05-01 17:28:32
sorry to sound silly, but what are we hoping to find?
Surely we are only talking about ~5-10% compression differences?
So are we doing compression vs CPU usage? or vs Time taken (as a proxy?)
Title: Once more: Time for a new lossless codec comparision?
Post by: derty2 on 2012-05-01 17:36:34
IMHO, no test should be without the king of instruments --the church organ-- and here is a benchmark example that will raise the hairs on the back of your neck:

Helmut Walcha, playing "Bach - Toccata and Fugue in D minor BWV 565", on the the Organ of St. Laurens Church, Alkmaar, Holland (1956)

and it is one of the tracks on this CD at ArkivMusic.com (http://www.arkivmusic.com/classical/album.jsp?album_id=1110)
Title: Once more: Time for a new lossless codec comparision?
Post by: kwanbis on 2012-05-01 18:31:32
To make it more representative, I would add some salsa, mambo, and reggaeton.
Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-05-01 18:43:29
Noise? Like Merzbow?


I'm not into that genre at all... but AFAIK it's not really popular. I might be a nice testcase for encoders but this test is not really about that kind of extremes

Quote
In the opposite direction: Bobby "Don't Worry, Be Happy" McFerrin has a live album called The Voice, which consists only of his voice a cappella.

Thanks for the suggestion.


Quote
(Myself I have enough music to do a comparison off my own collection. What stops me, is that I am missing a test suite which can to the job automatically and reliably ... (the Good Thing to use, I guess, would be some Linux live distribution running from RAM, and which does not try to go online for updates and download them ...) -- plus the fact that I don't really think people care that much anymore.)


I run all tests from RAM, and I wrote the test in PHP. It depends on some linux-specific tools like time and du for measuring used CPU time and disk usage. I'll run all tests on an idle PC from a ramdisk (which is not easy to set up in Windows too)

In terms of electronic music, I would look at throwing a dubstep band/artist in there like the previously suggest Skrillex.  I think Nero should be thrown in there with the album Welcome Reality.  To me, it spans various electronic sub-genres and kind of has them all covered.  As for NIN, I don't know if I would consider them electronic.  They have released some electronic sounding albums (Ghosts comes to mind) but they also have some industrial and metal elements.  I would also take Meteora out of the heavy category since it is more hard rock.  There are other entries featuring chug-chug-chug riffs.  Otep had a new album that came out a few months ago.  Marilyn Manson even has an album coming out which is supposed to be more of a return to his older style.

I'll put NIN under heavy (labeled as industrial) and remove Meteora. I'll  consider your suggested additions

Quote
Lastly, I wouldn't put Death Magnetic on there and instead focus on a more traditional metal album such as Megadeth - Thirteen or Anthrax - Worship Music.  My main argument against Death Magnetic is that it is mastered pretty poorly and I don't think it is a good representation of what you would find in the genre.  Had Metallica not blown it out of proportion, I would keep it on there but I think a different album should be used that better represents the genre.

I was hoping my collections would suffice for most categories, and I don't have any 'trash metal' besides that. However, I'll take a look

Quote
You may even want to think about throwing some country on there.  My experience with the genre is that it is very similar to pop but you may want to get all of your bases covered.

As I live in The Netherlands, I'm not familiar with country. As far as I know it, it should be put under world music/folk, but again, I really don't know anything about country.

[...]
It might be interesting to see how the formats/encoders do fare on these brickwalled recordings. Suggestion: make a selection of those and label them separately as 'prisoners of loudness war' as if that were a searate genre. (Eat that, Lars Ulrich.)


That would probably be interesting test case indeed.


[...]
Sure, for size. For reliable results when it comes to speed, you depend on your OS not starting any CPU- or hard drive-intensive processes in the meantime.


With 'time' I can measure CPU and 'real' time elapsed. In the test, I fetch both and check whether they match to 'verify' my results.

it might worth to look for the most used lossless codecs here 2012 ripping/encoding poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=92660)


Thanks. I'll put a litte more focus on ALAC this time I guess

Woops, I see there's a limit to the number of quote blocks I can include in my post... strange?
Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-05-01 18:43:52
sorry to sound silly, but what are we hoping to find?
Surely we are only talking about ~5-10% compression differences?
So are we doing compression vs CPU usage? or vs Time taken (as a proxy?)


I don't think this is a silly question at all. I am not even sure if there is much need for this test at all, but if it is, then I think your suggestion is probably the best comparison to make, along with ranking by size (as if speed were no objective).

For example, in addition to comparing size, what about the following: make a regression line time vs size, and for each codec report -- relative to the regression line -- the compression gain (positive or negative) or the time gain; The latter is probably a better suggestion, as we have two time variables (encoding and decoding) -- then one could report that codec X encodes 1% slower but decodes 5% faster than average?
Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-05-01 18:50:20
I'll continue

Apocalyptica is orchestral, isn't it?


No. If it should be categorized, it is 'heavy' or 'chamber music'. They're only with four, that doesn't make an orchestra 

Quote
If you want to cover the more traditional "Electronic" you should consider (any) Kraftwerk, Klaus Schulze or (not any) Vangelis.
As for Jazz, maybe a "classic" like Miles Davis - Kind of Blue.
If you want to add traditional Country, any album from Alison Krauss+Union Station would do.
For chamber music how about Vivaldi's the four seasons (performance of your choice).


Thanks for the additions.

Quote
I'm not trying to come up with rare albums, rather the more common ones.

Which makes it easy for me to get hold of it. Thanks!

Quote
You have already 2 live albums on the list, those might have as an added characteristic crowd noise/applause. That's OK, I would only suggest, not too many of those.


Actually, most orchestral items are live as well, but there you won't hear much 'live noises'. Besides that, it is very relevant for the other genres. I'll keep it in mind, thanks.

sorry to sound silly, but what are we hoping to find?
Surely we are only talking about ~5-10% compression differences?
So are we doing compression vs CPU usage? or vs Time taken (as a proxy?)


When comparing between encoders (all encoding the same stuff) there is little interesting to see. This test will update the results, as the last comprehensive test I could find has been some years ago, so this will update these results.

However, last time I found that between genres, encoders can perform different. Wavpack for example, did worse on single-instrument music compared to the other encoders. Probably this will show up with other source material as well. This will point out area's of improvement for certain codecs.

As said earlier, most people won't care much anymore. My motivation was the recent 'awakening' of the FLAC-mailinglist

IMHO, no test should be without the king of instruments --the church organ-- ...

Oh, right, I forgot to add one. I'll use organ spectacular by Jean Guillou, which is IMO very nicely recorded.

To make it more representative, I would add some salsa, mambo, and reggaeton.

Can you suggest some albums?


sorry to sound silly, but what are we hoping to find?
Surely we are only talking about ~5-10% compression differences?
So are we doing compression vs CPU usage? or vs Time taken (as a proxy?)


I don't think this is a silly question at all. I am not even sure if there is any need for this. And if it is, then I think your suggestion is probably the best comparison, along with ranking by size (speed no objective).

Speed no objective? I really won't compress my files to the heaviest Optimfrog compression. While such a list would state it as top-ranked (3 percentpoint better than FLAC -8) decoding ate 50% CPU on my previous computer. Even in seeking you would notice a lag. Speed  still is an objective.

There is no need. I just want to make a fair comparison with all styles equally balanced. Current test seem to lack that.

Quote
For example, in addition to comparing size, what about the following: make a regression line time vs size, and for each codec report the compression gain relative to the regression line, or the time gain. (The latter is probably a better suggestion, as we have two time variables (encoding and decoding) -- then one could report that codec X encodes 1% slower but decodes 5% faster than average?)

That's an option, but the problem is choosing that line. Tastes differ (there seem to be people actually using the OFR-setting I mentioned above) and I guess the usual graph will supply more information.
Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-05-01 19:02:43
Speed no objective? I really won't compress my files to the heaviest Optimfrog compression
[...]


I didn't say I wanted anyone to subscribe to the 'speed no objective' point of view, but I still think that there should be a ranking by compression alone (ranked as if speed were no objective).


That's an option, but the problem is choosing that line. Tastes differ (there seem to be people actually using the OFR-setting I mentioned above) and I guess the usual graph will supply more information.

Regression, as computed from all the encoders?
Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-05-01 20:35:52
Regression, as computed from all the encoders?

I guess I don't understand  what you mean by regression...
Title: Once more: Time for a new lossless codec comparision?
Post by: db1989 on 2012-05-01 21:09:50
http://en.wikipedia.org/wiki/Linear_regression (http://en.wikipedia.org/wiki/Linear_regression)

[…] the problem is choosing that line.

Quote
In statistics, linear regression is an approach to modeling the relationship between a scalar dependent variable y and one or more explanatory variables denoted X.
Quote
In statistics, simple linear regression is the least squares estimator of a linear regression model with a single explanatory variable. In other words, simple linear regression fits a straight line through the set of n points in such a way that makes the sum of squared residuals of the model (that is, vertical distances between the points of the data set and the fitted line) as small as possible.
Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-05-01 22:06:27
http://en.wikipedia.org/wiki/Linear_regression (http://en.wikipedia.org/wiki/Linear_regression)


Thanks, I was thinking of the word regression as they use it in software engineering.

I wonder whether linear regression would be the method to choose. If you would sacrifice going from 200x realtime to 101x realtime for 1 %-point, would you sacrifice going from 100x to 1x for the same 1%-point extra gain? If I would draw this same straight line through this half-logarithmic graph (http://www.icer.nl/losslesstest/decodelog.png), how is it called?
Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-05-02 00:13:00
regression as they use it in software engineering.


In the very least it wasn't http://newagevillage.com/wiki/index.php/Past_Life_Regression (http://newagevillage.com/wiki/index.php/Past_Life_Regression) .
Title: Once more: Time for a new lossless codec comparision?
Post by: kwanbis on 2012-05-02 04:35:45
Can you suggest some albums?

Let me see what I can do.
Title: Once more: Time for a new lossless codec comparision?
Post by: TBeck on 2012-05-02 22:13:21
For quite some time i wanted to write kind of a manual for audio compression tests but never had enough time to do it properly...

Therefore only some quick remarks.

Common properties important for efficiency differences in arbitrary order:

1) Does a file contain relative silent parts?
  This for instance reveals weaknesses of the final residual coding stage.
  Typical files: Classic, Singer-Songwriter, LossyWav
2) Fast alterations of the signal characteristics?
    Reveals, how fast a codec adapts to changes.
  Typical files: Electronic, Percussion
3) Predominant tonal instrument or vocal?
  Benefits extremely from a good and long predictor.
  Typical files: speech, solos in classic, singer-songwriter, some jazz, (extremely) harpsichord
4) A special but not so rare case: Is a file lowpassed, maybe with a lower-bitrate-mp3-like spectrum?
  Some codecs don't provide sufficent accuracy for the predictors in this case.

Well, i do care for all these factors, because i want to tune Tak for every aspect. But obviously their importance depends on the individual taste and music selection.

How to create a representative test collection?

If i only knew...

But i know about some pitfalls:

Mastering is a very important factor, especially since the begin of the loudness war. If you only use highly dynamically compressed sources, you really don't have to care much about the specific details listed above!

Because mastering of newly released music has become so important, i usually don't add whole cds to my test collection, but choose 3 songs, either randomly or by properties. Better more cds with possibly different masterings.

Some codecs apply very special techniques, which can result in quite extreme compression improvements, but only for relatively few cds. But if you happen to have overly many in your test set, this will severily reduce the represenetativity of your test. Another point for a many-cds-few-songs-approach.

The newest version of Optimfrog for instance can detect increases of the volume performed during the mastering process, determine the amplification factor, reverse the amplification and code a much lower signal. This can sometimes result in more than 10 percent better compression! I know about it, because i have investigated a similar approach for TAK. I found most opportunities in older masterings (1980-1990) of classic music released by one german label.

Sorry, not enough time to go into more detail...

Hope it helps a bit

  Thomas



Title: Once more: Time for a new lossless codec comparision?
Post by: saratoga on 2012-05-03 00:04:42
Is there some collection of losslessly encoded, public domain music that would be suitable for testing?  Having the tracks freely available would certainly make duplicating and updating the test easier.
Title: Once more: Time for a new lossless codec comparision?
Post by: TBeck on 2012-05-07 20:20:10
My primary test set consists of all the files which still seem to be available at rarewares.org: 30-seconds-samples (http://www.rarewares.org/test_samples/)

I can't say they are representative for some special musical taste, but i found that they are affecting nearly any compression-relevant codec properties i could identify.

It would be very nice, if you could include this (quite small) set into your tests.
Title: Once more: Time for a new lossless codec comparision?
Post by: smok3 on 2012-05-07 21:22:27
+ speech please.
Title: Once more: Time for a new lossless codec comparision?
Post by: moozooh on 2012-05-07 22:13:02
I think separating music by very arbitrary and vaguely defined genres is misleading and pointless, because the codecs don't know or care how you define the material they're compressing. In fact they don't know what music is at all, they only know about basic sound components and characteristics such as frequencies, dynamics, amplitudes, and so on. You could as well just lump every genre together, because many of the artists listed thus far have tracks of different genres on the same album, dirtying the experiment and rendering the results void.

Instead I would look for very particular, easily identifiable traits that actually show the differences between codecs. I would separate the test cases into the following groups, per-track.

1) vocal/choral/a capella with minimal-to-no musical accompaniment;
2) solo instrumental music (any solo albums will do);
3) high dynamic range orchestral music;
4) "natural" music—non-instrumental, non-percussive music filled with natural sounds such as rain, cicadas and leaf rustling (new age nature sound recordings, ambient artists such as Robert Rich and Alio Die);
5) "drone" music—slow, humming music revolving around low frequencies with little to no percussive elements (most of drone ambient, Quake OST, artists like Klaus Wiese, Inanna);
6) "beat" music—percussion-centric electronic dance music (various forms of techno, big beat, illbient, chopped&screwed, etc.);
7) "air" music—electronic music rich in high frequencies (predominantly trance, futurepop, 70s ambient such as Tangerine Dream);
8) "clicky" music—highly dynamic, filled with transient attacks and other stuff that transform codecs hate especially (basically anything from Raster-Noton, some Autechre, various glitch and clicks&cuts artists);
9) "wall of sound" music—noisy, "wet", abrasive, rich in all frequencies (various noise, shoegaze, drone doom, funeral doom);
10) "broken" music—fast-paced, snare drum and sawtooth wave-centric, sample-heavy genres (breakcore, brostep, drill & bass, and some of the stuff Aphex Twin did in late 90s will do);
11) lo-fi music—very old or artificially aged recordings, stuff that sounds like it was recorded on cheapo microphones (some Rangers, Boards of Canada, Aidan Baker tracks, etc.);
12) nicely compressed music—some well-mastered rock/metal albums (anything from MoFi will do);
13) poorly compressed music—poorly remastered versions of the albums in the previous group.

I think that kind of separation will give a much clearer insight into strong and weak points of particular codecs. It's not very important to represent particular music genres, rather it is important to represent different compositions of sound per se, or complexity thereof, or dynamics, that factor into bitrate distribution.
Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-05-10 21:04:30
I think separating music by very arbitrary and vaguely defined genres is misleading and pointless, because the codecs don't know or care how you define the material they're compressing. In fact they don't know what music is at all, they only know about basic sound components and characteristics such as frequencies, dynamics, amplitudes, and so on. You could as well just lump every genre together, because many of the artists listed thus far have tracks of different genres on the same album, dirtying the experiment and rendering the results void.

Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music.

Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total.

Another problem is that we don't know whether there might be important things missing in your list. I'm looking for something covering most 'usage cases', spanning all kinds of music (which is already pretty hard to accomplish) + pre-mix/pre-master stuff (I know of people saving their recordings in FLAC, usually 24-bit and having a lot of headroom) + speech. I'm not sure whether I can get those last two balanced: I don't have any speech samples and getting pre-mix stuff other than the classical/jazz stuff I record will be hard I guess.

P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development

Mastering is a very important factor, especially since the begin of the loudness war. If you only use highly dynamically compressed sources, you really don't have to care much about the specific details listed above!

That will be quite hard to balance I guess, because most music I have in my collection regarding metal, rock and pop is of the last 10 years. Hmm, I guess I'll reconsider the list

Quote
Because mastering of newly released music has become so important, i usually don't add whole cds to my test collection, but choose 3 songs, either randomly or by properties. Better more cds with possibly different masterings.

Some codecs apply very special techniques, which can result in quite extreme compression improvements, but only for relatively few cds. But if you happen to have overly many in your test set, this will severily reduce the represenetativity of your test. Another point for a many-cds-few-songs-approach.

Okay, I'll consider adding more CDs. Problem with choosing tracks is that the 'sound' can differ quite a lot between tracks and I was thinking having whole CDs (instead of just individual tracks, like most tests) this would be an easy way to cover a broader range of music.


It would be very nice, if you could include this (quite small) set into your tests.

Yes, I guess this is a nice set to include.

+ speech please.

I don't have much. I guess there aren't many lectures broadcast lossless to use?
Title: Once more: Time for a new lossless codec comparision?
Post by: BoraBora on 2012-05-10 21:47:24
I think you shoud include some mono music. People still listen to a lot of mono, and the encoder behavior on these files may be different (see this old thread: [a href='index.php?showtopic=37876']Wavpack compression ratio with mono music on CD[/a]).

I don't know what you should include. Maybe some pre-60's jazz (Ella Fitzgerald, Billie Holiday) or blues (Robert Johnson?). Or some mono Dylan or Beatles? A lot of mono classical is available too.
Title: Once more: Time for a new lossless codec comparision?
Post by: ExUser on 2012-05-10 22:00:03
The Electronic section is really not representative of some of the more esoteric (and arguably codec-challenging) material. Here are a few better alternatives, from various labels.

Aphex Twin - DrukQs or Squarepusher - Ultravisitor (Warp Records...)
Amon Tobin - Supermodified (Ninja Tune)
kid606 - Kill Sound Before Sound Kills You (Ipecac(!!))
SOUNDSHOCK: FM FUNK MADDNESS!! (Ubiktune: http://ubiktune.org/releases/ubi020-variou...m-funk-maddness (http://ubiktune.org/releases/ubi020-various-artists-soundshock-fm-funk-maddness) )

The first three albums cover almost every single "quasi-genre" moozooh lists. Each individual album covers most of the quasi-genres. I support Nine Inch Nails on a similar basis. Prodigy overlaps too much stylistically with NIN, with NIN providing better stylistic coverage. Kill the double Daft Punk listing.  Shillex belongs in Pop. I want Shillex to GTFO my Hydrogenaudio and will aggressively resist adding him to any list of representative artists. There are so many better electro-house/dubstep acts, and most of them are legit and have been building the scene for years/decades, unlike Skrillex's bandwagon jump from screamo.

I list SOUNDSHOCK additionally because it's so utterly unlike pretty much any other style listed. Compositionally-focused jazz-funk rendered entirely with FM instrumentation? It's like Zappa's Jazz From Hell, only actually enjoyable. It has the added bonus of being completely free.

If we're aiming to cover dubstep and give publicity to someone, please focus on someone good, and not another one of the bandwagon-jumpers. Nero's too new. Starkey (one of my faves, but a bit newish)! El-B! Slaughter Mob! Kode9! Plastician! There are plenty of awesome acts that are producing incredibly diverse music that have been there for more than a couple years. Here are a few recommendations from across the spectrum of time:

Kode9 and the Spaceape - Black Sun (Hyperdub, 2011)
Grime compilation (Rephlex, 2004)
Starkey - NC-17 (Dead Homies, 2007)

I'm leaving out many of the early underground labels in favour of stuff I think might be more interesting to an encoder.
Title: Once more: Time for a new lossless codec comparision?
Post by: smok3 on 2012-05-10 22:16:00
Quote
I don't have much. I guess there aren't many lectures broadcast lossless to use?

maybe http://archive.org/details/ShakespeareLectures (http://archive.org/details/ShakespeareLectures)
Title: Once more: Time for a new lossless codec comparision?
Post by: db1989 on 2012-05-10 22:35:09
SOUNDSHOCK: FM FUNK MADDNESS!! (Ubiktune: http://ubiktune.org/releases/ubi020-variou...m-funk-maddness (http://ubiktune.org/releases/ubi020-various-artists-soundshock-fm-funk-maddness) )
[…]
I list SOUNDSHOCK additionally because it's so utterly unlike pretty much any other style listed. Compositionally-focused jazz-funk rendered entirely with FM instrumentation? It's like Zappa's Jazz From Hell, only actually enjoyable. It has the added bonus of being completely free.

I was intrigued, and apparently this is relevant to my interests! So thanks.
Title: Once more: Time for a new lossless codec comparision?
Post by: moozooh on 2012-05-11 00:09:18
Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music.


My problem with that approach is that "kind of music" is so vague you can have completely different bitrate ranges for different tracks that will be grouped together as of one kind, which is the main indicator of the differences the encoder works with. What's even worse, another person would group them differently. Music genres is one of the worst conversation topics as everybody has their own unique set of opinions on particular tracks, let alone albums and artists as a whole. I think choosing a lossless encoder for a certain "kind" of music will never have enough factual ground to substantiate the preference due to this inherently faulty methodology. And there goes perhaps the most horrible question: what if I—or anybody else—like music of more than one kind? :P

Basically, what I'm doing at home to solve this problem is (re)compressing all lossless material that comes my way with FLACCL -11 and WavPacj -hx6; I take whichever ends up smaller. WavPack usually wins by 5–10 kbps per album, but sometimes loses to FLACCL (more often than not on highly tonal ambient and drone ambient), and sometimes it wins by over 20 kbps on a single album! All of that music is of "my kind", but what makes the two codecs perform so differently? That's the interesting bit. (I chose those two because they have the best compression/decoding speed ratio among codecs supported by Rockbox; I wish it were possible to use TAK there as well!)

Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total.


I can assemble a more specific list; just tell me how many tracks would be sufficient for a test. Cutting up samples would work too I suppose, that'll even make the experiment purer. Would, say, 20–30 thirty-second samples per group suffice? I expect in this case other people could help find a lot more representative examples, too.

As for covering more use cases, yes, it's quite possible that my list isn't comprehensive and other groups can be formed. For instance, I haven't even touched live recordings and 24-bit material, and those have unique traits of their own.

P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development


If the "usual" compression is determined by genre, especially if the genre is inherited from the whole CD, such a system will never work for the reasons described in the first paragraph of this post.

Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-05-11 08:12:54
SOUNDSHOCK [...]
like Zappa's Jazz From Hell, only actually enjoyable.


OK, that's a bait I'm downloading ...

... should hit some vibes with those who were playing computer games in the 80's ...
Title: Once more: Time for a new lossless codec comparision?
Post by: moozooh on 2012-05-11 09:26:07
why would you focus on improving compression for, say, 'clicky'-music in particular?
  Interesting case in point: just now I have compressed Ryoji Ikeda's Dataplex (a very representative example of this quasi-genre) with FLACCL and WavPack with settings as described above, and the difference is as follows:

    WavPack -hx6 — 283 385 585 bytes, 680 kbps;
FLACCL -11 — 269 342 146 bytes, 646 kbps.

    This is nothing short of a huge bitrate advantage for FLACCL, and it really makes WP looks bad in comparison even though WP compresses better in general. Maybe it has worse temporal resolution, maybe there's something else.

Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-05-11 20:52:02
SOUNDSHOCK [...]
like Zappa's Jazz From Hell, only actually enjoyable.


Had expected this to clock in at a lower bitrate, really :-o
Title: Once more: Time for a new lossless codec comparision?
Post by: moozooh on 2012-05-11 23:18:54
This is nothing short of a huge bitrate advantage for FLACCL, and it really makes WP looks bad in comparison even though WP compresses better in general. Maybe it has worse temporal resolution, maybe there's something else.


An even more startling example: Cyclo. — Id (http://www.discogs.com/Cyclo-Id/release/2744653). FLACCL: 763 kbps, WavPack: 824 kbps, a difference of 61 kbps (or 20 MB). Clearly WavPack's handling of this kind of waveform is grossly suboptimal.

Title: Once more: Time for a new lossless codec comparision?
Post by: Porcus on 2012-05-11 23:51:31
As much as moozooh's results make me curious, there is an 'on the other hand' to this:

Posting results in advance in this thread, kind of disrupts the design of the experiment. To the extent that this is a kind of competition between formats or encoders, then there is a bias in knowing the results before doing the selection. Then on the other hand, we are not breaking new ground here, we do have prior information, which is hard to avoid.

Picking problem samples might be entertaining, and for a developer it might be just what you want to know, and for the geek who reads actual tables rather than sorting them, it is interesting to see what are strengths and weaknesses. But choosing a representative test corpus for the purposes of averaging out a 'grand total' is a different purpose. And reality (probably) is, that the test results will be compiled to a 'grand total' with a 'winner' (maybe only by sortability of the table) – and if not by the person carrying out the test, then by somebody else.
Title: Once more: Time for a new lossless codec comparision?
Post by: Haubi on 2012-08-19 14:25:54
Here comes a suggestion for our comparison:

http://www.hydrogenaudio.org/forums/index....st&p=802586 (http://www.hydrogenaudio.org/forums/index.php?showtopic=96076&view=findpost&p=802586)
Title: Once more: Time for a new lossless codec comparision?
Post by: ktf on 2012-12-21 21:56:22
Hi guys,

It has been some time, but I decided to give this another try after I got stuck on this issue (balancing the test material) and suspended it for a few months. I have now chosen a different way: I took the list of genres for which Grammy's are being awarded as a starting point. I used this wiki-article (http://en.wikipedia.org/wiki/List_of_Grammy_Award_categories).

So after a few days of trying to find the right CDs, I've got this new list in which I think every genre is fairly credited. The only ones I am missing are country, reggae, blues and perhaps gospel.



I've tried to add two entries for each 'genre' (which wasn't always possible or necessary), and tried to pick two entries that are not too much alike. After running the Dynamic Range Offline Meter it seems to me that both 'loudness war' items as 'non-loudness war' items are present equally.

Just to emphasis once more: I want to assemble a representative test corpus in terms of real-world usage which fits all possible users (at least, when considering CD-audio) equally well, not to find weak spots in certain encoders  Now the question to you: am I overlooking something, would you suggest any additions or removals etc.

Thanks again

P.S.: I've already begun with running tests, all scripts are working fine again, so I am confident I will publish results this time