Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Once more: Time for a new lossless codec comparision? (Read 32856 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Once more: Time for a new lossless codec comparision?

Reply #25
+ speech please.
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

Once more: Time for a new lossless codec comparision?

Reply #26
I think separating music by very arbitrary and vaguely defined genres is misleading and pointless, because the codecs don't know or care how you define the material they're compressing. In fact they don't know what music is at all, they only know about basic sound components and characteristics such as frequencies, dynamics, amplitudes, and so on. You could as well just lump every genre together, because many of the artists listed thus far have tracks of different genres on the same album, dirtying the experiment and rendering the results void.

Instead I would look for very particular, easily identifiable traits that actually show the differences between codecs. I would separate the test cases into the following groups, per-track.

1) vocal/choral/a capella with minimal-to-no musical accompaniment;
2) solo instrumental music (any solo albums will do);
3) high dynamic range orchestral music;
4) "natural" music—non-instrumental, non-percussive music filled with natural sounds such as rain, cicadas and leaf rustling (new age nature sound recordings, ambient artists such as Robert Rich and Alio Die);
5) "drone" music—slow, humming music revolving around low frequencies with little to no percussive elements (most of drone ambient, Quake OST, artists like Klaus Wiese, Inanna);
6) "beat" music—percussion-centric electronic dance music (various forms of techno, big beat, illbient, chopped&screwed, etc.);
7) "air" music—electronic music rich in high frequencies (predominantly trance, futurepop, 70s ambient such as Tangerine Dream);
8) "clicky" music—highly dynamic, filled with transient attacks and other stuff that transform codecs hate especially (basically anything from Raster-Noton, some Autechre, various glitch and clicks&cuts artists);
9) "wall of sound" music—noisy, "wet", abrasive, rich in all frequencies (various noise, shoegaze, drone doom, funeral doom);
10) "broken" music—fast-paced, snare drum and sawtooth wave-centric, sample-heavy genres (breakcore, brostep, drill & bass, and some of the stuff Aphex Twin did in late 90s will do);
11) lo-fi music—very old or artificially aged recordings, stuff that sounds like it was recorded on cheapo microphones (some Rangers, Boards of Canada, Aidan Baker tracks, etc.);
12) nicely compressed music—some well-mastered rock/metal albums (anything from MoFi will do);
13) poorly compressed music—poorly remastered versions of the albums in the previous group.

I think that kind of separation will give a much clearer insight into strong and weak points of particular codecs. It's not very important to represent particular music genres, rather it is important to represent different compositions of sound per se, or complexity thereof, or dynamics, that factor into bitrate distribution.
Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3. 

Once more: Time for a new lossless codec comparision?

Reply #27
I think separating music by very arbitrary and vaguely defined genres is misleading and pointless, because the codecs don't know or care how you define the material they're compressing. In fact they don't know what music is at all, they only know about basic sound components and characteristics such as frequencies, dynamics, amplitudes, and so on. You could as well just lump every genre together, because many of the artists listed thus far have tracks of different genres on the same album, dirtying the experiment and rendering the results void.

Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music.

Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total.

Another problem is that we don't know whether there might be important things missing in your list. I'm looking for something covering most 'usage cases', spanning all kinds of music (which is already pretty hard to accomplish) + pre-mix/pre-master stuff (I know of people saving their recordings in FLAC, usually 24-bit and having a lot of headroom) + speech. I'm not sure whether I can get those last two balanced: I don't have any speech samples and getting pre-mix stuff other than the classical/jazz stuff I record will be hard I guess.

P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development

Mastering is a very important factor, especially since the begin of the loudness war. If you only use highly dynamically compressed sources, you really don't have to care much about the specific details listed above!

That will be quite hard to balance I guess, because most music I have in my collection regarding metal, rock and pop is of the last 10 years. Hmm, I guess I'll reconsider the list

Quote
Because mastering of newly released music has become so important, i usually don't add whole cds to my test collection, but choose 3 songs, either randomly or by properties. Better more cds with possibly different masterings.

Some codecs apply very special techniques, which can result in quite extreme compression improvements, but only for relatively few cds. But if you happen to have overly many in your test set, this will severily reduce the represenetativity of your test. Another point for a many-cds-few-songs-approach.

Okay, I'll consider adding more CDs. Problem with choosing tracks is that the 'sound' can differ quite a lot between tracks and I was thinking having whole CDs (instead of just individual tracks, like most tests) this would be an easy way to cover a broader range of music.


It would be very nice, if you could include this (quite small) set into your tests.

Yes, I guess this is a nice set to include.

+ speech please.

I don't have much. I guess there aren't many lectures broadcast lossless to use?
Music: sounds arranged such that they construct feelings.

Once more: Time for a new lossless codec comparision?

Reply #28
I think you shoud include some mono music. People still listen to a lot of mono, and the encoder behavior on these files may be different (see this old thread: [a href='index.php?showtopic=37876']Wavpack compression ratio with mono music on CD[/a]).

I don't know what you should include. Maybe some pre-60's jazz (Ella Fitzgerald, Billie Holiday) or blues (Robert Johnson?). Or some mono Dylan or Beatles? A lot of mono classical is available too.

Once more: Time for a new lossless codec comparision?

Reply #29
The Electronic section is really not representative of some of the more esoteric (and arguably codec-challenging) material. Here are a few better alternatives, from various labels.

Aphex Twin - DrukQs or Squarepusher - Ultravisitor (Warp Records...)
Amon Tobin - Supermodified (Ninja Tune)
kid606 - Kill Sound Before Sound Kills You (Ipecac(!!))
SOUNDSHOCK: FM FUNK MADDNESS!! (Ubiktune: http://ubiktune.org/releases/ubi020-variou...m-funk-maddness )

The first three albums cover almost every single "quasi-genre" moozooh lists. Each individual album covers most of the quasi-genres. I support Nine Inch Nails on a similar basis. Prodigy overlaps too much stylistically with NIN, with NIN providing better stylistic coverage. Kill the double Daft Punk listing.  Shillex belongs in Pop. I want Shillex to GTFO my Hydrogenaudio and will aggressively resist adding him to any list of representative artists. There are so many better electro-house/dubstep acts, and most of them are legit and have been building the scene for years/decades, unlike Skrillex's bandwagon jump from screamo.

I list SOUNDSHOCK additionally because it's so utterly unlike pretty much any other style listed. Compositionally-focused jazz-funk rendered entirely with FM instrumentation? It's like Zappa's Jazz From Hell, only actually enjoyable. It has the added bonus of being completely free.

If we're aiming to cover dubstep and give publicity to someone, please focus on someone good, and not another one of the bandwagon-jumpers. Nero's too new. Starkey (one of my faves, but a bit newish)! El-B! Slaughter Mob! Kode9! Plastician! There are plenty of awesome acts that are producing incredibly diverse music that have been there for more than a couple years. Here are a few recommendations from across the spectrum of time:

Kode9 and the Spaceape - Black Sun (Hyperdub, 2011)
Grime compilation (Rephlex, 2004)
Starkey - NC-17 (Dead Homies, 2007)

I'm leaving out many of the early underground labels in favour of stuff I think might be more interesting to an encoder.

Once more: Time for a new lossless codec comparision?

Reply #30
Quote
I don't have much. I guess there aren't many lectures broadcast lossless to use?

maybe http://archive.org/details/ShakespeareLectures
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

Once more: Time for a new lossless codec comparision?

Reply #31
SOUNDSHOCK: FM FUNK MADDNESS!! (Ubiktune: http://ubiktune.org/releases/ubi020-variou...m-funk-maddness )
[…]
I list SOUNDSHOCK additionally because it's so utterly unlike pretty much any other style listed. Compositionally-focused jazz-funk rendered entirely with FM instrumentation? It's like Zappa's Jazz From Hell, only actually enjoyable. It has the added bonus of being completely free.

I was intrigued, and apparently this is relevant to my interests! So thanks.

Once more: Time for a new lossless codec comparision?

Reply #32
Depends on your goal of course. First of all, I want to update the comparison for the current versions of encoders and I want to do that with a well balanced mix of sources. However, when I did such a test some years ago, it became clear that there were quite some people interested in getting a more narrowed-down graph just for their kind of music.


My problem with that approach is that "kind of music" is so vague you can have completely different bitrate ranges for different tracks that will be grouped together as of one kind, which is the main indicator of the differences the encoder works with. What's even worse, another person would group them differently. Music genres is one of the worst conversation topics as everybody has their own unique set of opinions on particular tracks, let alone albums and artists as a whole. I think choosing a lossless encoder for a certain "kind" of music will never have enough factual ground to substantiate the preference due to this inherently faulty methodology. And there goes perhaps the most horrible question: what if I—or anybody else—like music of more than one kind? :P

Basically, what I'm doing at home to solve this problem is (re)compressing all lossless material that comes my way with FLACCL -11 and WavPacj -hx6; I take whichever ends up smaller. WavPack usually wins by 5–10 kbps per album, but sometimes loses to FLACCL (more often than not on highly tonal ambient and drone ambient), and sometimes it wins by over 20 kbps on a single album! All of that music is of "my kind", but what makes the two codecs perform so differently? That's the interesting bit. (I chose those two because they have the best compression/decoding speed ratio among codecs supported by Rockbox; I wish it were possible to use TAK there as well!)

Indeed, your classification would be better for identifying encoders strengths and weaknesses, but I'm not sure such a test would help a developer (why would you focus on improving compression for, say, 'clicky'-music in particular? Tests with specific (short) samples are much more helpful for optimization than whole "genres" which smooths 'difficult' samples) and as a user, I'm not really interested in such a categorization. Moreover, these definitions are (at least, to me) hard to turn into recommendations for certain tracks or albums, neither do they seem to add up to a balanced total.


I can assemble a more specific list; just tell me how many tracks would be sufficient for a test. Cutting up samples would work too I suppose, that'll even make the experiment purer. Would, say, 20–30 thirty-second samples per group suffice? I expect in this case other people could help find a lot more representative examples, too.

As for covering more use cases, yes, it's quite possible that my list isn't comprehensive and other groups can be formed. For instance, I haven't even touched live recordings and 24-bit material, and those have unique traits of their own.

P.S. What I just thought of, building some system testing individual tracks and returning a warning when a codec performs significantly worse or better than usual might be a nice way to help development


If the "usual" compression is determined by genre, especially if the genre is inherited from the whole CD, such a system will never work for the reasons described in the first paragraph of this post.

Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3. 


Once more: Time for a new lossless codec comparision?

Reply #34
why would you focus on improving compression for, say, 'clicky'-music in particular?
  Interesting case in point: just now I have compressed Ryoji Ikeda's Dataplex (a very representative example of this quasi-genre) with FLACCL and WavPack with settings as described above, and the difference is as follows:

    WavPack -hx6 — 283 385 585 bytes, 680 kbps;
FLACCL -11 — 269 342 146 bytes, 646 kbps.

    This is nothing short of a huge bitrate advantage for FLACCL, and it really makes WP looks bad in comparison even though WP compresses better in general. Maybe it has worse temporal resolution, maybe there's something else.

Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3. 


Once more: Time for a new lossless codec comparision?

Reply #36
This is nothing short of a huge bitrate advantage for FLACCL, and it really makes WP looks bad in comparison even though WP compresses better in general. Maybe it has worse temporal resolution, maybe there's something else.


An even more startling example: Cyclo. — Id. FLACCL: 763 kbps, WavPack: 824 kbps, a difference of 61 kbps (or 20 MB). Clearly WavPack's handling of this kind of waveform is grossly suboptimal.

Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3. 

Once more: Time for a new lossless codec comparision?

Reply #37
As much as moozooh's results make me curious, there is an 'on the other hand' to this:

Posting results in advance in this thread, kind of disrupts the design of the experiment. To the extent that this is a kind of competition between formats or encoders, then there is a bias in knowing the results before doing the selection. Then on the other hand, we are not breaking new ground here, we do have prior information, which is hard to avoid.

Picking problem samples might be entertaining, and for a developer it might be just what you want to know, and for the geek who reads actual tables rather than sorting them, it is interesting to see what are strengths and weaknesses. But choosing a representative test corpus for the purposes of averaging out a 'grand total' is a different purpose. And reality (probably) is, that the test results will be compiled to a 'grand total' with a 'winner' (maybe only by sortability of the table) – and if not by the person carrying out the test, then by somebody else.


Once more: Time for a new lossless codec comparision?

Reply #39
Hi guys,

It has been some time, but I decided to give this another try after I got stuck on this issue (balancing the test material) and suspended it for a few months. I have now chosen a different way: I took the list of genres for which Grammy's are being awarded as a starting point. I used this wiki-article.

So after a few days of trying to find the right CDs, I've got this new list in which I think every genre is fairly credited. The only ones I am missing are country, reggae, blues and perhaps gospel.

  • 30 Seconds to Mars - This Is War (rock)
  • Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
  • Daft Punk - Alive 2007 (electronic, live)
  • Dan Brown - Angels and Demons (audiobook)
  • Enya - Amarantine (new age)
  • Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
  • Gilberto Santa Rosa - Esencia (salsa)
  • Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (classical, romantic)
  • Howard Shore - The Hobbit: An Unexpected Journey (soundtrack, orchestral)
  • Iron Maiden - Brave New World (metal)
  • John Coltrane - Live at the half note, vol. 1 (jazz)
  • Joss Stone - Mind, Body & Soul (soul)
  • J.S. Bach - Magnificat (orchestral, baroque)
  • Koninklijke Militaire Kapel - [no name] (military brass, vinyl rip)
  • Kraftwerk - Autobahn (electronic)
  • Lana del Rey - Born to Die (pop)
  • Lil Wayne - Tha Carter III (hip hop)
  • Martin O'Donnell & Michael Salvatori - Halo: Reach O.S.T. (ambient-ish, soundtrack)
  • Michael Bublé - meets madison square garden (pop, live)
  • Mike Oldfield - Tubular Bells (not classifiable)
  • Rosenberg Trio - Djangologists (jazz)
  • Rush - Grace under Pressure (1997 remaster) (rock)
  • System of a Down - Mezmerize (metal)
  • Szakcsi - Virágom, virágom (world music)
  • Tiësto - In Search of Sunrise 7: Asia (dance)
  • Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
  • Various - Latin Village, CD 1: Salsa (compilation of salsa)
  • Xzibit - At the Speed of Life (hip hop)
  • Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism, soundtrack)


I've tried to add two entries for each 'genre' (which wasn't always possible or necessary), and tried to pick two entries that are not too much alike. After running the Dynamic Range Offline Meter it seems to me that both 'loudness war' items as 'non-loudness war' items are present equally.

Just to emphasis once more: I want to assemble a representative test corpus in terms of real-world usage which fits all possible users (at least, when considering CD-audio) equally well, not to find weak spots in certain encoders  Now the question to you: am I overlooking something, would you suggest any additions or removals etc.

Thanks again

P.S.: I've already begun with running tests, all scripts are working fine again, so I am confident I will publish results this time
Music: sounds arranged such that they construct feelings.