Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Lossless Audio Codec Comparison '07 (Read 36864 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Lossless Audio Codec Comparison '07

I've just completed and compiled a formal comparison of the major players in the lossless audio arena, including, but not limited to, FLAC, MPEG-4 ALS, and iTunes ALAC. Results, along with pretty chart, here, but also pasted below for your convenience.





La takes the cake, achieving 65.33% compression -- the closest to half off you're going to get. It should also be noted that MPEG-4 ALS is heavily based on LPAC, thus, it makes perfect sense for them to achieve identical results. FLAC, ALS, and ALAC are all pretty much on even footing, and right around the midpoint as far as the results go. I'm still surprised at how terrible bzip2 did. I guess it's just not suited for this kind of data the way it is for text and source code.

If I've left any out, let me know.

Oh, and watch the links in the Codec Comparisons section of the site -- they're not all functioning yet and none of my other comparisons are up yet.

Lossy audio coming soon.

Oh, and mods, feel free to move this to the lossless section. I posted it here because it doesn't fit into any of the individual subforums, but rather all of them as it involves FLAC, WavPack, and others.

Lossless Audio Codec Comparison '07

Reply #1
Lossy audio coming soon.


What will you compare among lossy codecs?

Also, what settings did you use for each lossless codec, and what about speed comparisons? These days I consider size to be about the least important characteristic in lossless because they are not that different, and storage has become so cheap.

Lossless Audio Codec Comparison '07

Reply #2
Quote
Highest compression/speed used on all encoders except ALS due to minimal size difference, and 7z due to absurd compression times (1 day)


Lossless Audio Codec Comparison '07

Reply #3
What will you compare among lossy codecs?
The usual. Vorbis, MP3(pro), MP2, (HE)AAC, Musepack, WMA, Cook. Multiple encoders for each format, of course (ND, iTunes, FAAC etc). I think I named them all but my memory might be lapsing.

Lossless Audio Codec Comparison '07

Reply #4
I think he means how will you compare the lossy codecs? Obviously this can only be done subjectively.

Lossless Audio Codec Comparison '07

Reply #5
One wave from one album is hardly a fair test battery since different encoders perform differently on different genres.  Without noting the encoding time there's no real frame of reference for comparison... compression ratios aren't the whole story.  What criteria did you use to conclude that "Liquid Audio takes the cake" since it saved less than a megabyte over Monkey?  Why does your graph state that LA achieved 65.61% compression but your text says 65.33%?  As tests go, this one is lacking a significant amount utility in the end.

Lossless Audio Codec Comparison '07

Reply #6
IIRC Winzip uses wavpack, otherwise I highly doubt a standard zip and wavpack would have the same ratio. I like how your link for TAK links to the wikipedia article when clearly in the external links section there is a link to the official site www.thbeck.de/Tak/Tak.html
[span style=\'font-size:8pt;line-height:100%\']"We will restore chaos"-Bush on Iraq[/span]

Lossless Audio Codec Comparison '07

Reply #7
With what options did you test? MP4ALS is not based on LPAC, and should be better than Monkey's Audio.

Lossless Audio Codec Comparison '07

Reply #8
Tough crowd :|
Quote
I think he means how will you compare the lossy codecs? Obviously this can only be done subjectively.
Right, that's the idea. Listening test.
Quote
One wave from one album is hardly a fair test battery
Yeah, about that. It's been brought up elsewhere. I'm adding a few more discs/genres to the test soon. I probably should've considered doing that before I posted the results.
Quote
Without noting the encoding time
This comes up a lot too. The thing is, speed isn't important to me here. I'm not testing for that. Plus, I used the slowest settings for each encoder, so comparing speed would be slightly unfair.
Quote
What criteria did you use to conclude that "Liquid Audio takes the cake" since it saved less than a megabyte over Monkey?
Uh... maybe the fact that it saves space over Monkey? How is the difference relevant to choosing a winner? Besides, with enough albums, less than a megabyte can become a few hundred.
Quote
Why does your graph state that LA achieved 65.61% compression but your text says 65.33%?
Man, what in the blazes are you talking about? The graph data is exactly the same as the text.
Quote
As tests go, this one is lacking
Thanks. I love you too.
Quote
MP4ALS is not based on LPAC
Audiocoding.com begs to differ.
Quote
* Reference model for new MPEG-4 Lossless Audio Coding / ALS
Quote
With what options did you test?
Like I said, best compression in all cases except ALS, with which I used the default reference encoder values. I did the same encode later with the heaviest compression settings, but the filesize difference was minimal. I'll consider doing this with the rest of the samples given my comment earlier about "hundreds of megabytes", as right now it seems a bit hypocritical.
Quote
I like how your link for TAK links to the wikipedia article when clearly in the external links section there is a link to the official site www.thbeck.de/Tak/Tak.html
You mean the official site that's in German? Yeah. That's useful to English speakers trying to find out what the hell TAK is.


Lossless Audio Codec Comparison '07

Reply #10
Quote
Why does your graph state that LA achieved 65.61% compression but your text says 65.33%?
Man, what in the blazes are you talking about? The graph data is exactly the same as the text.


Quote
La takes the cake, achieving 65.33% compression -- the closest to half off you're going to get.



Lossless Audio Codec Comparison '07

Reply #11
[quote name='PHOYO' date='Jul 1 2007, 10:59' post='501889'][quote name='Tab' post='501875' date='Jul 1 2007, 16:45']
Quote
Quote
La takes the cake, achieving 65.33% compression -- the closest to half off you're going to get.
Ah, good eye. I'll have to fix that. I still don't see what it has to do with the graph, though...

Lossless Audio Codec Comparison '07

Reply #12
Tough crowd :|

Or a crowd that just needs tests like these to be fair and useful.

This comes up a lot too. The thing is, speed isn't important to me here. I'm not testing for that. Plus, I used the slowest settings for each encoder, so comparing speed would be slightly unfair... Uh... maybe the fact that it saves space over Monkey? How is the difference relevant to choosing a winner?

You have to recognize the reality that if encoding an album takes 4 minutes with Monkey's Audio and 5 hours with Liquid Audio, then declaring that Liquid Audio has "taken the cake" is really quite foolish since very few people would be happy with that.  Hence the near total absence of LA as a current format.  You keep talking about what's important to you, but if you're going to do a test for everyone else, then you need to keep in mind that they require a frame of reference for the compression ratios, otherwise it's just useless.  If I could design an encoder that obtained a 65.2% compression ratio but took over 3 years to do the compression, I would expect that people would have an issue with me posting the ratios and declaring that my encoder has "taken the cake."  It's hardly unfair to post how long the encodings take... it's an integral part of determining what encoder a person wants to use (which should be the focus of such testing).

Man, what in the blazes are you talking about? The graph data is exactly the same as the text.

As PHOYO has already pointed out, you're the one not observing your own work here.  However, your response certainly is indicative of your helpfulness to the larger group.  The table is part of your graph... the only part that has any meaning to anyone since the bar-chart uses repetitive colours and is unlabeled.  One can therefore only conclude that the graph does not in fact equal the text.

As QHOBBES 2.0 has also pointed out, it appears that you have used WinZIP and attributed its compression success as indicative of the ZIP format in general which is misleading.  Since WinZIP uses WavPack to compress *.wav files, there probably isn't much purpose to its inclusion.

I applaud testing that can help people determine what software will best meet their needs.  The criticisms in this thread are largely because your test - as is - lacks the information that would help people make an informed decision.  To be valuable information, it must be useful information.  Rather than react defensively and somewhat juvenile, maybe you want to approach things in a spirit of learning and growth thereby enhancing the community around here.  I think that's all that anyone is getting at.


Lossless Audio Codec Comparison '07

Reply #14
Like I said, best compression in all cases except ALS, with which I used the default reference encoder values. I did the same encode later with the heaviest compression settings, but the filesize difference was minimal.


What settings did you try as "heaviest compression settings"?

Lossless Audio Codec Comparison '07

Reply #15
Quote
I've just completed and compiled a formal comparison of the major players in the lossless audio arena, including, but not limited to, FLAC, MPEG-4 ALS, and iTunes ALAC. Results, along with pretty chart, here


Thanks, but 2 problems:

- LA 228 65.61 | APE 229 65.61 ??? Also, you should calculate the "%" from exact sizes, not round to MB first

- ZIP is a good as WAVPACK ???  Typo or fraud ?

Quote
IIRC Winzip uses wavpack, otherwise I highly doubt a standard zip and wavpack would have the same ratio.


Then it's not a standard ZIP ... it's very unfair to WAVPACK developers ... and I doubt that you can extract such a "ZIP" with anything except "official" WINZIP 12  WAVPACK developers did the work, and WINZIP company collects the money
/\/\/\/\/\/\

Lossless Audio Codec Comparison '07

Reply #16
Quote
IIRC Winzip uses wavpack, otherwise I highly doubt a standard zip and wavpack would have the same ratio.


Then it's not a standard ZIP ... it's very unfair to WAVPACK developers ... and I doubt that you can extract such a "ZIP" with anything except "official" WINZIP 12  WAVPACK developers did the work, and WINZIP company collects the money


Agree with you on the first part (the ZIP used should be something openable by ALL zip files not winzip's implementation), but not the second (wavpack developers did the work...). Winzip using wavpack has been discussed before in the forum, and the developer of wavpack has no problems with it. I guess we shouldn't either.


Anyway about the test, I think it's useful, but not displaying the encode and decode times would be very misleading. I know it's your test and you're free to do as you want, but people looking at the test will be misled if they use the test in its current form as a guide. I mean sure LA is the best compressing, but both encoding and decoding rates are downright pathetic. I don't think anyone here *listens* to the LA compressed songs--- just archival. I think at the very least, the encoding time should be published.

Lossless Audio Codec Comparison '07

Reply #17
Then it's not a standard ZIP ... it's very unfair to WAVPACK developers ... and I doubt that you can extract such a "ZIP" with anything except "official" WINZIP 12  WAVPACK developers did the work, and WINZIP company collects the money

You're absolutely right that the ZIP files using WavPack are only unzippable with the official WinZip 11, but hopefully other utilities will handle these also someday. After all, the specs are open and the software is free.

However, I'm not sure how this is so unfair to me. The license of WavPack certainly allows commercial products assuming proper credit is given (which is done here) and I think that the endorsement of WinZip can only help WavPack's status as a robust product. It certainly can't hurt.

Yeah, it would be nice to get some of the money that they presumably make, but that's not the idea of writing free software. The world wouldn't have been better off if they got one of their programmers to design a new algorithm (unless it was better than WavPack, which is not too likely considering how much work making a good compressor is).

Lossless Audio Codec Comparison '07

Reply #18
You're absolutely right that the ZIP files using WavPack are only unzippable with the official WinZip 11, but hopefully other utilities will handle these also someday. After all, the specs are open and the software is free.
Well, is there any known way to force it to use standard zip? If so, I'll update the comparison.

 

Lossless Audio Codec Comparison '07

Reply #19
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul.  Check it out:  SS's lossless comparison

Lossless Audio Codec Comparison '07

Reply #20
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul.  Check it out:  SS's lossless comparison


SSoul's comparison is nice. But if Tab can provide his computer specs as well as the encoding/decoding speeds, it'd be a really helpful addition.



Lossless Audio Codec Comparison '07

Reply #23
You're absolutely right that the ZIP files using WavPack are only unzippable with the official WinZip 11, but hopefully other utilities will handle these also someday. After all, the specs are open and the software is free.
Well, is there any known way to force it to use standard zip? If so, I'll update the comparison.

use another program. I believe that KZIP offers the best compression for zip files. It beats 7-Zip by a few bytes.

http://advsys.net/ken/util/kzip.exe

Lossless Audio Codec Comparison '07

Reply #24
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul.  Check it out:  SS's lossless comparison
Much appreciated.

SSoul's comparison is nice.
Nice?!!  You philistine, it's f***ing marvelous!

SSoul's  comparison is nice. But if Tab can provide his computer specs as well  as the encoding/decoding speeds, it'd be a really helpful addition.
In all seriousness, I totally agree. 

Firstly, I do not test strong compression settings, or LA at all.  My test was originally to compare TAK to other, relative, formats.  Also very few people seem interested in compressing every last MiB, and are more concerned with the compression/speed trade-off.

Secondly, the more (good) comparisons the better, IMHO.  Recent testing with the updated WavPack - where my machine showed little improvement although many newer systems did - hit home to me that my test is not reepresentative of everyone's PC, only mine.  The more data we have, the more accurate results we can get.

I would like to know the exact settings used for this test, and see more sources being tested.
I'm on a horse.