I've just completed and compiled a formal comparison of the major players in the lossless audio arena, including, but not limited to, FLAC, MPEG-4 ALS, and iTunes ALAC. Results, along with pretty chart, here (http://www.brutus-music.com/Mike/ALossless.html), but also pasted below for your convenience.
(http://www.brutus-music.com/Mike/Images/acc.png)
(http://www.brutus-music.com/Mike/Images/cw.png)
La takes the cake, achieving 65.33% compression -- the closest to half off you're going to get. It should also be noted that MPEG-4 ALS is heavily based on LPAC, thus, it makes perfect sense for them to achieve identical results. FLAC, ALS, and ALAC are all pretty much on even footing, and right around the midpoint as far as the results go. I'm still surprised at how terrible bzip2 did. I guess it's just not suited for this kind of data the way it is for text and source code.
If I've left any out, let me know.
Oh, and watch the links in the Codec Comparisons section of the site -- they're not all functioning yet and none of my other comparisons are up yet.
Lossy audio coming soon.
Oh, and mods, feel free to move this to the lossless section. I posted it here because it doesn't fit into any of the individual subforums, but rather all of them as it involves FLAC, WavPack, and others.
Lossy audio coming soon.
What will you compare among lossy codecs?
Also, what settings did you use for each lossless codec, and what about speed comparisons? These days I consider size to be about the least important characteristic in lossless because they are not that different, and storage has become so cheap.
Highest compression/speed used on all encoders except ALS due to minimal size difference, and 7z due to absurd compression times (1 day)
What will you compare among lossy codecs?
The usual. Vorbis, MP3(pro), MP2, (HE)AAC, Musepack, WMA, Cook. Multiple encoders for each format, of course (ND, iTunes, FAAC etc). I think I named them all but my memory might be lapsing.
I think he means how will you compare the lossy codecs? Obviously this can only be done subjectively.
One wave from one album is hardly a fair test battery since different encoders perform differently on different genres. Without noting the encoding time there's no real frame of reference for comparison... compression ratios aren't the whole story. What criteria did you use to conclude that "Liquid Audio takes the cake" since it saved less than a megabyte over Monkey? Why does your graph state that LA achieved 65.61% compression but your text says 65.33%? As tests go, this one is lacking a significant amount utility in the end.
IIRC Winzip uses wavpack, otherwise I highly doubt a standard zip and wavpack would have the same ratio. I like how your link for TAK links to the wikipedia article when clearly in the external links section there is a link to the official site www.thbeck.de/Tak/Tak.html
With what options did you test? MP4ALS is not based on LPAC, and should be better than Monkey's Audio.
Tough crowd :|
I think he means how will you compare the lossy codecs? Obviously this can only be done subjectively.
Right, that's the idea. Listening test.
One wave from one album is hardly a fair test battery
Yeah, about that. It's been brought up elsewhere. I'm adding a few more discs/genres to the test soon. I probably should've considered doing that before I posted the results.
Without noting the encoding time
This comes up a lot too. The thing is, speed isn't important to me here. I'm not testing for that. Plus, I used the slowest settings for each encoder, so comparing speed would be slightly unfair.
What criteria did you use to conclude that "Liquid Audio takes the cake" since it saved less than a megabyte over Monkey?
Uh... maybe the fact that it saves space over Monkey? How is the difference relevant to choosing a winner? Besides, with enough albums, less than a megabyte can become a few hundred.
Why does your graph state that LA achieved 65.61% compression but your text says 65.33%?
Man, what in the blazes are you talking about? The graph data is exactly the same as the text.
As tests go, this one is lacking
Thanks. I love you too.
MP4ALS is not based on LPAC
Audiocoding.com begs to differ.
* Reference model for new MPEG-4 Lossless Audio Coding / ALS
With what options did you test?
Like I said, best compression in all cases except ALS, with which I used the default reference encoder values. I did the same encode later with the heaviest compression settings, but the filesize difference was minimal. I'll consider doing this with the rest of the samples given my comment earlier about "hundreds of megabytes", as right now it seems a bit hypocritical.
I like how your link for TAK links to the wikipedia article when clearly in the external links section there is a link to the official site www.thbeck.de/Tak/Tak.html
You mean the official site that's in German? Yeah. That's useful to English speakers trying to find out what the hell TAK is.
Just want to point out that bitrate is almost always decimal (not binary) so CD bitrate is 1411.2 kbps...
Why does your graph state that LA achieved 65.61% compression but your text says 65.33%?
Man, what in the blazes are you talking about? The graph data is exactly the same as the text.
La takes the cake, achieving 65.33% compression -- the closest to half off you're going to get.
(http://img76.imageshack.us/img76/2552/acc123sk3.png)
[quote name='PHOYO' date='Jul 1 2007, 10:59' post='501889'][quote name='Tab' post='501875' date='Jul 1 2007, 16:45']
La takes the cake, achieving 65.33% compression -- the closest to half off you're going to get.
Ah, good eye. I'll have to fix that. I still don't see what it has to do with the graph, though...
Tough crowd :|
Or a crowd that just needs tests like these to be fair and useful.
This comes up a lot too. The thing is, speed isn't important to me here. I'm not testing for that. Plus, I used the slowest settings for each encoder, so comparing speed would be slightly unfair... Uh... maybe the fact that it saves space over Monkey? How is the difference relevant to choosing a winner?
You have to recognize the reality that if encoding an album takes 4 minutes with Monkey's Audio and 5 hours with Liquid Audio, then declaring that Liquid Audio has "taken the cake" is really quite foolish since very few people would be happy with that. Hence the near total absence of LA as a current format. You keep talking about what's important to
you, but if you're going to do a test for everyone else, then you need to keep in mind that they require a frame of reference for the compression ratios, otherwise it's just useless. If I could design an encoder that obtained a 65.2% compression ratio but took over 3 years to do the compression, I would expect that people would have an issue with me posting the ratios and declaring that my encoder has "taken the cake." It's hardly unfair to post how long the encodings take... it's an integral part of determining what encoder a person wants to use (which should be the focus of such testing).
Man, what in the blazes are you talking about? The graph data is exactly the same as the text.
As PHOYO has already pointed out, you're the one not observing your own work here. However, your response certainly is indicative of your helpfulness to the larger group. The table is part of your graph... the only part that has any meaning to anyone since the bar-chart uses repetitive colours and is unlabeled. One can therefore only conclude that the graph does not in fact equal the text.
As QHOBBES 2.0 has also pointed out, it appears that you have used WinZIP and attributed its compression success as indicative of the ZIP format in general which is misleading. Since WinZIP uses WavPack to compress *.wav files, there probably isn't much purpose to its inclusion.
I applaud testing that can help people determine what software will best meet their needs. The criticisms in this thread are largely because your test - as is - lacks the information that would help people make an informed decision. To be valuable information, it must be useful information. Rather than react defensively and somewhat juvenile, maybe you want to approach things in a spirit of learning and growth thereby enhancing the community around here. I think that's all that anyone is getting at.
MP4ALS [...] should be better than Monkey's Audio.
Got any test data to back this up?
Like I said, best compression in all cases except ALS, with which I used the default reference encoder values. I did the same encode later with the heaviest compression settings, but the filesize difference was minimal.
What settings did you try as "heaviest compression settings"?
I've just completed and compiled a formal comparison of the major players in the lossless audio arena, including, but not limited to, FLAC, MPEG-4 ALS, and iTunes ALAC. Results, along with pretty chart, here
Thanks, but 2 problems:
- LA 228 65.61 | APE 229 65.61 ??? Also, you should calculate the "%" from exact sizes, not round to MB first
- ZIP is a good as WAVPACK ??? Typo or fraud ?
IIRC Winzip uses wavpack, otherwise I highly doubt a standard zip and wavpack would have the same ratio.
Then it's not a standard ZIP ... it's very unfair to WAVPACK developers ... and I doubt that you can extract such a "ZIP" with anything except "official" WINZIP 12 WAVPACK developers did the work, and WINZIP company collects the money
IIRC Winzip uses wavpack, otherwise I highly doubt a standard zip and wavpack would have the same ratio.
Then it's not a standard ZIP ... it's very unfair to WAVPACK developers ... and I doubt that you can extract such a "ZIP" with anything except "official" WINZIP 12 WAVPACK developers did the work, and WINZIP company collects the money
Agree with you on the first part (the ZIP used should be something openable by ALL zip files not winzip's implementation), but not the second (wavpack developers did the work...). Winzip using wavpack has been discussed before in the forum, and the developer of wavpack has no problems with it. I guess we shouldn't either.
Anyway about the test, I think it's useful, but not displaying the encode and decode times would be very misleading. I know it's your test and you're free to do as you want, but people looking at the test will be misled if they use the test in its current form as a guide. I mean sure LA is the best compressing, but both encoding and decoding rates are downright pathetic. I don't think anyone here *listens* to the LA compressed songs--- just archival. I think at the very least, the encoding time should be published.
Then it's not a standard ZIP ... it's very unfair to WAVPACK developers ... and I doubt that you can extract such a "ZIP" with anything except "official" WINZIP 12 WAVPACK developers did the work, and WINZIP company collects the money
You're absolutely right that the ZIP files using WavPack are only unzippable with the official WinZip 11, but hopefully other utilities will handle these also someday. After all, the specs are open and the software is free.
However, I'm not sure how this is so unfair to me. The license of WavPack certainly allows commercial products assuming proper credit is given (which is done here (http://www.winzip.com/comp_info.htm)) and I think that the endorsement of WinZip can only help WavPack's status as a robust product. It certainly can't hurt.
Yeah, it would be nice to get some of the money that they presumably make, but that's not the idea of writing free software. The world wouldn't have been better off if they got one of their programmers to design a new algorithm (unless it was better than WavPack, which is not too likely considering how much work making a good compressor is).
You're absolutely right that the ZIP files using WavPack are only unzippable with the official WinZip 11, but hopefully other utilities will handle these also someday. After all, the specs are open and the software is free.
Well, is there any known way to force it to use standard zip? If so, I'll update the comparison.
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul. Check it out: SS's lossless comparison (http://www.synthetic-soul.co.uk/comparison/lossless/)
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul. Check it out: SS's lossless comparison (http://www.synthetic-soul.co.uk/comparison/lossless/)
SSoul's comparison is nice. But if Tab can provide his computer specs as well as the encoding/decoding speeds, it'd be a really helpful addition.
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul. Check it out: SS's lossless comparison (http://www.synthetic-soul.co.uk/comparison/lossless/)
http://wiki.hydrogenaudio.org/index.php?ti...ons_comparisons (http://wiki.hydrogenaudio.org/index.php?title=Lossless_comparison#Other_lossless_compressions_comparisons)
http://wiki.hydrogenaudio.org/index.php?ti...ons_comparisons (http://wiki.hydrogenaudio.org/index.php?title=Lossless_comparison#Other_lossless_compressions_comparisons)
heh...well, there are some others I overlooked.
You're absolutely right that the ZIP files using WavPack are only unzippable with the official WinZip 11, but hopefully other utilities will handle these also someday. After all, the specs are open and the software is free.
Well, is there any known way to force it to use standard zip? If so, I'll update the comparison.
use another program. I believe that KZIP offers the best compression for zip files. It beats 7-Zip by a few bytes.
http://advsys.net/ken/util/kzip.exe (http://advsys.net/ken/util/kzip.exe)
I don't mean to be the poop in the punchbowl here, but on top of the valid criticism of this test, there's also the fact that a pretty damn good lossless comparison has already been done by our very own Synthetic Soul. Check it out: SS's lossless comparison (http://www.synthetic-soul.co.uk/comparison/lossless/)
Much appreciated.
SSoul's comparison is nice.
Nice?!! You philistine, it's f***ing marvelous!
SSoul's comparison is nice. But if Tab can provide his computer specs as well as the encoding/decoding speeds, it'd be a really helpful addition.
In all seriousness, I totally agree.
Firstly, I do not test strong compression settings, or LA at all. My test was originally to compare TAK to other, relative, formats. Also very few people seem interested in compressing every last MiB, and are more concerned with the compression/speed trade-off.
Secondly, the more (good) comparisons the better, IMHO. Recent testing with the updated WavPack - where my machine showed little improvement although many newer systems did - hit home to me that my test is not reepresentative of everyone's PC, only mine. The more data we have, the more accurate results we can get.
I would like to know the exact settings used for this test, and see more sources being tested.
for what kind of audio are u testing?speech?dance?classical??
What criteria did you use to conclude that "Liquid Audio takes the cake" since it saved less than a megabyte over Monkey? Why does your graph state that LA achieved 65.61% compression but your text says 65.33%?
What criteria did you use to conclude that "Liquid Audio takes the cake" since it saved less than a megabyte over Monkey?
Uh... maybe the fact that it saves space over Monkey?
You have to recognize the reality that if encoding an album takes 4 minutes with Monkey's Audio and 5 hours with Liquid Audio, then declaring that Liquid Audio has "taken the cake" is really quite foolish since very few people would be happy with that. Hence the near total absence of LA as a current format.
Tab doesn't seem to mind, but
La (mind the lower case
a) stands for
Lossless audio, a proprietary codec by German Michael Bevin, which unfortunately we haven't heard much about for 3 years now. La was and still is on par with OptimFROG's strongest compression settings, but a tad faster. That's why La is still my long-term archival codec of choice.
Liquid Audio, a (former?) software audio player, has nothing at all to do with
La.
I like how your link for TAK links to the wikipedia article when clearly in the external links section there is a link to the official site www.thbeck.de/Tak/Tak.html
You mean the official site that's in German? Yeah. That's useful to English speakers trying to find out what the hell TAK is.
What a nice, cosmopolitan and open-minded point of view. The fact that probably at least half the readers of Hydrogenaudio, let alone the potential users of TAK, are not native speakers of either English or German, wouldn't maybe happen to have crossed your mind for a moment?
edit: phrasing
Firstly, I do not test strong compression settings, or LA at all.
You did test OptimFROG's Extranew mode, didn't you? I wouldn't call that one the weakest of the lot. Or the fastest, for that matter. Your
f***ing marvelous (sic) tables wouldn't do so either
According to HansHeijden's stats (http://web.inter.nl.net/users/hvdh/lossless/All.htm), La is both at least as good a compressor, and almost twice as fast as OptimFROG Extranew. If you ever have some spare time, it'd be nice to check out how TAK compares to, say,
La -high -noseek.
Edit 1: spelling.
Edit 2: actually, I forgot about Joseph Pohm's tests (http://synthetic-soul.co.uk/comparison/josef/) that you host on your site in this respect.
Firstly, I do not test strong compression settings, or LA at all.
You did test OptimFROG's Extranew mode, didn't you? I wouldn't call that one the weakest of the lot. Or the fastest, for that matter. Your f***ing marvelous (sic) tables wouldn't do so either
True, I did test some high compression settings, but I meant to say that I specifically did not look for the
optimum compression settings for each codec, or even test some (like La). I picked the OptimFROG settings not knowing anything about them, so extranew snuck in really. If I knew more about the OptimFROG settings I could possibly use settings better suited to the table. Extranew is nothing compared to some of the OptimFROG settings though, as demonstrated by Josef and Johan De Bock's results.
According to HansHeijden's stats (http://web.inter.nl.net/users/hvdh/lossless/All.htm), La is both at least as good a compressor, and almost twice as fast as OptimFROG Extranew. If you ever have some spare time, it'd be nice to check out how TAK compares to, say, La -high -noseek.
I did have La default in the table with the original format, but when I moved to CPU-only times I dropped it. La -high -noseek is really getting away from the settings I was interested in.
That said, I actually have a little free time at the moment, so I may be able to add in some slower codecs/settings to fill the table out a bit. Looking at the table OptimFROG does provide some nice comparison for some of the slower TAK, FLAC and WavPAck modes, and it seems unfair not to show La in comparison to OptimFROG.
What a nice, cosmopolitan and open-minded point of view. The fact that probably at least half the readers of Hydrogenaudio, let alone the potential users of TAK, are not native speakers of either English or German, wouldn't maybe happen to have crossed your mind for a moment?
It's not a good idea to criticize when you don't know the facts. This comparison was not done exclusively for this forum. In fact, this is one of the last places I posted it. Most of the other places it was posted, and in particular, the most important places, all contain solely English-speaking audiences. Don't be a douche if you don't know what you're talking about.
bryant wrote:
> only unzippable with the official WinZip 11, but hopefully other utilities will handle these also
Other archivers "must" bloat also
> license of WavPack certainly allows commercial products assuming proper credit is given
Thank you. I indeed prefer BSD from GPL also ...
> WinZip can only help WavPack's status as a robust product. It certainly can't hurt.
Some (most) people might miss WAVPACK and think the better audio compression is an achievement exclusive to WinZIP 12 (or 11 ?) ...
Tab wrote:
> Well, is there any known way to force it to use standard zip? If so, I'll update the comparison.
There used to be something like " PKZIP 2.xx compatible " checkbox in WinZIP cca 9 ... no idea whether it persists in WinZIP 11 / 12 since I don't use it Please update the comparison with standard ZIP :-)
Manqix wrote:
> I believe that KZIP offers the best compression for zip files. It beats 7-Zip by a few bytes.
Right. KZIP offers the best PKZIP 2.xx comaptible ZIP's - it's just a bit slow.
It's not a good idea to criticize when you don't know the facts.
Do I detect just a tiny bit of psychological projection there? Oh, and have a look at http://www.hydrogenaudio.org/forums/index....showtopic=43696 (http://www.hydrogenaudio.org/forums/index.php?showtopic=43696) and draw whichever conclusion you like.
This comparison was not done exclusively for this forum. In fact, this is one of the last places I posted it. Most of the other places it was posted, and in particular, the most important places, all contain solely English-speaking audiences.
I'm sure you gave your findings much thought before sharing them with half the audio world, as the replies in this thread seem to suggest. Mind sharing with us which forums you multi-posted to, so we can learn a bit from that? Where did you get your sociolinguistics data from, by the way?
Don't be a douche if you don't know what you're talking about.
How well do you know me to know that I don't know what I'm talking about? Did you know what you were talking about when carefully composing that final sentence?
Just a piece of advice: settle down, man. Why do expect any goodwill from your sour ventilations? If I were you, I would have stuck to that selfproclaimed specialty domain of yours, i.e. maximum setting lossless audio compression, and refrain from that silly "English über alles" attitude. Otherwise you
are really all too transparently projecting your own
Don't be a douche if you don't know what you're talking about onto someone else again.
Back on topic. How about enlightening us somewhat on which compression settings you used?
Tab doesn't seem to mind, but La (mind the lower case a) stands for Lossless audio, a proprietary codec by German Michael Bevin, which unfortunately we haven't heard much about for 3 years now... Liquid Audio, a (former?) software audio player, has nothing at all to do with La.
Fair enough, though I think my point still stands. It's all a little meaningless though it would seem, since the OP seems incapable of learning and growing (up).
since the OP seems incapable of learning and growing (up).
Mind explaining exactly where this petty and childish comment is coming from?
Don't be a douche if you don't know what you're talking about.
Perhaps because of responses like this? Many of your responses to criticism have been rude and defensive.
Can we
please stop the name calling and finger pointing people, and keep this constructive?
Edit: Tab, I really don't think your figures mean anything until you specify exactly what settings you used. You may believe that you have used the best settings, but it's possible you didn't, and until you can specify exact switches then the data doesn't really have context.
The fact that only one album was tested (IIRC) is not so bad - although it would be nice to see a wider range of source material - as, IMHO, any
valid test is useful, and the tester should be encouraged.
That said, I actually have a little free time at the moment, so I may be able to add in some slower codecs/settings to fill the table out a bit. Looking at the table OptimFROG does provide some nice comparison for some of the slower TAK, FLAC and WavPAck modes, and it seems unfair not to show La in comparison to OptimFROG.
That's fantastic, Synthetic Soul. Looking forward to that.
for what kind of audio are u testing?speech?dance?classical??
Interesting question. After all, 349 MBytes of uncompressed source material, worth just over half an hour of CD audio, is not a very broad base to be anything of an authoritative graduator. If indeed we're talking about 16 bit stereo CDDA, is still an open question, but very relevant to be able to judge the various codecs' performances. FLAC, e.g., is known not to be at its best with 24 bit samples.
Edit:
It's only now that it's occurred to me that the source is this CDDA album:
http://en.wikipedia.org/wiki/Nothing_and_Nowhere (http://en.wikipedia.org/wiki/Nothing_and_Nowhere)
Sorry about that.
Well, is there any known way to force it to use standard zip? If so, I'll update the comparison.
Use WinZip versions older than 11 ...
Or set the compression level to "Maximum (portable)"
That said, I actually have a little free time at the moment, so I may be able to add in some slower codecs/settings to fill the table out a bit. Looking at the table OptimFROG does provide some nice comparison for some of the slower TAK, FLAC and WavPAck modes, and it seems unfair not to show La in comparison to OptimFROG.
That's fantastic, Synthetic Soul. Looking forward to that.
OK, La default, -high, and -high -noseek have been added to the list (http://www.synthetic-soul.co.uk/comparison/lossless/index.asp).
I may mix and match the OptimFROG settings a bit in the future, when I get some more time.
OK, La default, -high, and -high -noseek have been added to the list (http://www.synthetic-soul.co.uk/comparison/lossless/index.asp).
Thanks for the effort, Synthetic Soul. Most interesting.
And if I may say so, all the more credit to La. Even without much development for almost 3.5 years, your comparison along with the ones listed on the Wiki (http://wiki.hydrogenaudio.org/index.php?title=Lossless_comparison#Other_lossless_compressions_comparisons) claim that, on 16 bit 44.1 kHz stereo CD audio, La is only univocally surpassed in compression ratio by OptimFROG's most recent
bestnew,
ultranew and
maximumcompression presets. But with quite a heavy speed penalty compared to already slow La.
I may mix and match the OptimFROG settings a bit in the future, when I get some more time.
Side remark: high, extra, best, highnew, extranew, bestnew, ultranew, maximumcompression ... Not the most straightforward naming scheme, frankly speaking.
Don't get me wrong, OptimFROG undeniably has its merits, especially in its faster
normal,
high and
extra modes, and because of its highly versatile format. It's just that, since both La and OptimFROG are aiming at the very top of the compression ratio spectrum, it is (imho) debatable whether OF's
bestnew, and especially
ultranew and
maximumcompression modes are worth the time and CPU resources to still actually be of any practical use. Again, imho, they're not, which is why I prefer La for long-term archival, and FLAC and TAK for everyday use, i.e. playback and transcoding.
Now for the origin of this thread. I'm still wondering if Tab is willing to share the specifics of his encodings with us, and maybe those other forums he mentioned. Can't blame us for not asking.
@SSoul: how come the original file for you compare is 2030.59 MiB?
@SSoul: how come the original file for you compare is 2030.59 MiB?
Probably because that falls just under the 2 gB limit of a wav file.
@SSoul: how come the original file for you compare is 2030.59 MiB?
My test corpus is made up of 50 files which total 2030.59 MiB (http://www.synthetic-soul.co.uk/comparison/lossless/corpus.asp). If you click on a setting in the main list you will be taken to a page which details the results for each file in the corpus for that setting, e.g.: TAK Normal (http://www.synthetic-soul.co.uk/comparison/lossless/setting.asp?SettingPKID=90).
La default, -high, and -high -noseek have been added to the list (http://www.synthetic-soul.co.uk/comparison/lossless/index.asp).
I may mix and match the OptimFROG settings a bit in the future, when I get some more time.
In the meantime, fancy benchmarking OptimFROG
best,
highnew,
bestnew,
ultranew and
maximumcompression, and Monkey's Audio
insane? All of which should at least provide an interesting comparison at the higher end of the compression scale.
Also, OptimFROG
fast is one of the more usable settings to be measured up against TAK.