At the moment, I'm comparing FLAC, WavPack, APE, WMA Lossless, and Apple Lossless codecs as apart of the research project. I will be assessing them based on these factors:
- Compression Speed (Time in Minutes:Seconds)
- Compression Rate (Percentage & MB output)
- Decompression Speed (Time in Minutes:Seconds)
- Compression Settings (How many are avaliable)
- Metadata Support/Tagging
- Open Source/Propriety?
- Replay Gain
- OS Support
- Hardware Support
- Software Support
- Hybrid/Lossy?
So far I have chosen 15 albums to test with. Is this a good amount for statistical significance?
Btw, I'm comparing/assessing the aforementioned codecs because I need to evaluate codecs that are popular in terms of hardware/software support and have been updated within last 2 years. I thought of evaluating TAK at first, but it has practically no hardware support.
Are the 15 albums all clustered in the same genre, or is there a wide variety of musical styles being represented?
Are the 15 albums all clustered in the same genre, or is there a wide variety of musical styles being represented?
It's a wide variety, trust me. I don't think that the genre of the music will matter though.
The genre can make quite a difference. A quiet classical piece usually takes about half the bitrate of contemporary metal at the edge of noise.
Here are the CDs that I plan to test:
Gorillaz by the Gorillaz (Hip-Hop/Alternative)
Surrealistic Airplanes by Jefferson Airplane (Psychadelic Rock)
NOW That's What I Call Country by Various Artists
OK Computer (Alternative/Electronica)
20 Best of Jazz by Various Artists (Jazz)
Rock Steady by No Doubt (Pop/Ska)
The Good The Bad The Queen by The Good The Bad The Queen (Pop/Alternative)
Lost Odyssey 2 Disc Soundtrack by Nobuo Uematsu (Classical/Orchestral/Electronic)
Silver Side Up by Nickelback (Heavy Rock)
Sidewalks & Happy Endings by Playing in the Streets (Rock/Pop/Hip-Hop)
All World by LL Cool J (Rap/Hip-Hop)
Raditude by Weezer (Pop/Rock/Alternative)
Marvin Gaye Collection Volume 2 by Marvin Gaye (Blues/R&B/Jazz)
Stadium Arcadium 2 Dic by the Red Hot Chili Peppers (Pop/Rock)
New Moon Soundtrack by Various Artists (Alternative/Various)
I realize that I don't have much Classical Musical Samples. Neither I or many of friends that I could borrow CDs from have any. I could always download some classical musical compressed as flacs/lossless songs, and then decode them. What do you all think
Why do whole albums?
The type of music makes a huge difference. IMO you're better off doing a wider variety of tracks.
Furthermore I'd look at the replay gain values of each track.
Have a look at this for example:
(http://www.giantpygmy.net/sections/play/img/lossy_lossless_bitrates.png)
Notice that the same Doors track, original versus remaster, gives very different results. So just because you've got 60's rock in there it means nothing, since a modern remaster will make it as loud (and crap sounding) as modern pop.
There's a graph of these results here (http://www.giantpygmy.net/sections/play/img/lossy_lossless_by_music.png).
Also, why no TAK?
C.
Why do whole albums?
The type of music makes a huge difference. IMO you're better off doing a wider variety of tracks.
Furthermore I'd look at the replay gain values of each track.
Have a look at this for example:
(http://www.giantpygmy.net/sections/play/img/lossy_lossless_bitrates.png)
Notice that the same Doors track, original versus remaster, gives very different results. So just because you've got 60's rock in there it means nothing, since a modern remaster will make it as loud (and crap sounding) as modern pop.
There's a graph of these results here (http://www.giantpygmy.net/sections/play/img/lossy_lossless_by_music.png).
Also, why no TAK?
C.
I would test a wide variety of tracks, but I don't really have the necessary samples ready at hand. ATM i'm trying to get a hold of a lossless classical music torrent so I can increase the variety of samples I have. Since a couple of the codecs I'm trying to test don't feature any replay gain option, I'm going to compare each codec within their normal/default compression setting while evaluating the other settings at the same time. As I mentioned in my starting post "I'm comparing/assessing the aforementioned codecs because I need to evaluate codecs that are popular in terms of hardware/software support and have been updated within last 2 years. I thought of evaluating TAK at first, but it has practically no hardware support."
It's a wide variety, trust me.
I would test a wide variety of tracks, but I don't really have the necessary samples ready at hand.
That isn't a wide variety IMO. For a start you've completely missed out metal as a genre, this being a prime example of something that requires a higher than average bitrate to encode.
You're best choosing tracks as well.
... Since a couple of the codecs I'm trying to test don't feature any replay gain option, I'm going to compare each codec within their normal/default compression setting while evaluating the other settings at the same ...
Replay Gain and the compression level settings are not anyhow related to each other.
In general, an audio file format does not need to have any kind of built-in Replay Gain support. FLAC and WavPack have some features that support Replay Gain, but those features are not necessary for using Replay Gain. You can use Replay Gain with WMA Lossless, Apple Lossless or Monkey's Audio files just fine. I.e you can analyze the files, store the measured gain correction and peak level values, and use a playback correction system. Naturally the used player must support Replay Gain, but this is true also for FLAC or WavPack.
I hate to be a buzzkill, but with at least a half-dozen thorough comparisons of lossless codecs available on the internet, I don't see how this is going to cover any new territory.
At the moment, I'm comparing FLAC, WavPack, APE, WMA Lossless, and Apple Lossless codecs as apart of the research project. I will be assessing them based on these factors:
- Compression Speed (Time in Minutes:Seconds)
- Compression Rate (Percentage & MB output)
- Decompression Speed (Time in Minutes:Seconds)
- Compression Settings (How many are avaliable)
- Metadata Support/Tagging
- Open Source/Propriety?
- Replay Gain
- OS Support
- Hardware Support
- Software Support
- Hybrid/Lossy?
So far I have chosen 15 albums to test with. Is this a good amount for statistical significance?
Btw, I'm comparing/assessing the aforementioned codecs because I need to evaluate codecs that are popular in terms of hardware/software support and have been updated within last 2 years. I thought of evaluating TAK at first, but it has practically no hardware support.
Are you going to compare them on anything but x86? I ask because I wonder why did you left TAK out.
Actually I came here only to see a well done comparison that involves it...I've seen many comparisons of the rest already.
Just a hint: you can get classical music CDs and lots of other music from your public library. But it might be illegal to copy it to your PC, at least where I live it's perfectly legal unless the CDs have a working copy-protection (no, scratches, although very common with library CDs, do not count as a copy protection, although practically they are one, haha).
I got the idea of evaluating/comparing different lossless codecs by using a variety of CDs by overviewing Bobulous's "Audio Format's Comparison" (http://www.bobulous.org.uk/misc/audio_formats_comparison_2006.html). So at the moment I'll focus on comparing each codec by using an equal amount of CDs for each genre
a variety of CDs by overviewing Bobulous's "Audio Format's Comparison" (http://www.bobulous.org.uk/misc/audio_formats_comparison_2006.html).
I'm fascinated by what people think is a wide variety of music.
Surely you'd find out more by not thinking in terms of bands one likes or considers popular, but by the make-up of the different instruments in each ensemble (whether it's 2 electric guitars + bass + drums versus solo cello etc ...).
For example, how many of the bands (/instrument groupings) in the list have no percussion? Crashing symbols produce much more complex harmonics than a piano note. So some things will be hard to compress and other things easy, and IMO it's much more interesting to find out how each codec performs on simple as well has complex sounds.
C.
a variety of CDs by overviewing Bobulous's "Audio Format's Comparison" (http://www.bobulous.org.uk/misc/audio_formats_comparison_2006.html).
I'm fascinated by what people think is a wide variety of music.
Surely you'd find out more by not thinking in terms of bands one likes or considers popular, but by the make-up of the different instruments in each ensemble (whether it's 2 electric guitars + bass + drums versus solo cello etc ...).
For example, how many of the bands (/instrument groupings) in the list have no percussion? Crashing symbols produce much more complex harmonics than a piano note. So some things will be hard to compress and other things easy, and IMO it's much more interesting to find out how each codec performs on simple as well has complex sounds.
C.
This is basically what I wrote on the original topic, but I think it might be better if we pm instead
After contemplating this over and over again, I'm finally going to agree with you. I recently got a hold of more CDs and files, and now I can focus more on evaluating each lossless codec with different tracks. The only thing I need to figure out now is how many samples/tracks I should evaluate in order to produce meaningful results. Since testing individual tracks will probably be less time consuming than testing albums, I can now take into consideration on what other parameters each codec uses that I can evaluate. What do you think?
The only roadblock I can think of right now would evaluating with WMA Lossless. With the Windows Media Player, you can only compress music by ripping them through a disc unless you have some third party frontend/converter. Most of my raw music are only on my computer, and I don't really know how well frontends/converters like dBpoweramp work.
scratches, although very common with library CDs, do not count as a copy protection, although practically they are one, haha
Considering how these copy protection schemes work, it is a fairly apt comparison.
(And the combination is even more effective!)
I don't really know how well frontends/converters like dBpoweramp work.
Why would they not work well; why would they work differently at all? In the case of WMA, they probably just pass audio to Microsoft's encoder.
http://www.hydrogenaudio.org/forums/index....showtopic=71612 (http://www.hydrogenaudio.org/forums/index.php?showtopic=71612)
http://www.icer.nl/losslesstest/ (http://www.icer.nl/losslesstest/)
Chris
Doesn't flake outperform the official FLAC encoder?
It depends on the music. FLAC is better with some classical tracks (piano sonatas), while Flake performs better on noisy tracks (especially japanese noise like Merzbow, Dissecting Table, Aube, Masonna, ...) but also with most "normal" music you get slightly better results when you use -9 -10 or -11 with Flake compaed to FLAC -8.
Here's are the current codecs I plan to evaluate:
Free Lossless Audio Codec 1.2.1b through FLAC Frontend OR foobar2000 v1.0
Monkey's Audio 4.06 through Monkey's Audio Frontend
Apple Lossless through iTunes 9.0.3.15
WavPack through WavPack Frontend 2.0 OR foobar2000 v1.0
WMA Lossless 9.2 through dVpoweramp Music Converter
The issue I am facing now is to determining the encoding/decoding rate of the file samples. For instance, Monkey's Audio Frontend accurately displays the time it takes to encode/decode each sample, while all the others (except dBpoweramp which just rounds up the en/dec rate to the nearest min:sec) don't. What would be a good way of approaching this? The main alternative I can think of would be to use a stop watch, but that sound's pretty lame...
I thought there might exist command line programs that can call and time others, so I searched Google for "command line timer". This was the first result: Gammadyne's Free DOS Utilities: TIMER.EXE (http://www.gammadyne.com/cmdline.htm#timer). There are probably other/better ways: it seems that you must run TIMER.EXE then the encoder, rather than simply passing the latter's command line to the former; if workable, this would probably involve batch files. Still, it's a start.
Edit: timeit.exe from the Windows Server 2003 Resource Kit (http://stackoverflow.com/questions/673523/how-to-measure-execution-time-of-command-in-windows-command-line) is probably better.
accurate timing itself is a hard enough problem. process time is more accurate but doesn't capture i/o. i/o time is very dependent on your hardware and is confounded by things like caching.
if you want overall time, it's best to run a large sample, preferably multiple times to avoid cache effects, and take an average. then make sure your results are in reasonable given the other published results, e.g.
http://flac.sourceforge.net/comparison.html (http://flac.sourceforge.net/comparison.html)
http://www.synthetic-soul.co.uk/comparison/lossless/ (http://www.synthetic-soul.co.uk/comparison/lossless/)
http://web.inter.nl.net/users/hvdh/lossless/lossless.htm (http://web.inter.nl.net/users/hvdh/lossless/lossless.htm)
The best alternative I can see with these issues is to encode/decode WMA Lossless, Apple Lossless, and FLAC with dBpoweramp. dBpoweramp supports all of the codecs I mentioned and feature the amount of time it took to encode/decode in min:sec. I can then use Synthetic's Soul's method to determine the the enc/dec rate for WavPack and Monkey's Audio own front end for .ape which can give me accurate timing.
I thought there might exist command line programs that can call and time others, so I searched Google for "command line timer". This was the first result: Gammadyne's Free DOS Utilities: TIMER.EXE (http://www.gammadyne.com/cmdline.htm#timer). There are probably other/better ways: it seems that you must run TIMER.EXE then the encoder, rather than simply passing the latter's command line to the former; if workable, this would probably involve batch files. Still, it's a start.
Edit: timeit.exe from the Windows Server 2003 Resource Kit (http://stackoverflow.com/questions/673523/how-to-measure-execution-time-of-command-in-windows-command-line) is probably better.
I used Timer with batch files: http://www.hydrogenaudio.org/forums/index....showtopic=50954 (http://www.hydrogenaudio.org/forums/index.php?showtopic=50954)
I was using an MS app (may have been TimeIt), but it reported CPU+IO time, so I switched to Timer to report only CPU time. (I believe this is documented in the depths of these forums also.)
Edit: Sorry, on checking, I used a
different TIMER.EXE! http://www.7-zip.org/utils.html (http://www.7-zip.org/utils.html)
Thanks, Synthetic Soul.
I know that this may be a redundant question, but is converting any lossless format to a .wav file the same as decoding?
It depends on the music. FLAC is better with some classical tracks (piano sonatas), while Flake performs better on noisy tracks (especially japanese noise like Merzbow, Dissecting Table, Aube, Masonna, ...) but also with most "normal" music you get slightly better results when you use -9 -10 or -11 with Flake compaed to FLAC -8.
I was thinking of speed, too.
I know that this may be a redundant question, but is converting any lossless format to a .wav file the same as decoding?
Yes, you are decoding to WAVE.
It depends on the music. FLAC is better with some classical tracks (piano sonatas), while Flake performs better on noisy tracks (especially japanese noise like Merzbow, Dissecting Table, Aube, Masonna, ...) but also with most "normal" music you get slightly better results when you use -9 -10 or -11 with Flake compaed to FLAC -8.
I was thinking of speed, too.
When talking about Flac and its maximums Flacuda should be mentioned. It compresses clearly stronger and much faster as Flac 1.21 AND the latest flake SVN i could get at -8.
Its developer Mr. Chudov thinks this is how far the Flac standard itself can go. No further improvisations compression wise should be expected cause it uses even much brute-force already.
Unfortunately it is net and nvidia only atm.