Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Lossless Comparison Test (Read 12162 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Lossless Comparison Test

At the moment, I'm comparing FLAC, WavPack, APE, WMA Lossless, and Apple Lossless codecs as apart of the research project. I will be assessing them based on these factors:
  • Compression Speed (Time in Minutes:Seconds)
  • Compression Rate (Percentage & MB output)
  • Decompression Speed (Time in Minutes:Seconds)
  • Compression Settings (How many are avaliable)
  • Metadata Support/Tagging
  • Open Source/Propriety?
  • Replay Gain
  • OS Support
  • Hardware Support
  • Software Support
  • Hybrid/Lossy?


So far I have chosen 15 albums to test with. Is this a good amount for statistical significance?

Btw, I'm comparing/assessing the aforementioned codecs because I need to evaluate codecs that are popular in terms of hardware/software support and have been updated within last 2 years. I thought of evaluating TAK at first, but it has practically no hardware support.
 

Lossless Comparison Test

Reply #1 –
Are the 15 albums all clustered in the same genre, or is there a wide variety of musical styles being represented?

Lossless Comparison Test

Reply #2 –
Quote from: Tahnru on
Are the 15 albums all clustered in the same genre, or is there a wide variety of musical styles being represented?


It's a wide variety, trust me. I don't think that the genre of the music will matter though.

Lossless Comparison Test

Reply #3 –
The genre can make quite a difference. A quiet classical piece usually takes about half the bitrate of contemporary metal at the edge of noise.

Lossless Comparison Test

Reply #4 –
Here are the CDs that I plan to test:

Gorillaz by the Gorillaz (Hip-Hop/Alternative)
Surrealistic Airplanes by Jefferson Airplane (Psychadelic Rock)
NOW That's What I Call Country by Various Artists
OK Computer (Alternative/Electronica)
20 Best of Jazz by Various Artists (Jazz)
Rock Steady by No Doubt (Pop/Ska)
The Good The Bad The Queen by The Good The Bad The Queen (Pop/Alternative)
Lost Odyssey 2 Disc Soundtrack by Nobuo Uematsu (Classical/Orchestral/Electronic)
Silver Side Up by Nickelback (Heavy Rock)
Sidewalks & Happy Endings by Playing in the Streets (Rock/Pop/Hip-Hop)
All World by LL Cool J (Rap/Hip-Hop)
Raditude by Weezer (Pop/Rock/Alternative)
Marvin Gaye Collection Volume 2 by Marvin Gaye (Blues/R&B/Jazz)
Stadium Arcadium 2 Dic by the Red Hot Chili Peppers (Pop/Rock)
New Moon Soundtrack by Various Artists (Alternative/Various)

I realize that I don't have much Classical Musical Samples. Neither I or many of friends that I could borrow CDs from have any. I could always download some classical musical compressed as flacs/lossless songs, and then decode them. What do you all think


Lossless Comparison Test

Reply #5 –
Why do whole albums?

The type of music makes a huge difference. IMO you're better off doing a wider variety of tracks.
Furthermore I'd look at the replay gain values of each track.

Have a look at this for example:



Notice that the same Doors track, original versus remaster, gives very different results. So just because you've got 60's rock in there it means nothing, since a modern remaster will make it as loud (and crap sounding) as modern pop.

There's a graph of these results here.

Also, why no TAK?

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

Lossless Comparison Test

Reply #6 –
Quote from: carpman on
Why do whole albums?

The type of music makes a huge difference. IMO you're better off doing a wider variety of tracks.
Furthermore I'd look at the replay gain values of each track.

Have a look at this for example:



Notice that the same Doors track, original versus remaster, gives very different results. So just because you've got 60's rock in there it means nothing, since a modern remaster will make it as loud (and crap sounding) as modern pop.

There's a graph of these results here.

Also, why no TAK?

C.


I would test a wide variety of tracks, but I don't really have the necessary samples ready at hand. ATM i'm trying to get a hold of a lossless classical music torrent so I can increase the variety of samples I have. Since a couple of the codecs I'm trying to test don't feature any replay gain option, I'm going to compare each codec within their normal/default compression setting while evaluating the other settings at the same time. As I mentioned in my starting post "I'm comparing/assessing the aforementioned codecs because I need to evaluate codecs that are popular in terms of hardware/software support and have been updated within last 2 years. I thought of evaluating TAK at first, but it has practically no hardware support."

Lossless Comparison Test

Reply #8 –
That isn't a wide variety IMO. For a start you've completely missed out metal as a genre, this being a prime example of something that requires a higher than average bitrate to encode.

You're best choosing tracks as well.

Lossless Comparison Test

Reply #9 –
Quote from: ezlez on
... Since a couple of the codecs I'm trying to test don't feature any replay gain option, I'm going to compare each codec within their normal/default compression setting while evaluating the other settings at the same ...

Replay Gain and the compression level settings are not anyhow related to each other.

In general, an audio file format does not need to have any kind of built-in Replay Gain support. FLAC and WavPack have some features that support Replay Gain, but those features are not necessary for using Replay Gain. You can use Replay Gain with WMA Lossless, Apple Lossless or Monkey's Audio files just fine. I.e you can analyze the files, store the measured gain correction and peak level values, and use a playback correction system. Naturally the used player must support Replay Gain, but this is true also for FLAC or WavPack.

Lossless Comparison Test

Reply #10 –
I hate to be a buzzkill, but with at least a half-dozen thorough comparisons of lossless codecs available on the internet, I don't see how this is going to cover any new territory.

Lossless Comparison Test

Reply #11 –
Quote from: ezlez on
At the moment, I'm comparing FLAC, WavPack, APE, WMA Lossless, and Apple Lossless codecs as apart of the research project. I will be assessing them based on these factors:
  • Compression Speed (Time in Minutes:Seconds)
  • Compression Rate (Percentage & MB output)
  • Decompression Speed (Time in Minutes:Seconds)
  • Compression Settings (How many are avaliable)
  • Metadata Support/Tagging
  • Open Source/Propriety?
  • Replay Gain
  • OS Support
  • Hardware Support
  • Software Support
  • Hybrid/Lossy?


So far I have chosen 15 albums to test with. Is this a good amount for statistical significance?

Btw, I'm comparing/assessing the aforementioned codecs because I need to evaluate codecs that are popular in terms of hardware/software support and have been updated within last 2 years. I thought of evaluating TAK at first, but it has practically no hardware support.

Are you going to compare them on anything but x86? I ask because I wonder why did you left TAK out.
Actually I came here only to see a well done comparison that involves it...I've seen many comparisons of the rest already.

Lossless Comparison Test

Reply #12 –
Just a hint: you can get classical music CDs and lots of other music from your public library. But it might be illegal to copy it to your PC, at least where I live it's perfectly legal unless the CDs have a working copy-protection (no, scratches, although very common with library CDs, do not count as a copy protection, although practically they are one, haha).

Lossless Comparison Test

Reply #13 –
I got the idea of evaluating/comparing different lossless codecs by using a variety of CDs by overviewing Bobulous's "Audio Format's Comparison" (http://www.bobulous.org.uk/misc/audio_formats_comparison_2006.html). So at the moment I'll focus on comparing each codec by using an equal amount of CDs for each genre

Lossless Comparison Test

Reply #14 –
Quote from: ezlez on
a variety of CDs by overviewing Bobulous's "Audio Format's Comparison" (http://www.bobulous.org.uk/misc/audio_formats_comparison_2006.html).

I'm fascinated by what people think is a wide variety of music.
Surely you'd find out more by not thinking in terms of bands one likes or considers popular, but by the make-up of the different instruments in each ensemble (whether it's 2 electric guitars + bass + drums versus solo cello etc ...).

For example, how many of the bands (/instrument groupings) in the list have no percussion? Crashing symbols produce much more complex harmonics than a piano note. So some things will be hard to compress and other things easy, and IMO it's much more interesting to find out how each codec performs on simple as well has complex sounds.

C.
PC = TAK + LossyWAV  ::  Portable = Opus (130)

Lossless Comparison Test

Reply #15 –
Quote from: carpman on
Quote from: ezlez on
a variety of CDs by overviewing Bobulous's "Audio Format's Comparison" (http://www.bobulous.org.uk/misc/audio_formats_comparison_2006.html).

I'm fascinated by what people think is a wide variety of music.
Surely you'd find out more by not thinking in terms of bands one likes or considers popular, but by the make-up of the different instruments in each ensemble (whether it's 2 electric guitars + bass + drums versus solo cello etc ...).

For example, how many of the bands (/instrument groupings) in the list have no percussion? Crashing symbols produce much more complex harmonics than a piano note. So some things will be hard to compress and other things easy, and IMO it's much more interesting to find out how each codec performs on simple as well has complex sounds.

C.


This is basically what I wrote on the original topic, but I think it might be better if we pm instead

After contemplating this over and over again, I'm finally going to agree with you. I recently got a hold of more CDs and files, and now I can focus more on evaluating each lossless codec with different tracks. The only thing I need to figure out now is how many samples/tracks I should evaluate in order to produce meaningful results. Since testing individual tracks will probably be less time consuming than testing albums, I can now take into consideration on what other parameters each codec uses that I can evaluate. What do you think?

The only roadblock I can think of right now would evaluating with WMA Lossless. With the Windows Media Player, you can only compress music by ripping them through a disc unless you have some third party frontend/converter. Most of my raw music are only on my computer, and I don't really know how well frontends/converters like dBpoweramp work.

Lossless Comparison Test

Reply #16 –
Quote from: Fandango on
scratches, although very common with library CDs, do not count as a copy protection, although practically they are one, haha


Considering how these copy protection schemes work, it is a fairly apt comparison.

(And the combination is even more effective!)

Lossless Comparison Test

Reply #17 –
Quote
I don't really know how well frontends/converters like dBpoweramp work.

Why would they not work well; why would they work differently at all? In the case of WMA, they probably just pass audio to Microsoft's encoder.

Lossless Comparison Test

Reply #19 –
Doesn't flake outperform the official FLAC encoder?

Lossless Comparison Test

Reply #20 –
It depends on the music. FLAC is better with some classical tracks (piano sonatas), while Flake performs better on noisy tracks (especially japanese noise like Merzbow, Dissecting Table, Aube, Masonna, ...) but also with most "normal" music you get slightly better results when you use -9 -10 or -11 with Flake compaed to FLAC -8.
WavPack 4.50.1 -hhx6 | LAME 3.98.2 -V 0

Lossless Comparison Test

Reply #21 –
Here's are the current codecs I plan to evaluate:

Free Lossless Audio Codec 1.2.1b through FLAC Frontend OR foobar2000 v1.0
Monkey's Audio 4.06 through Monkey's Audio Frontend
Apple Lossless through iTunes 9.0.3.15
WavPack through WavPack Frontend 2.0 OR foobar2000 v1.0
WMA Lossless 9.2 through dVpoweramp Music Converter

The issue I am facing now is to determining the encoding/decoding rate of the file samples. For instance, Monkey's Audio Frontend accurately displays the time it takes to encode/decode each sample, while all the others (except dBpoweramp which just rounds up the en/dec rate to the nearest min:sec) don't. What would be a good way of approaching this? The main alternative I can think of would be to use a stop watch, but that sound's pretty lame...

Lossless Comparison Test

Reply #22 –
I thought there might exist command line programs that can call and time others, so I searched Google for "command line timer". This was the first result: Gammadyne's Free DOS Utilities: TIMER.EXE. There are probably other/better ways: it seems that you must run TIMER.EXE then the encoder, rather than simply passing the latter's command line to the former; if workable, this would probably involve batch files. Still, it's a start.

Edit: timeit.exe from the Windows Server 2003 Resource Kit is probably better.

Lossless Comparison Test

Reply #23 –
accurate timing itself is a hard enough problem.  process time is more accurate but doesn't capture i/o.  i/o time is very dependent on your hardware and is confounded by things like caching.

if you want overall time, it's best to run a large sample, preferably multiple times to avoid cache effects, and take an average.  then make sure your results are in reasonable given the other published results, e.g.

http://flac.sourceforge.net/comparison.html
http://www.synthetic-soul.co.uk/comparison/lossless/
http://web.inter.nl.net/users/hvdh/lossless/lossless.htm

Lossless Comparison Test

Reply #24 –
The best alternative I can see with these issues is to encode/decode WMA Lossless, Apple Lossless, and FLAC with dBpoweramp. dBpoweramp supports all of the codecs I mentioned and feature the amount of time it took to encode/decode in min:sec. I can then use Synthetic's Soul's method to determine the the enc/dec rate for WavPack and Monkey's Audio own front end for .ape which can give me accurate timing.