HydrogenAudio

Lossless Audio Compression => Lossless / Other Codecs => Topic started by: ktf on 2013-01-05 20:58:29

Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-01-05 20:58:29
Hi all,

For the last few weeks I've been busy writing a new lossless audio codec comparison, after lots of talk about it here (http://www.hydrogenaudio.org/forums/index.php?showtopic=94685) and here (http://www.hydrogenaudio.org/forums/index.php?showtopic=71612).

The full test report can be found here (PDF) (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%201.pdf), the raw data and graphs can be found here (zip) (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20raw%20data%20revision%201.zip).

While the full report has lots of graphs it also has lots of text, so the 'main' graphs of the report are posted below.
(http://www.icer.nl/losslesstest/all-enc.png)
(http://www.icer.nl/losslesstest/all-dec.png)

The CDs used


For more information on why I chose these CDs, what strange things happened with the pure speech data (Dan Browns audiobook) and tests on 96kHz/24-bit or surround sound audio, check the full report (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%201.pdf).

Its labeled revision 1 because I intend to keep it updated. Any comment (on my grammar too, as I'm not a native speaker) is welcome.
Title: New lossless codec comparison (Jan '13)
Post by: IgorC on 2013-01-06 17:41:42
ktf,

Thank You for sharing interesting comparison.

What decoder and/or application have You used for TAK? Last time  TAK native app decodes as fast as FLAC. TAK foobar plugin was somewhat slower.

I was curious to try Monkey's Audio c3000 ('High') on Galaxy II phone with Rockbox app for Android . Got 17 hours of a battery life. And it uses the lowest frequency step so I suspect APE 'Extra High' would last comparable time.  Given that an energy efficiency of proccesors grows fast it's not crazy to think about Monkey's Audio on mobile devices. Anyway I don't use lossless on portable and use FLAC lossless storage  for a fast transcoding to lossy formats.


Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-01-06 18:16:16
What decoder and/or application have You used for TAK? Last time  TAK native app decodes as fast as FLAC.

I've used the Takc.exe command line encoder which I downloaded straight from TAK's homepage. This was run over Wine, but that shouldn't make much of a difference, as Wine only emulates system calls.

The reason that TAK decodes as fast as FLAC with you is probably because not your CPU but your harddisk is the bottleneck. In this test, CPU-time was measured and the test were done with a ramdisk to bypass the harddisk, so this might be the difference. After all, those codecs decode so fast on desktop CPU's that harddisks can't keep up.

Regarding Monkey's Audio, I've seen some devices advertising supporting it recently, so your observation is correct.
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-01-19 21:23:56
I know lossless audio codec comparisons have lost interest with most of you, but only one reply? I hope that PDF wasn't too overwhelming. 

Anyway, a short version of this comparison (and the PDF) is on the FLAC website now: http://flac.sourceforge.net/comparison.html (http://flac.sourceforge.net/comparison.html)
Title: New lossless codec comparison (Jan '13)
Post by: knutinh on 2013-01-19 21:30:47
what kind of pc hardware did you use? did you use special compiles for that hardware?

regards
k
Title: New lossless codec comparison (Jan '13)
Post by: Fabiolander on 2013-01-19 21:33:53
What a job !

Thank you but could you translate the results for newbees ? 

Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-01-19 21:41:27
what kind of pc hardware did you use? did you use special compiles for that hardware?

The PC was a HP Elitebook 8530w, unmodified. It features an Intel Core2Duo T9600 with 4GB of ram. The CPU has been clocked down to 2.13GHz to avoid thermal throttle. The OS was Kubuntu Linux 12.10 64-bit. The FLAC and WavPack binaries were packed with Kubuntu (64-bit ones, but not optimized for any kind of hardware), all other codecs (except ALAC, WMA Lossless and Real Audio) were run with native Linux binaries or over Wine. The three exceptions were tested on a Windows box, but please refer to the PDF if you want to know more.

Thank you but could you translate the results for newbees ? 

I thought the comparison on the FLAC-website (http://flac.sourceforge.net/comparison.html) is quite accessible to newbies? Can you clarify what it is that needs translation?
Title: New lossless codec comparison (Jan '13)
Post by: romor on 2013-01-19 21:46:44
Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.

Thanks for sharing your results. I believe there is no much interest, because lossless tests were discussed recently

If I may comment on latex article: it has unnecessary huge margins if intended for screen reading, which coupled with many formats tested and bitmap instead vector graphs, makes some graphs hardly readable. I wish more tables then graphs
Title: New lossless codec comparison (Jan '13)
Post by: urak on 2013-01-19 23:29:57
but only one reply? I hope that PDF wasn't too overwhelming. 

To me the charts are very confused (poor color readability, squares too small). Thanks
Title: New lossless codec comparison (Jan '13)
Post by: Gregory S. Chudov on 2013-01-20 08:21:41
Great job!
Shameless self-promotion on my part follows:
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.
Title: New lossless codec comparison (Jan '13)
Post by: C.R.Helmrich on 2013-01-20 11:18:06
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

Indeed, I would have liked to see that comparison as well to find out how the claim "capable of so much more" translates into numbers.

ktf, other than that you comparison (e.g. using a ramdisk) is very well done! But the amount of data is indeed a bit overwhelming. Can you briefly explain what changed in the outcome compared to your own 2009 (http://www.icer.nl/losslesstest/old_index.html) and Synthetic Soul's 2008 (http://synthetic-soul.co.uk/comparison/lossless/) test? I see the inclusion of a RealAudio lossless codec, but other than that the ranking seems to be the same? Meaning TAK and FLAC are most efficient, ending up to the lower right of the scale most of the time?

Chris


Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-01-20 13:31:31
Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.

I haven't had any problems lately?

Quote
and bitmap instead vector graphs, makes some graphs hardly readable.

To me the charts are very confused (poor color readability, squares too small).

I know of a few PDF-readers which do not scaling bitmap-images properly indeed, however, the toolkit I used was bitmap only. The problem with bigger squares is that they tend to overlap (which they already do with this size)


It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

I tried, but couldn't get it running on my notebook. I should try harder I guess. The problem is that there are a few FLAC-encoders out there, but including them might be confusing. Not everyone understands the difference between an codec and a format, just see MP3-vs-LAME: People complain about MP3 while they are actually complaining about the rubbish encoder some used. I might include it in the next revision of this document.

Can you briefly explain what changed in the outcome compared to your own 2009 (http://www.icer.nl/losslesstest/old_index.html) and Synthetic Soul's 2008 (http://synthetic-soul.co.uk/comparison/lossless/) test?

Nothing stunning really. TAK is even faster now on encoding and ALS -7 didn't work as well as it did last time. So nothing surprising on the part of CD-audio. However, in the PDF there are a few graphs on 96kHz/24bit audio ("High-res") and multichannel audio. Nothing very exciting there either, but some codecs have some pecularities, for example, FLAC being slower than usual on both encoding and decoding when compared to other codecs. There's a note on the performance of MLP (that's the codec used for Audio DVD's) too. The only unusual results are on stereo-encoded mono, which was already known to be a problem with some encoders.

So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.
Title: New lossless codec comparison (Jan '13)
Post by: Dynamic on 2013-01-21 10:17:53
So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.


I suspect a lot of that is less to do the genre's musical nature and more to do with the loudness war (continuously filling all 16 bits for metal, usually using the lowest 12 to 14 bits for choral, leaving about 2 to 4 bits of extra random/unpredictable noise to encode much of the time). I suspect that heavy metal and choral music both adjusted to, say, 83 dB SPL Album Gain level, would be a much closer match. (This tends to be supported by lossyWAV's results, and pre-lossyWAV tests of applying Album Gain (technically a loss of data, albeit mostly noise) before lossless encoding)
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-01-21 13:42:11
I suspect a lot of that is less to [..]
What do you mean by 'that'? The average compression achieved or the observation that no codec performs much better on certain music? In the first case, I don't agree (see below), for the latter, I don't understand what you mean.

For example, take a look at the graphs (see the raw material ZIP) for Rush - Grace under Pressure and Howard Shore - The Hobbit: An Unexpected Journey. Both are DR 10 (see here for more information on DR (http://www.jokhan.demon.nl/DynamicRange/index.htm)),  the average compression of the first ('80 rock) is about 70% and for the latter (2012 soundtrack material) it is 51%. As the DR measure is the same for both, a difference this large (19 percentage points!) can't be assumed to be side-effects of the loudness war only. However, still, the graphs look very similar when comparing codecs amongst each other.

edit: I just checked the replaygain values of both albums, for the first it is -7.0dB, for the second it is -4.9dB. That's a difference of 2.1dB. If we assume every 6dB (or 1 bit) of noise added halves the achieved compression, then "adding 2.1dB of noise" to the soundtrack would result in a compression ratio of  1 - ( 1 - 0.51 ) / 10 ^ (2.1 / 20) = 62%. This is still 8 percentage point away from the observed 70%, so I can't be all loudness war. But then again, ReplayGain is probably not a good measure either...

edit2: I just measured the RMS, for Rush it's -14.6dB and for Howard Shore its -13.6dB, so it's even the other way round: the one that is compressed more has a higher RMS in this case...
Title: New lossless codec comparison (Jan '13)
Post by: Dynamic on 2013-01-21 20:13:06
I was meaning the average compression for a particular genre and that the lossless compressed bitrate had a lot to do with the average loudness (in my experience, Replay Gain was a good benchmark) as you surmised.

I've got the impression that lossless bitrates above about 900 kbps were pretty rare in the late 90s and early 2000s CDs, but are now commonplace in much of the 2010s popular releases, with over 1000kbps being pretty frequent.

I'm not familiar with how the DR value or even its version of RMS is calculated in detail (e.g. gated or not), which is sometimes necessary, because it mentions things like ignoring long-term dynamics and presumably combining measurments reflecting medium term variations and transient attacks to derive the DR number. People get in trouble making assumptions about the ReplayGain method too.

I'm not sure what type of thing is on that soundtrack. One thing that tends to be the case on albums with multiple CD releases over the decades is that a 1983 release, a 1993 release, a 2003 release, and a 2013 release say, would tend to get louder decade by decade (so they sound loud enough on a CD changer, I guess) and the lossless bitrate would tend to increase too.

My expectation (based on only my anecdotal evidence and selective memory, and potentially a fautly mental model of why!) is that there's a tendency that any popular music format (pop, rock, dance) follows that trend to increased loudness at first, then finally reduced dynamic range, and I wouldn't be surprised that picking out those genres only it might be possible to plot a graph versus year.

One neat idea for someone with the right CD collection is picking all versions of a long-running pop compilation such as the UK & Eire's "Now, That What I Call Music!" (http://en.wikipedia.org/wiki/Now_That%27s_What_I_Call_Music!#Original_United_Kingdom_.26_Republic_of_Ireland_series) pressed from the mid 80s volume 10 double CD to the present day volume 83) and plotting bitrates of the lossless files versus date, we'd see a steady increase in loudness and bitrate over the years (as a trend - there's bound to be variation), and almost certainly a decline in DR value too (albeit that it seems fairly constrained).

x-axis = Date of Release, Left y-axis = FLAC bitrate, Right y-axis = ReplayGain value
Dual plot - bitrate vs date, RG Album Gain vs date (or versus the NOW! number, as in NOW! 36)
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-03-19 08:44:16
If I may comment on latex article: it has unnecessary huge margins if intended for screen reading, which coupled with many formats tested and bitmap instead vector graphs, makes some graphs hardly readable. I wish more tables then graphs

To me the charts are very confused (poor color readability, squares too small). Thanks


I have updated the PDF, this time it is optimized for screen reading and readability of the graphs has been improved. No changes have been made to the contents (so no new test results). New version download (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%202.pdf)
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-03-21 21:10:31
Once again hi all,

Today I've been busy trying to add refalac. Does anyone have an opinion about which switches to add? I only saw --fast, are there any other? Can't find them in the usage info.
Title: New lossless codec comparison (Jan '13)
Post by: lvqcl on 2013-03-22 14:09:16
BTW, there are command-line WMAL encoder (http://www.hydrogenaudio.org/forums/index.php?showtopic=90519) and decoder (http://forum.doom9.org/showthread.php?t=140273) (both require Windows Media Format runtime (http://www.citizeninsomniac.com/WMV/#Codecs))


...and if you want to use Windows for WMAL tests: you can use timer.exe from 7-Benchmark (http://sourceforge.net/projects/sevenmax/files/7-Benchmark/) (7bench1200.7z)
Title: New lossless codec comparison (Jan '13)
Post by: spicymeatball77 on 2013-04-05 14:26:13
I've got a bit of a noob question.  I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life.  Right now I'm using FLAC-8.  I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
Title: New lossless codec comparison (Jan '13)
Post by: Dynamic on 2013-04-05 17:13:27
Yes, and the flac -8 point is the leftmost value on FLAC, though all are pretty close. While it doesn't translate 1-for-1 to battery life, decoding speed is a strong indicator (well correlated).
Title: New lossless codec comparison (Jan '13)
Post by: Propheticus on 2013-04-05 18:10:54
If I look at those graphs I see no point in using anything higher than FLAC -5. The extra compression you get is minimal and en-/decoding takes longer. Quite extreme case of diminishing returns I'd say.
Title: New lossless codec comparison (Jan '13)
Post by: marc2003 on 2013-04-05 18:33:57
I've got a bit of a noob question.  I just RockBox'd a Sansa Clip+


admittedly old now (2010) but there have been benchmarks done already.

http://www.rockbox.org/wiki/CodecPerforman...ARM926EJ_45S_41 (http://www.rockbox.org/wiki/CodecPerformanceComparison#AMS_AS3525v2_w_47_40MHz_PClK_40ARM926EJ_45S_41)

Code: [Select]
flac_5.flac     2936.56% realtime     Decode time - 5.99s     8.17MHz
flac_8.flac     2748.43% realtime     Decode time - 6.40s     8.73MHz


you can always run tests yourself to see if there have been any improvements to the rockbox code in the intervening years.

Title: New lossless codec comparison (Jan '13)
Post by: db1989 on 2013-04-05 18:34:31
If I look at those graphs I see no point in using anything higher than FLAC -5. The extra compression you get is minimal and en-/decoding takes longer. Quite extreme case of diminishing returns I'd say.
I think you’re actually talking about -4. In the upper graph, that’s the one with about 135 degrees between it and the next setting on the right, and as Dynamic said, the graph goes from -8 on the left to -0 on the right. The lower graph seems to have only 8 settings represented (rather than 9), although I presume that’s just a result of visual obstruction as some of them are very close together.
Title: New lossless codec comparison (Jan '13)
Post by: Propheticus on 2013-04-05 19:15:16
Yes -4 is the most clear point as you said. But -5 seems to be the point after which the graph levels out (asymptotic almost)*. So anything higher is virtually wasted effort.

*also see the document and individual cases.
Title: New lossless codec comparison (Jan '13)
Post by: spicymeatball77 on 2013-04-05 20:13:43
I transcoded my FLACs from -8 to -5 and there was *very* little size increase.  This will likely help out my battery life at a tiny size cost. Thanks for the input guys.
Title: New lossless codec comparison (Jan '13)
Post by: Soap on 2013-04-05 23:44:29
I transcoded my FLACs from -8 to -5 and there was *very* little size increase.  This will likely help out my battery life at a tiny size cost. Thanks for the input guys.


saratoga can probably speak as to the specifics of that player, but on Porta Player devices the difference in decode speed would equal ZERO, not a little, but zero power savings as the CPU doesn't clock lower than 30Mhz.  Whereas the size difference would equal (probably insignificant but) real power consumption differences due to increased storage access.  Flash obviously cuts the power consumption of storage access down vs spinning rust, but the number is real.
Title: New lossless codec comparison (Jan '13)
Post by: ChronoSphere on 2013-04-07 14:31:30
I've got a bit of a noob question.  I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life.  Right now I'm using FLAC-8.  I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
I've ran a test on my clip+ with an album encoded with flac -11 (CUETools's flaCL) and the same with wavpack -hhx and the difference was only about 10 minutes in runtime, so I'd say lossless encoding options don't affect battery life that much anymore.
Title: New lossless codec comparison (Jan '13)
Post by: TBeck on 2013-06-21 22:19:58
I should have replied a lot earlier, because your publication immediately made me very happy.

Eventually a lossless codec comparison i feel i can trust!

In the past it was Synthetic Soul's comparison, although his sample selection was not sufficiently diverse (at least for my purposes).

When i am thinking about tuning opportunities for my codec, i often look at your results.

I think, you aren't  getting as much feedback as you deserved here at hydrogen, but hey, your comparison has been published on the FLAC web site, and i am sure, the guys over there too know what constitutes a meaningful lossless codec comparison.

I must admit, i am selfish. I wanted to thank you but i also want to encourage you to continue your valuable work. Because it's so important to me, to have an independent instance, which verifies the outcome of my work.

Again

thank you! 

  Thomas
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-06-22 10:43:00
I should have replied a lot earlier, because your publication immediately made me very happy.

Eventually a lossless codec comparison i feel i can trust!

Well, that's nice to hear!

Quote
In the past it was Synthetic Soul's comparison, although his sample selection was not sufficiently diverse (at least for my purposes).

As the issue is brought up now, I would like to ask for a little help once again. I've been looking for more music to make the sample selection even more diverse, and I now have this list of 43 albums, where the previous comparison was with 29 albums.



The bold items are new. I think this is a nice list so far, any comments?

Quote
I must admit, i am selfish. I wanted to thank you but i also want to encourage you to continue your valuable work. Because it's so important to me, to have an independent instance, which verifies the outcome of my work.

But the comparison wasn't totally fair yet. I ran your codec in Wine, and while one could argue Wine doesn't add much overhead, it's a little weird I didn't run FLAC trough wine too, for example.

Anyway, the next version of this comparison will have all codecs tested on a Windows computer, as I finally figured out a utility that reported the CPU cycles used by a certain program instead of time elapsed. I rewrote my scripts to fit in a Windows enviroment, technically I'm ready to run it, but I am waiting for WavPack 4.70 final to be released (so I can test new versions of FLAC, TAK, Monkeys Audio and WavPack), so the result will be published somewhere this summer I suppose.

If there are still any remarks on the test setup, please tell me. I can still fix things now

edit: oh, and for the record, I will be testing on per-track basis instead of per-album basis now. Every album will be weighted equally just as the previous comparison, but now I will be able to identify outliers. Maybe something interesting pops up.
Title: New lossless codec comparison (Jan '13)
Post by: ozok on 2013-06-22 10:49:32
...
Anyway, the next version of this comparison will have all codecs tested on a Windows computer, as I finally figured out a utility that reported the CPU cycles used by a certain program instead of time elapsed.
...

What is the name of this utility?
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-06-22 11:07:28
What is the name of this utility?

It's timeit.exe, from the Windows Server 2003 Resource Kit Tools. However, it doesn't work with Windows 7 (I run Windows XP) but I found something that indicated powershell might have similar capabilities, but I didn't test. If anyone is going to dive into this, please report any finding, I'm not entirely sure this timeit utility reports are resembling the truth. It says process time, but the results I got back were sometimes weird for no apparent reason.
Title: New lossless codec comparison (Jan '13)
Post by: ChronoSphere on 2013-06-22 13:16:24
I've got a bit of a noob question.  I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life.  Right now I'm using FLAC-8.  I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
I've ran a test on my clip+ with an album encoded with flac -11 (CUETools's flaCL) and the same with wavpack -hhx and the difference was only about 10 minutes in runtime, so I'd say lossless encoding options don't affect battery life that much anymore.
To update on this, I ran some more tests and the differences are bigger. I found out flac -11 sometimes fails to decode on the clip+ (probably because it's not a subset file), so I switched to flac -8 and battery wise, flac is a clear winner:

Code: [Select]
Flac -8:         16:09:49
Wv -hh -b384x:    11:17:12
mp3 v0:            11:48:20
vorbis q5:        12:17:56
opus 160:        10:17:00


Complete battery_bench logs here (https://www.dropbox.com/s/lnhzisi3tgy217d/Clip%2B%20battery%20bench.zip)
Title: New lossless codec comparison (Jan '13)
Post by: marc2003 on 2013-06-22 13:35:52
It's timeit.exe, from the Windows Server 2003 Resource Kit Tools. However, it doesn't work with Windows 7 (I run Windows XP) but I found something that indicated powershell might have similar capabilities, but I didn't test.


just yesterday i was wondering if there was a tool for benchmarking command line programs and i found this.

I use timer.exe from 7-Benchmark: http://sourceforge.net/projects/sevenmax/files/7-Benchmark/ (http://sourceforge.net/projects/sevenmax/files/7-Benchmark/)


it works on windows 7. i also tried the powershell command. you can see the results here - i did a quick test of flac and takc on a single album image.

https://dl.dropboxusercontent.com/u/2280132.../june/bench.png (https://dl.dropboxusercontent.com/u/22801321/2013/june/bench.png)
Title: New lossless codec comparison (Jan '13)
Post by: Wombat on 2013-06-22 14:13:27
I hope this time your Windows computer can run flacCL. Would be nice to see it included.
Title: New lossless codec comparison (Jan '13)
Post by: ozok on 2013-06-22 15:57:35
Thanks ktf.
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-06-23 08:16:31
just yesterday i was wondering if there was a tool for benchmarking command line programs and i found this.

Yes, I saw that one too, but the problem is it reports running time, not CPU time used. I just took another look at powershell, it seems to report real time as well, not CPU time.

I hope this time your Windows computer can run flacCL. Would be nice to see it included.

I'm sorry, the test needs several weeks to run, so I can only use an old computer that's not longer used from before the GPGPU era to do this work.

However, I might as well add a section on different FLAC-encoders run on a different computer, but I will have to work that out.
Title: New lossless codec comparison (Jan '13)
Post by: lvqcl on 2013-06-23 08:51:20
Yes, I saw that one too, but the problem is it reports running time, not CPU time used.

I thought that "Process Time" is CPU time. What is the difference between them?
Title: New lossless codec comparison (Jan '13)
Post by: [JAZ] on 2013-06-23 10:19:06
Since common PCs do have more than one running core for several years now, there is a distinction between the time required for a task (elapsed time) and the time required for one working unit to complete it (cpu time).

In Windows, there's a gauge showing the usage from 0 to 100%, but in linux, the "top" utility can show usages of 150%, 200% or more. This is translated as using more than the time that a single working unit is able to provide.
Title: New lossless codec comparison (Jan '13)
Post by: lvqcl on 2013-06-23 11:25:15
I'm sorry, the test needs several weeks to run, so I can only use an old computer that's not longer used from before the GPGPU era to do this work.

What CPU does it have and what instructions does it support (SSE, SSE2, etc.)?
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-06-23 14:10:35
I thought that "Process Time" is CPU time. What is the difference between them?
You're right. I don't know why I concluded that 7bench is not able to measure CPU-time, because it clearly can. It does give the same results as timeit (and the same pecularities, like having a smallest increment of 0.015s at my system) so I suppose it uses the same kernel hooks.

What CPU does it have and what instructions does it support (SSE, SSE2, etc.)?
If you really want to know, the introduction date of this processor was March 10, 2005, I bought the laptop around the time the first dual-cores hit the mainstream market. Details are below. Wikipedia says: MMX, Enhanced 3DNow!, SSE, SSE2, SSE3, AMD64, PowerNow!, NX Bit
Code: [Select]
processor       : 0
vendor_id      : AuthenticAMD
cpu family      : 15
model          : 36
model name      : AMD Turion™ 64 Mobile Technology ML-34
stepping        : 2
cpu MHz        : 1800.141
cache size      : 1024 KB
fpu            : yes
fpu_exception  : yes
cpuid level    : 1
wp              : yes
flags          : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
lm 3dnowext 3dnow rep_good pni lahf_lm
bogomips        : 3600.28
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes  : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
Title: New lossless codec comparison (Jan '13)
Post by: TBeck on 2013-06-23 22:14:45
As the issue is brought up now, I would like to ask for a little help once again. I've been looking for more music to make the sample selection even more diverse, and I now have this list of 43 albums, where the previous comparison was with 29 albums.

Usually more is better. As long as the codec relevant audio properties and there share of all audio files can't be fully determined, i don't know a better way to improve the representativity.

Do you know if your selection contains some hypercompressed files?  Since they are so common know, this seems to be one more important category. I don't really like to ask for it, because i know that TAK performs less well on them...

I don't thinky you would need plenty such files. From my experience the effect of other codec relevant audio properties usually is quite small, if a file is hypercompressed.

But the comparison wasn't totally fair yet. I ran your codec in Wine, and while one could argue Wine doesn't add much overhead, it's a little weird I didn't run FLAC trough wine too, for example.

I wouldn't expect a significant effect. I have examined some comparisons which ran in Wine, and the results were in line with windows comparisons. I really like that you care so much about so many details, but i think it's nearly impossible to create a practicable comparison that everybody regards as fair. One could criticize, that you will be using a quite old cpu, because it lacks some instruction set extensions or because possibly newer compilers will not generate optimal code for it. On the other hand there are also good arguments against the use of a newer cpu. But someone will always complain.

I myself feel comfortable with your cpu choice. While it dosen't support SSSE3, what will make TAK's stronger presets somewhat slower, it's microarchitecture doesn't differ too much from more recent cpus.

Well, i probably would protest, if you were using one of those strange Pentium 4 - cpus...

edit: oh, and for the record, I will be testing on per-track basis instead of per-album basis now. Every album will be weighted equally just as the previous comparison, but now I will be able to identify outliers. Maybe something interesting pops up.

Great!
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-06-23 23:12:55
Do you know if your selection contains some hypercompressed files?  Since they are so common know, this seems to be one more important category. I don't really like to ask for it, because i know that TAK performs less well on them...

I added Metallica's Death Magnetic just because it is probably the best known example of hyper-compressed stuff.

Quote
I myself feel comfortable with your cpu choice. While it dosen't support SSSE3, what will make TAK's stronger presets somewhat slower, it's microarchitecture doesn't differ too much from more recent cpus.

Yeah, another complaint might be that I run in 32-bit, while 64-bit is more or less standard these days. Sadly I don't have other machines available. It's either this with Windows XP or the setup as used in the previous comparison. The first results are very, very similar to my 64-bit linux environment (Intel Core2Duo T9600), the only codec that gets a significant performance hit is FLAC's decoding, which is about 20% slower.
Title: New lossless codec comparison (Jan '13)
Post by: ktf on 2013-08-11 11:19:01
I just finished a new revision of this codec comparison. It features new versions of FLAC, TAK, Monkey's Audio, WavPack and OptimFROG, runs on Windows instead of Linux and contrary to what I said earlier, the codecs were run on an AMD A4-3400. Furthermore, the test corpus has been expanded from 29 albums to 43 albums. WMA Lossless is now tested properly, refalac has been added and Real Lossless has been dropped. (edit: OS was Windows 7 SP1 64-bit)

No big changes in the results, except TAK being a whole lot faster.

The PDF and raw results are available here: http://www.icer.nl/losslesstest (http://www.icer.nl/losslesstest)

(http://www.icer.nl/misc_stuff/losslesscompv3-encoding.png)
(http://www.icer.nl/misc_stuff/losslesscompv3-decoding.png)
Title: New lossless codec comparison (Jan '13)
Post by: ChronoSphere on 2013-08-11 11:31:18
It's interesting to see that TAK starts where FLAC stops (decoding vs. compression), it's like they're subsets of the same codec.
Title: New lossless codec comparison (Jan '13)
Post by: DARcode on 2013-08-11 12:43:09
Fantastic work, many thanks for your time and sharing it, keep it up.
Title: New lossless codec comparison (Jan '13)
Post by: Propheticus on 2013-08-11 16:04:33
Interesting to see the 4th node of TAK (from right to left) is faster than the 2nd and 3rd node while offering better compression. This renders these two, but especially the 3rd, rather useless IMO. Same counts for the 5th and 6th (7th is better and faster), and the 9th (10th is better).
Title: New lossless codec comparison (Jan '13)
Post by: TBeck on 2013-08-11 17:49:53
From TAK's Readme:

Quote
Each preset selects a set of internal encoder options. Some of them affect only the encoding speed, the others also the decoding speed.

Slower decoding usually means higher hardware requirements (memory size and/or cpu power) for a playback device. Some devices may be too weak to support the more demanding internal encoder options. But any option, which only affects the encoding speed and complexity, is always applicable.

Therefore each preset consists of two components:
  • The profile selects internal encoder options which affect both encoding and decoding speed. Profiles are declared as a number from 0 to 4 (fastest to strongest).
  • The evaluation level selects only internal encoder options, which have no effect on the decoding speed. The evaluation levels are named Standard, Extra and Max and abbreviated as s, e and m (fastest to strongest).

A hardware manufacturer supporting TAK has only to specify the strongest profile it's hardware can decode, because the evaluation level does not affect the hardware requirements.

Hint: If you want higher compression and fast encoding, and are able to accept some decrease in decoding speed, it is usually preferable to select a higher profile instead of increasing the evaluation level.
Title: New lossless codec comparison (Jan '13)
Post by: Propheticus on 2013-08-11 23:47:40
The CPU used in this test might have been too fast compared to mobile devices to show, but I don't see any big variations in the decoding graph between the mentioned profiles.
Title: New lossless codec comparison (Jan '13)
Post by: TBeck on 2013-08-12 08:25:35
Depending on the profile TAK is using up to 4, 12, 32, 80 or 160 predictors for it's primary filter. A corresponding number of multiplications per sample has to be performed. FLAC supports up to 32 predictors, but it's restricted subset which is recommended for maximum compatibility is limited to 12 (preset -8). And i remember reports on devices not capable to cope with 32 predictors.

I have no practical experience with the DSPs or cpus used in mobile devices, but from what i have read here at hydrogen and at Rockbox, higher predictor counts seem to be critical.

Indeed the decoding speed results show only a minor effect of the perdictor count, but that's misleading. Reasons:



BTW: I think, TAK could be faster in this test. On the AMD A4-3400 it will only use MMX instructions although SSE2 is supported. Currently TAK will only use SSE2 is SSSE3 is supported too. My fault...
Title: New lossless codec comparison (Jan '13)
Post by: Propheticus on 2013-08-12 08:59:04
Thats a clear explanation. The lack of difference makes sense now. Thanks.