Hi all,
For the last few weeks I've been busy writing a new lossless audio codec comparison, after lots of talk about it here (http://www.hydrogenaudio.org/forums/index.php?showtopic=94685) and here (http://www.hydrogenaudio.org/forums/index.php?showtopic=71612).
The full test report can be found here (PDF) (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%201.pdf), the raw data and graphs can be found here (zip) (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20raw%20data%20revision%201.zip).
While the full report has lots of graphs it also has lots of text, so the 'main' graphs of the report are posted below.
(http://www.icer.nl/losslesstest/all-enc.png)
(http://www.icer.nl/losslesstest/all-dec.png)
The CDs used
- 30 Seconds to Mars - This Is War (rock)
- Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
- Daft Punk - Alive 2007 (electronic, live)
- Dan Brown - Angels and Demons (audiobook)
- Enya - Amarantine (new age)
- Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
- Gilberto Santa Rosa - Esencia (salsa)
- Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (classical, romantic)
- Howard Shore - The Hobbit: An Unexpected Journey (soundtrack, orchestral)
- Iron Maiden - Brave New World (metal)
- John Coltrane - Live at the half note, vol. 1 (jazz)
- Joss Stone - Mind, Body & Soul (soul)
- J.S. Bach - Magnificat (orchestral, baroque)
- Koninklijke Militaire Kapel - [no name] (military brass, vinyl rip)
- Kraftwerk - Autobahn (electronic)
- Lana del Rey - Born to Die (pop)
- Lil Wayne - Tha Carter III (hip hop)
- Martin O'Donnell & Michael Salvatori - Halo: Reach O.S.T. (ambient-ish, soundtrack)
- Michael Bublé; - meets madison square garden (pop, live)
- Mike Oldfield - Tubular Bells (not classifiable)
- Rosenberg Trio - Djangologists (jazz)
- Rush - Grace under Pressure (1997 remaster) (rock)
- System of a Down - Mezmerize (metal)
- Szakcsi - Virágom, virágom (world music)
- Tiësto - In Search of Sunrise 7: Asia (dance)
- Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
- Various - Latin Village, CD 1: Salsa (compilation of salsa)
- Xzibit - At the Speed of Life (hip hop)
- Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism, soundtrack)
For more information on why I chose these CDs, what strange things happened with the pure speech data (Dan Browns audiobook) and tests on 96kHz/24-bit or surround sound audio, check the full report (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%201.pdf).
Its labeled revision 1 because I intend to keep it updated. Any comment (on my grammar too, as I'm not a native speaker) is welcome.
ktf,
Thank You for sharing interesting comparison.
What decoder and/or application have You used for TAK? Last time TAK native app decodes as fast as FLAC. TAK foobar plugin was somewhat slower.
I was curious to try Monkey's Audio c3000 ('High') on Galaxy II phone with Rockbox app for Android . Got 17 hours of a battery life. And it uses the lowest frequency step so I suspect APE 'Extra High' would last comparable time. Given that an energy efficiency of proccesors grows fast it's not crazy to think about Monkey's Audio on mobile devices. Anyway I don't use lossless on portable and use FLAC lossless storage for a fast transcoding to lossy formats.
What decoder and/or application have You used for TAK? Last time TAK native app decodes as fast as FLAC.
I've used the Takc.exe command line encoder which I downloaded straight from TAK's homepage. This was run over Wine, but that shouldn't make much of a difference, as Wine only emulates system calls.
The reason that TAK decodes as fast as FLAC with you is probably because not your CPU but your harddisk is the bottleneck. In this test, CPU-time was measured and the test were done with a ramdisk to bypass the harddisk, so this might be the difference. After all, those codecs decode so fast on desktop CPU's that harddisks can't keep up.
Regarding Monkey's Audio, I've seen some devices advertising supporting it recently, so your observation is correct.
I know lossless audio codec comparisons have lost interest with most of you, but only one reply? I hope that PDF wasn't too overwhelming.
Anyway, a short version of this comparison (and the PDF) is on the FLAC website now: http://flac.sourceforge.net/comparison.html (http://flac.sourceforge.net/comparison.html)
what kind of pc hardware did you use? did you use special compiles for that hardware?
regards
k
What a job !
Thank you but could you translate the results for newbees ?
what kind of pc hardware did you use? did you use special compiles for that hardware?
The PC was a HP Elitebook 8530w, unmodified. It features an Intel Core2Duo T9600 with 4GB of ram. The CPU has been clocked down to 2.13GHz to avoid thermal throttle. The OS was Kubuntu Linux 12.10 64-bit. The FLAC and WavPack binaries were packed with Kubuntu (64-bit ones, but not optimized for any kind of hardware), all other codecs (except ALAC, WMA Lossless and Real Audio) were run with native Linux binaries or over Wine. The three exceptions were tested on a Windows box, but please refer to the PDF if you want to know more.
Thank you but could you translate the results for newbees ?
I thought the comparison on the FLAC-website (http://flac.sourceforge.net/comparison.html) is quite accessible to newbies? Can you clarify what it is that needs translation?
Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.
Thanks for sharing your results. I believe there is no much interest, because lossless tests were discussed recently
If I may comment on latex article: it has unnecessary huge margins if intended for screen reading, which coupled with many formats tested and bitmap instead vector graphs, makes some graphs hardly readable. I wish more tables then graphs
but only one reply? I hope that PDF wasn't too overwhelming.
To me the charts are very confused (poor color readability, squares too small). Thanks
Great job!
Shameless self-promotion on my part follows:
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.
Indeed, I would have liked to see that comparison as well to find out how the claim "capable of so much more" translates into numbers.
ktf, other than that you comparison (e.g. using a ramdisk) is very well done! But the amount of data is indeed a bit overwhelming. Can you briefly explain what changed in the outcome compared to your own 2009 (http://www.icer.nl/losslesstest/old_index.html) and Synthetic Soul's 2008 (http://synthetic-soul.co.uk/comparison/lossless/) test? I see the inclusion of a RealAudio lossless codec, but other than that the ranking seems to be the same? Meaning TAK and FLAC are most efficient, ending up to the lower right of the scale most of the time?
Chris
Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.
I haven't had any problems lately?
and bitmap instead vector graphs, makes some graphs hardly readable.
To me the charts are very confused (poor color readability, squares too small).
I know of a few PDF-readers which do not scaling bitmap-images properly indeed, however, the toolkit I used was bitmap only. The problem with bigger squares is that they tend to overlap (which they already do with this size)
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.
I tried, but couldn't get it running on my notebook. I should try harder I guess. The problem is that there are a few FLAC-encoders out there, but including them might be confusing. Not everyone understands the difference between an codec and a format, just see MP3-vs-LAME: People complain about MP3 while they are actually complaining about the rubbish encoder some used. I might include it in the next revision of this document.
Can you briefly explain what changed in the outcome compared to your own 2009 (http://www.icer.nl/losslesstest/old_index.html) and Synthetic Soul's 2008 (http://synthetic-soul.co.uk/comparison/lossless/) test?
Nothing stunning really. TAK is even faster now on encoding and ALS -7 didn't work as well as it did last time. So nothing surprising on the part of CD-audio. However, in the PDF there are a few graphs on 96kHz/24bit audio ("High-res") and multichannel audio. Nothing very exciting there either, but some codecs have some pecularities, for example, FLAC being slower than usual on both encoding and decoding when compared to other codecs. There's a note on the performance of MLP (that's the codec used for Audio DVD's) too. The only unusual results are on stereo-encoded mono, which was already known to be a problem with some encoders.
So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.
So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.
I suspect a lot of that is less to do the genre's musical nature and more to do with the loudness war (continuously filling all 16 bits for metal, usually using the lowest 12 to 14 bits for choral, leaving about 2 to 4 bits of extra random/unpredictable noise to encode much of the time). I suspect that heavy metal and choral music both adjusted to, say, 83 dB SPL Album Gain level, would be a much closer match. (This tends to be supported by lossyWAV's results, and pre-lossyWAV tests of applying Album Gain (technically a loss of data, albeit mostly noise) before lossless encoding)
I suspect a lot of that is less to [..]
What do you mean by 'that'? The average compression achieved or the observation that no codec performs much better on certain music? In the first case, I don't agree (see below), for the latter, I don't understand what you mean.
For example, take a look at the graphs (see the raw material ZIP) for Rush - Grace under Pressure and Howard Shore - The Hobbit: An Unexpected Journey. Both are DR 10 (see here for more information on DR (http://www.jokhan.demon.nl/DynamicRange/index.htm)), the average compression of the first ('80 rock) is about 70% and for the latter (2012 soundtrack material) it is 51%. As the DR measure is the same for both, a difference this large (19 percentage points!) can't be assumed to be side-effects of the loudness war only. However, still, the graphs look very similar when comparing codecs amongst each other.
edit:
I just checked the replaygain values of both albums, for the first it is -7.0dB, for the second it is -4.9dB. That's a difference of 2.1dB. If we assume every 6dB (or 1 bit) of noise added halves the achieved compression, then "adding 2.1dB of noise" to the soundtrack would result in a compression ratio of 1 - ( 1 - 0.51 ) / 10 ^ (2.1 / 20) = 62%. This is still 8 percentage point away from the observed 70%, so I can't be all loudness war. But then again, ReplayGain is probably not a good measure either...
edit2: I just measured the RMS, for Rush it's -14.6dB and for Howard Shore its -13.6dB, so it's even the other way round: the one that is compressed more has a higher RMS in this case...
I was meaning the average compression for a particular genre and that the lossless compressed bitrate had a lot to do with the average loudness (in my experience, Replay Gain was a good benchmark) as you surmised.
I've got the impression that lossless bitrates above about 900 kbps were pretty rare in the late 90s and early 2000s CDs, but are now commonplace in much of the 2010s popular releases, with over 1000kbps being pretty frequent.
I'm not familiar with how the DR value or even its version of RMS is calculated in detail (e.g. gated or not), which is sometimes necessary, because it mentions things like ignoring long-term dynamics and presumably combining measurments reflecting medium term variations and transient attacks to derive the DR number. People get in trouble making assumptions about the ReplayGain method too.
I'm not sure what type of thing is on that soundtrack. One thing that tends to be the case on albums with multiple CD releases over the decades is that a 1983 release, a 1993 release, a 2003 release, and a 2013 release say, would tend to get louder decade by decade (so they sound loud enough on a CD changer, I guess) and the lossless bitrate would tend to increase too.
My expectation (based on only my anecdotal evidence and selective memory, and potentially a fautly mental model of why!) is that there's a tendency that any popular music format (pop, rock, dance) follows that trend to increased loudness at first, then finally reduced dynamic range, and I wouldn't be surprised that picking out those genres only it might be possible to plot a graph versus year.
One neat idea for someone with the right CD collection is picking all versions of a long-running pop compilation such as the UK & Eire's "Now, That What I Call Music!" (http://en.wikipedia.org/wiki/Now_That%27s_What_I_Call_Music!#Original_United_Kingdom_.26_Republic_of_Ireland_series) pressed from the mid 80s volume 10 double CD to the present day volume 83) and plotting bitrates of the lossless files versus date, we'd see a steady increase in loudness and bitrate over the years (as a trend - there's bound to be variation), and almost certainly a decline in DR value too (albeit that it seems fairly constrained).
x-axis = Date of Release, Left y-axis = FLAC bitrate, Right y-axis = ReplayGain value
Dual plot - bitrate vs date, RG Album Gain vs date (or versus the NOW! number, as in NOW! 36)
If I may comment on latex article: it has unnecessary huge margins if intended for screen reading, which coupled with many formats tested and bitmap instead vector graphs, makes some graphs hardly readable. I wish more tables then graphs
To me the charts are very confused (poor color readability, squares too small). Thanks
I have updated the PDF, this time it is optimized for screen reading and readability of the graphs has been improved. No changes have been made to the contents (so no new test results).
New version download (http://www.icer.nl/losslesstest/Lossless%20audio%20codec%20comparison%20-%20revision%202.pdf)
Once again hi all,
Today I've been busy trying to add refalac. Does anyone have an opinion about which switches to add? I only saw --fast, are there any other? Can't find them in the usage info.
BTW, there are command-line WMAL encoder (http://www.hydrogenaudio.org/forums/index.php?showtopic=90519) and decoder (http://forum.doom9.org/showthread.php?t=140273) (both require Windows Media Format runtime (http://www.citizeninsomniac.com/WMV/#Codecs))
...and if you want to use Windows for WMAL tests: you can use timer.exe from 7-Benchmark (http://sourceforge.net/projects/sevenmax/files/7-Benchmark/) (7bench1200.7z)
I've got a bit of a noob question. I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life. Right now I'm using FLAC-8. I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
Yes, and the flac -8 point is the leftmost value on FLAC, though all are pretty close. While it doesn't translate 1-for-1 to battery life, decoding speed is a strong indicator (well correlated).
If I look at those graphs I see no point in using anything higher than FLAC -5. The extra compression you get is minimal and en-/decoding takes longer. Quite extreme case of diminishing returns I'd say.
I've got a bit of a noob question. I just RockBox'd a Sansa Clip+
admittedly old now (2010) but there have been benchmarks done already.
http://www.rockbox.org/wiki/CodecPerforman...ARM926EJ_45S_41 (http://www.rockbox.org/wiki/CodecPerformanceComparison#AMS_AS3525v2_w_47_40MHz_PClK_40ARM926EJ_45S_41)
flac_5.flac 2936.56% realtime Decode time - 5.99s 8.17MHz
flac_8.flac 2748.43% realtime Decode time - 6.40s 8.73MHz
you can always run tests yourself to see if there have been any improvements to the rockbox code in the intervening years.
If I look at those graphs I see no point in using anything higher than FLAC -5. The extra compression you get is minimal and en-/decoding takes longer. Quite extreme case of diminishing returns I'd say.
I think you’re actually talking about -4. In the upper graph, that’s the one with about 135 degrees between it and the next setting on the right, and as Dynamic said, the graph goes from -8 on the left to -0 on the right. The lower graph seems to have only 8 settings represented (rather than 9), although I presume that’s just a result of visual obstruction as some of them are very close together.
Yes -4 is the most clear point as you said. But -5 seems to be the point after which the graph levels out (asymptotic almost)*. So anything higher is virtually wasted effort.
*also see the document and individual cases.
I transcoded my FLACs from -8 to -5 and there was *very* little size increase. This will likely help out my battery life at a tiny size cost. Thanks for the input guys.
I transcoded my FLACs from -8 to -5 and there was *very* little size increase. This will likely help out my battery life at a tiny size cost. Thanks for the input guys.
saratoga can probably speak as to the specifics of that player, but on Porta Player devices the difference in decode speed would equal ZERO, not a little, but zero power savings as the CPU doesn't clock lower than 30Mhz. Whereas the size difference would equal (probably insignificant but) real power consumption differences due to increased storage access. Flash obviously cuts the power consumption of storage access down vs spinning rust, but the number is real.
I've got a bit of a noob question. I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life. Right now I'm using FLAC-8. I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
I've ran a test on my clip+ with an album encoded with flac -11 (CUETools's flaCL) and the same with wavpack -hhx and the difference was only about 10 minutes in runtime, so I'd say lossless encoding options don't affect battery life that much anymore.
I should have replied a lot earlier, because your publication immediately made me very happy.
Eventually a lossless codec comparison i feel i can trust!
In the past it was Synthetic Soul's comparison, although his sample selection was not sufficiently diverse (at least for my purposes).
When i am thinking about tuning opportunities for my codec, i often look at your results.
I think, you aren't getting as much feedback as you deserved here at hydrogen, but hey, your comparison has been published on the FLAC web site, and i am sure, the guys over there too know what constitutes a meaningful lossless codec comparison.
I must admit, i am selfish. I wanted to thank you but i also want to encourage you to continue your valuable work. Because it's so important to me, to have an independent instance, which verifies the outcome of my work.
Again
thank you!
Thomas
I should have replied a lot earlier, because your publication immediately made me very happy.
Eventually a lossless codec comparison i feel i can trust!
Well, that's nice to hear!
In the past it was Synthetic Soul's comparison, although his sample selection was not sufficiently diverse (at least for my purposes).
As the issue is brought up now, I would like to ask for a little help once again. I've been looking for more music to make the sample selection even more diverse, and I now have this list of 43 albums, where the previous comparison was with 29 albums.
- 30 Seconds to Mars - This Is War (rock)
- Abba - Gold: Greatest Hits (1992) (pop)
- Albert King - Years Gone By (blues)
- Alicia Keys - The Diary of Alicia Keys (R&B, soul)
- Alison Krauss Union Station - New Favorite (country, bluegrass)
- Armik - Rain Dance (flamenco/jazz)
- Café del Mar - Volumen nueve (chill-out)
- Coldplay - Parachutes (pop)
- Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
- Daft Punk - Alive 2007 (electronic)
- Dan Brown - Angels and Demons (audiobook)
- Enya - Amarantine (new age)
- Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
- Gilberto Santa Rosa - Esencia (salsa)
- Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (orchestral, romantic)
- Howard Shore - The Hobbit: An Unexpected Journey (orchestral, OST)
- Iron Maiden - Brave New World (metal)
- James Brown - Love, Power, Peace (Live at the Olympia, Paris, 1971) (funk)
- John Coltrane - Live at the half note, vol. 1 (jazz)
- Joss Stone - Mind, Body & Soul (soul)
- J.S. Bach - Magnificat (orchestral, baroque)
- Koninklijke Militaire Kapel - [no name] (military brass)
- Kraftwerk - Autobahn (electronic)
- Lana del Rey - Born to Die (pop)
- Lil Wayne - Tha Carter III (hip hop)
- Martin O'Donnell & Michael Salvatori - Halo: Reach (ambient-ish, OST)
- Metallica - Death Magnetic (metal)
- Michael Bublé - meets madison square garden (pop, jazz)
- Mike Oldfield - Tubular Bells (2000 remaster) (not classifiable)
- 'N Sync - Home for Christmas (pop)
- Nicholas Hooper - Harry Potter and the Order of the Phoenix (soundtrack)
- Peter Tosh - Equal Rights (2011 legacy edition) (reggae)
- Rosenberg Trio - Djangologists (jazz)
- Rush - Grace under Pressure (1997 remaster) (rock)
- System of a Down - Mezmerize (metal)
- Szakcsi - Virágom, virágom (world music)
- Tiësto - In Search of Sunrise 7: Asia (dance)
- The Lumineers - The Lumineers (folk)
- Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
- Waylon Jennings - Honky Tonk Heroes (country)
- Various - Latin Village, CD 1: Salsa (salsa)
- Xzibit - At the Speed of Life (hip hop)
- Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism)
The bold items are new. I think this is a nice list so far, any comments?
I must admit, i am selfish. I wanted to thank you but i also want to encourage you to continue your valuable work. Because it's so important to me, to have an independent instance, which verifies the outcome of my work.
But the comparison wasn't totally fair yet. I ran your codec in Wine, and while one could argue Wine doesn't add much overhead, it's a little weird I didn't run FLAC trough wine too, for example.
Anyway, the next version of this comparison will have all codecs tested on a Windows computer, as I finally figured out a utility that reported the CPU cycles used by a certain program instead of time elapsed. I rewrote my scripts to fit in a Windows enviroment, technically I'm ready to run it, but I am waiting for WavPack 4.70 final to be released (so I can test new versions of FLAC, TAK, Monkeys Audio and WavPack), so the result will be published somewhere this summer I suppose.
If there are still any remarks on the test setup, please tell me. I can still fix things now
edit: oh, and for the record, I will be testing on per-track basis instead of per-album basis now. Every album will be weighted equally just as the previous comparison, but now I will be able to identify outliers. Maybe something interesting pops up.
...
Anyway, the next version of this comparison will have all codecs tested on a Windows computer, as I finally figured out a utility that reported the CPU cycles used by a certain program instead of time elapsed.
...
What is the name of this utility?
What is the name of this utility?
It's timeit.exe, from the Windows Server 2003 Resource Kit Tools. However, it doesn't work with Windows 7 (I run Windows XP) but I found something that indicated powershell might have similar capabilities, but I didn't test. If anyone is going to dive into this, please report any finding, I'm not entirely sure this timeit utility reports are resembling the truth. It says process time, but the results I got back were sometimes weird for no apparent reason.
I've got a bit of a noob question. I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life. Right now I'm using FLAC-8. I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?
I've ran a test on my clip+ with an album encoded with flac -11 (CUETools's flaCL) and the same with wavpack -hhx and the difference was only about 10 minutes in runtime, so I'd say lossless encoding options don't affect battery life that much anymore.
To update on this, I ran some more tests and the differences are bigger. I found out flac -11 sometimes fails to decode on the clip+ (probably because it's not a subset file), so I switched to flac -8 and
battery wise, flac is a clear winner:
Flac -8: 16:09:49
Wv -hh -b384x: 11:17:12
mp3 v0: 11:48:20
vorbis q5: 12:17:56
opus 160: 10:17:00
Complete battery_bench logs here (https://www.dropbox.com/s/lnhzisi3tgy217d/Clip%2B%20battery%20bench.zip)
It's timeit.exe, from the Windows Server 2003 Resource Kit Tools. However, it doesn't work with Windows 7 (I run Windows XP) but I found something that indicated powershell might have similar capabilities, but I didn't test.
just yesterday i was wondering if there was a tool for benchmarking command line programs and i found this.
I use timer.exe from 7-Benchmark: http://sourceforge.net/projects/sevenmax/files/7-Benchmark/ (http://sourceforge.net/projects/sevenmax/files/7-Benchmark/)
it works on windows 7. i also tried the powershell command. you can see the results here - i did a quick test of flac and takc on a single album image.
https://dl.dropboxusercontent.com/u/2280132.../june/bench.png (https://dl.dropboxusercontent.com/u/22801321/2013/june/bench.png)
I hope this time your Windows computer can run flacCL. Would be nice to see it included.
Thanks ktf.
just yesterday i was wondering if there was a tool for benchmarking command line programs and i found this.
Yes, I saw that one too, but the problem is it reports running time, not CPU time used. I just took another look at powershell, it seems to report real time as well, not CPU time.
I hope this time your Windows computer can run flacCL. Would be nice to see it included.
I'm sorry, the test needs several weeks to run, so I can only use an old computer that's not longer used from before the GPGPU era to do this work.
However, I might as well add a section on different FLAC-encoders run on a different computer, but I will have to work that out.
Yes, I saw that one too, but the problem is it reports running time, not CPU time used.
I thought that "Process Time"
is CPU time. What is the difference between them?
Since common PCs do have more than one running core for several years now, there is a distinction between the time required for a task (elapsed time) and the time required for one working unit to complete it (cpu time).
In Windows, there's a gauge showing the usage from 0 to 100%, but in linux, the "top" utility can show usages of 150%, 200% or more. This is translated as using more than the time that a single working unit is able to provide.
I'm sorry, the test needs several weeks to run, so I can only use an old computer that's not longer used from before the GPGPU era to do this work.
What CPU does it have and what instructions does it support (SSE, SSE2, etc.)?
I thought that "Process Time" is CPU time. What is the difference between them?
You're right. I don't know why I concluded that 7bench is not able to measure CPU-time, because it clearly can. It does give the same results as timeit (and the same pecularities, like having a smallest increment of 0.015s at my system) so I suppose it uses the same kernel hooks.
What CPU does it have and what instructions does it support (SSE, SSE2, etc.)?
If you really want to know, the introduction date of this processor was March 10, 2005, I bought the laptop around the time the first dual-cores hit the mainstream market. Details are below. Wikipedia says: MMX, Enhanced 3DNow!, SSE, SSE2, SSE3, AMD64, PowerNow!, NX Bit
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 36
model name : AMD Turion 64 Mobile Technology ML-34
stepping : 2
cpu MHz : 1800.141
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
lm 3dnowext 3dnow rep_good pni lahf_lm
bogomips : 3600.28
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
As the issue is brought up now, I would like to ask for a little help once again. I've been looking for more music to make the sample selection even more diverse, and I now have this list of 43 albums, where the previous comparison was with 29 albums.
Usually more is better. As long as the codec relevant audio properties and there share of all audio files can't be fully determined, i don't know a better way to improve the representativity.
Do you know if your selection contains some hypercompressed files? Since they are so common know, this seems to be one more important category. I don't really like to ask for it, because i know that TAK performs less well on them...
I don't thinky you would need plenty such files. From my experience the effect of other codec relevant audio properties usually is quite small, if a file is hypercompressed.
But the comparison wasn't totally fair yet. I ran your codec in Wine, and while one could argue Wine doesn't add much overhead, it's a little weird I didn't run FLAC trough wine too, for example.
I wouldn't expect a significant effect. I have examined some comparisons which ran in Wine, and the results were in line with windows comparisons. I really like that you care so much about so many details, but i think it's nearly impossible to create a practicable comparison that everybody regards as fair. One could criticize, that you will be using a quite old cpu, because it lacks some instruction set extensions or because possibly newer compilers will not generate optimal code for it. On the other hand there are also good arguments against the use of a newer cpu. But someone will always complain.
I myself feel comfortable with your cpu choice. While it dosen't support SSSE3, what will make TAK's stronger presets somewhat slower, it's microarchitecture doesn't differ too much from more recent cpus.
Well, i probably would protest, if you were using one of those strange Pentium 4 - cpus...
edit: oh, and for the record, I will be testing on per-track basis instead of per-album basis now. Every album will be weighted equally just as the previous comparison, but now I will be able to identify outliers. Maybe something interesting pops up.
Great!
Do you know if your selection contains some hypercompressed files? Since they are so common know, this seems to be one more important category. I don't really like to ask for it, because i know that TAK performs less well on them...
I added Metallica's Death Magnetic just because it is probably the best known example of hyper-compressed stuff.
I myself feel comfortable with your cpu choice. While it dosen't support SSSE3, what will make TAK's stronger presets somewhat slower, it's microarchitecture doesn't differ too much from more recent cpus.
Yeah, another complaint might be that I run in 32-bit, while 64-bit is more or less standard these days. Sadly I don't have other machines available. It's either this with Windows XP or the setup as used in the previous comparison. The first results are very, very similar to my 64-bit linux environment (Intel Core2Duo T9600), the only codec that gets a significant performance hit is FLAC's decoding, which is about 20% slower.
I just finished a new revision of this codec comparison. It features new versions of FLAC, TAK, Monkey's Audio, WavPack and OptimFROG, runs on Windows instead of Linux and contrary to what I said earlier, the codecs were run on an AMD A4-3400. Furthermore, the test corpus has been expanded from 29 albums to 43 albums. WMA Lossless is now tested properly, refalac has been added and Real Lossless has been dropped. (edit: OS was Windows 7 SP1 64-bit)
No big changes in the results, except TAK being a whole lot faster.
The PDF and raw results are available here: http://www.icer.nl/losslesstest (http://www.icer.nl/losslesstest)
(http://www.icer.nl/misc_stuff/losslesscompv3-encoding.png)
(http://www.icer.nl/misc_stuff/losslesscompv3-decoding.png)
It's interesting to see that TAK starts where FLAC stops (decoding vs. compression), it's like they're subsets of the same codec.
Fantastic work, many thanks for your time and sharing it, keep it up.
Interesting to see the 4th node of TAK (from right to left) is faster than the 2nd and 3rd node while offering better compression. This renders these two, but especially the 3rd, rather useless IMO. Same counts for the 5th and 6th (7th is better and faster), and the 9th (10th is better).
From TAK's Readme:
Each preset selects a set of internal encoder options. Some of them affect only the encoding speed, the others also the decoding speed.
Slower decoding usually means higher hardware requirements (memory size and/or cpu power) for a playback device. Some devices may be too weak to support the more demanding internal encoder options. But any option, which only affects the encoding speed and complexity, is always applicable.
Therefore each preset consists of two components:
- The profile selects internal encoder options which affect both encoding and decoding speed. Profiles are declared as a number from 0 to 4 (fastest to strongest).
- The evaluation level selects only internal encoder options, which have no effect on the decoding speed. The evaluation levels are named Standard, Extra and Max and abbreviated as s, e and m (fastest to strongest).
A hardware manufacturer supporting TAK has only to specify the strongest profile it's hardware can decode, because the evaluation level does not affect the hardware requirements.
Hint: If you want higher compression and fast encoding, and are able to accept some decrease in decoding speed, it is usually preferable to select a higher profile instead of increasing the evaluation level.
The CPU used in this test might have been too fast compared to mobile devices to show, but I don't see any big variations in the decoding graph between the mentioned profiles.
Depending on the profile TAK is using up to 4, 12, 32, 80 or 160 predictors for it's primary filter. A corresponding number of multiplications per sample has to be performed. FLAC supports up to 32 predictors, but it's restricted subset which is recommended for maximum compatibility is limited to 12 (preset -8). And i remember reports on devices not capable to cope with 32 predictors.
I have no practical experience with the DSPs or cpus used in mobile devices, but from what i have read here at hydrogen and at Rockbox, higher predictor counts seem to be critical.
Indeed the decoding speed results show only a minor effect of the perdictor count, but that's misleading. Reasons:
- TAK will only use as many predictors as are advantegous. Because not many files benefit from high predictor orders, the average predictor count of the profile p4 might be for instance 70 (or even less) and not 160. But if you want to be able to decode any file without stuttering on a slow device, the device has to be powerful enough to execute the maximum of 160 multiplications in realtime.
- On a desktop system multiplications are really fast, especially if you are using SIMD instructions like MMX or SSEx. On a mobile device they can be a lot slower.
BTW: I think, TAK could be faster in this test. On the AMD A4-3400 it will only use MMX instructions although SSE2 is supported. Currently TAK will only use SSE2 is SSSE3 is supported too. My fault...
Thats a clear explanation. The lack of difference makes sense now. Thanks.