I have been working on speed enhancements for WavPack and think I have made enough progress to justify a release. The improvement seems to vary quite a bit on different systems, but I couldn't find a system where this didn't make some improvement (even a few percent) and on some systems (like my old 1.7 GHz P4) it is significant. The improvements apply to both packing and unpacking stereo files. Note that this is still pure C; there is no hand-coded assembler.
This also includes the MMX code (via intrinsics) from Joachim Henke. Unfortunately, this only seems to provide an improvement for bitdepths over 16, so it's restricted to that for now (although it does make a nice improvement for 24-bit or IEEE float data). Regarding MMX support, is that ubiquitous enough now that programs simply use it without checking for it? Since WavPack now requires Windows 98, it is reasonable to assume that any computer running Windows 98 also has MMX?
A bug has been fixed that sometimes caused a crash if a metadata source file specified on the command-line was not found, but otherwise this release only involves optimization. The output files should always be bit-identical to 4.40 regardless of the mode. This code hasn't been checked in to SVN and I haven't tested it on Linux, but that will come shortly.
Many thanks to the WavPack users and also thanks in advance for any comparison testing done on this new version, which can be found here (http://www.wavpack.com/wavpack-4.41.0-beta.zip).
David
Well, AFAIK, you can run Windows 98 on a 486DX without MMX, but I doubt anyone is doing that. Here's a list of CPUs that support MMX (German, but you should easily get the point):
http://de.wikipedia.org/wiki/MMX#CPUs_mit_MMX (http://de.wikipedia.org/wiki/MMX#CPUs_mit_MMX)
Thank you very much for this new beta release, David Your work is very much appreciated
I sometimes have some periods where i totally change my mind about something and this happened a little while ago, where i changed from WavPack to FLAC. Now their has gone about 14 days and now i'm beginning to come back to my old way of thinking again, and so i again changed my mind back and am now in the process of transcoding my FLAC images back to WavPack
Again, many thank's for your great work David and i humbly apologise for my last 14 days of "un-loyalty" to the great WavPack format, and i hope that you will forgive me from changing "sides" momentarily and to let me pass into the WavPack community again
CU, Martin.
Again, many thank's for your great work David and i humbly apologise for my last 14 days of "un-loyalty" to the great WavPack format, and i hope that you will forgive me from changing "sides" momentarily and to let me pass into the WavPack community again
Hi Martin!
I actually almost thanked all the
faithful WavPack users in my original post as a subtle reference to your very visible defection, but my good judgement prevailed. Glad you're back!
After what sometimes seems like daily announcements for FLAC support in various software and hardware, it doesn't surprise me that WavPack's popularity has slipped over time (at least with HA members). I have done everything I possibly can to make incorporating WavPack support in a product painless, but I suspect that if a vendor has to choose a single (or first) format to support then FLAC is the obvious choice simply because it already has the widest support. My hope is that some WavPack fan in the right position will steer a decision my way, or I will get a job with a company making audio devices and do the same (I interviewed at a well-known music server maker, but never got a call back... ).
In any event, WavPack can continue to provide a viable alternative, especially in applications where one of its features helps fill a niche (like the hybrid/lossy mode, float support, RIFF chunks, integer-based encoding, etc.), and can also provide Josh with incentive to continue pushing the envelope with FLAC.
That is, of course, until everyone switches to Tak!
Well, AFAIK, you can run Windows 98 on a 486DX without MMX, but I doubt anyone is doing that. Here's a list of CPUs that support MMX (German, but you should easily get the point):
http://de.wikipedia.org/wiki/MMX#CPUs_mit_MMX (http://de.wikipedia.org/wiki/MMX#CPUs_mit_MMX)
Thanks for the info. I guess I'll not worry about it for now except maybe a notice somewhere that audio resolutions greater than 16-bit require an MMX-capable CPU.
You sound like being a bit sad because of hardware support which is bigger for FLAC.
Hope you're not really dissappointed. You do such a great work.
And may be when lossless encoding becomes more widespread than now also top quality lossy encoding becomes more widespread cause it saves some 50% of file size or more compared to lossless while keeping extremely good quality.
As for lossy/hybrid mode wavPack has the best hardware suppport. When we'll have 16 or 32 GB flash players (expected to arrive soon) with rockbox support (hopefully) or maybe even natively (even better), wavPack lossy will be one of the most adequate fomats to many people.
Keep up the good work !
I'm not sure Martin. I'm not sure that I want to hang around with a "FLAC user". As long as you can promise that you have kicked the habit completely, and will not "use" again then perhaps we can begin your rehabilitation one step at a time...
Hey, who am I kidding: you're a very useful person to have onside. Welcome back brother!
I'm a bit gutted actually. I've been running tests all day, on various settings using both 4.40 and 4.41, and the testing had about 10-15 minutes to go and I had to shut down the PC so that we could put my son to bed. Wife's orders.
Looks like I will have to finish the decoding tomorrow (20:00pm here) and post then - around 15 hours' time. Shame, as I was really looking forward to seeing the figures
Anyways, thanks for the update David.
Just wanted to add my 2 cents. I've been using WavPack for quite some time now. I admit I still use FLAC, but WavPack is my favorite. I find it particularly good for my own music, since my masters are 32-bit audio. I'm not sure if FLAC even supports that bit depth. but 32/96 and 32/48 audio takes up some space, and WavPack compresses these files quite nicely (-hhxm). One thing I would like to see is support for WavPack in the Squeezebox, at least in the software. I wouldn't worry about hardware support too much, I really don't think that many people put lossless audio on portable players. Might be something to make a poll for. So keep up the development. Been doing a great job so far. Fact is WavPack was basically unknown a few years ago and now seems to be in the top 3.
Hi Martin!
I actually almost thanked all the faithful WavPack users in my original post as a subtle reference to your very visible defection, but my good judgement prevailed. Glad you're back!
Thank you very much David, i really appreciate it Btw, i also changed to using only GNU/Linux(Debian) instead of win32, besides changing to FLAC, but now i'm back for good with WavPack and on win32
Thank's again, my friend
CU, Martin.
I'm not sure Martin. I'm not sure that I want to hang around with a "FLAC user". As long as you can promise that you have kicked the habit completely, and will not "use" again then perhaps we can begin your rehabilitation one step at a time...
ROFLMAO You are hillarious, Synthetic Soul
Hey, who am I kidding: you're a very useful person to have onside. Welcome back brother!
Thank you very much for the kind words, my friend
CU, Martin.
sweet. changes are now in SVN . now if only Visual C++ didn't crash everytime i started compiling
Why not use the provided binary (http://www.wavpack.com/wavpack-4.41.0-beta.zip)?
right now, that's what i'm using(although i havent' encoded any songs yet). but for some reason, if i compile my own versions, then i get a faster binary which is faster by 10 seconds. it's not much, but i guess it's worth it.
i personally want to use MinGW to compile it and compare differences between VC++ and GCC since VC++ usually makes bloated exes.
I'm a bit gutted actually. I've been running tests all day, on various settings using both 4.40 and 4.41, and the testing had about 10-15 minutes to go and I had to shut down the PC so that we could put my son to bed. Wife's orders.
Looks like I will have to finish the decoding tomorrow (20:00pm here) and post then - around 15 hours' time. Shame, as I was really looking forward to seeing the figures
I too am interested to see what happens on other systems. This optimization business can be kind of a circus with all the different CPUs and compilers and their switches. Sometimes I'd come up with an improvement and it would be 20% faster on my primary system, then I'd try it at work and it would be much slower! What I finally came up with was faster on all three systems I tried, so my fingers are crossed...
I just built it on gcc for Linux and most of the improvements seem to work there as well, especially when I tell gcc to optimize for my exact CPU (a feature which Microsoft seems to have decided was more trouble than it was worth and deleted from VC8).
Anyway, Neil, thanks for giving it a go and thanks to everyone for the kind words and support!
Now
my wife is trying to put
me to bed, so I'm going to have to wait also...
David
OK, finally, here they are:
| | 4.40.0 | 4.41
Setting | Comp % | Enc Dec | Enc Dec
=========|===========|==============|============
f | 66.741% | 68x 92x | 73x 97x
x | 66.539% | 38x | 40x
x2 | 66.445% | 28x | 29x
x3 | 66.386% | 16x | 17x
default | 65.582% | 59x 80x | 64x 82x
x | 65.370% | 29x | 30x
x2 | 65.312% | 18x | 20x
x3 | 65.285% | 10x | 10x
h | 64.877% | 40x 65x | 43x 67x
x | 64.764% | 20x | 21x
x2 | 64.679% | 12x | 12x
x3 | 64.679% | 6x | 6x
hh | 64.487% | 32x 53x | 34x 54x
x | 64.422% | 14x | 16x
x2 | 64.394% | 8x | 9x
x3 | 64.378% | 4x | 4x
There is definate improvement, most noticeable when encoding when the x switches are not used, and more prominent with the faster settings.
All 16 bit files I'm afraid, so no test for the MMX code.
NB: Given that 144 people so far have stated that WavPack is their codec of choice in the 2007 poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=51493), it would be nice if more than three people provided some data.
C'mon kiddies: time to show the nice man your appreciation...
Hi Bryant,
A quick note to add my thanks for your efforts.
I've been using it for a couple of years now and, coming from an audio (rather than software) background, WavPack has been one of the things that made me a convert to PC-based audio.
Best
Midas
You are right Synthetic Soul.
I was using FLAC/REACT to backup my collection, then I had a HD crash (after about 100 CDs).
Anyhow, after I got the computer back and running, I seriously looked at WV versus FLAC. I made the switch to wavpack images to backup my collection. I love it.
I also tried 3 CD rips using REACT/wavpack beta, and it works great. I don't have the numbers to back up my claim, but the speed increase is great.
Thank you for such a great program.
And if there is any concern, it works great on Vista.
There is definate improvement, most noticeable when encoding when the x switches are not used, and more prominent with the faster settings.
Thanks for the test! It's interesting that the improvement was so modest, but it's comforting to see that none of the settings seemed to get slower.
On my older system the decoding improvement was about 15% across all modes and the encoding improvement was as much as 30% in the higher modes. Hopefully some other results will show more significance; I can't imagine this alone being worth its own release (although it's obviously worth keeping).
NB: Given that 144 people so far have stated that WavPack is their codec of choice in the 2007 poll (http://www.hydrogenaudio.org/forums/index.php?showtopic=51493), it would be nice if more than three people provided some data.
C'mon kiddies: time to show the nice man your appreciation...
Easier said than done.
Ok, I fetched your
comparison-scripts-v1.2 (http://www.synthetic-soul.co.uk/files/comparison-scripts-v1.2.zip) (hope it's the latest version) pack and modified it for comparing 4.40 against 4.41...
that's two times 16 folders. What's the maximum disc requirement I can expect to be used? I hope the test en-/decodes will be deleted right after the processing so I won't end up with 8-16 times the size of the source folder, right?
Does it calculate the average times or do I have to do that myself (I think I've seen the word "spreadsheet" mentioned in the lossless comparison thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=50954)... I guess I could get that to work with
OpenOffice Calc).
Another question: Can I skip the md5 part? I don't think it's necessary and will only make the test longer (and more likely that I will cancel it).
Currently fb2k is still decoding the originals.
Hi David,
In any event, WavPack can continue to provide a viable alternative, especially in applications where one of its features helps fill a niche (like the hybrid/lossy mode, float support, RIFF chunks, integer-based encoding, etc.), and can also provide Josh with incentive to continue pushing the envelope with FLAC.
That is, of course, until everyone switches to Tak!
I will be happy, if TAK will find it's niche, at least big enough to justify all the work put into it's release... But thank you very much for the encouragement!
BTW: For me WavPack is a great codec! Especially for it's simplicity. Please don't take me wrong, i want to say in my bad english: simple + efficient = ingenious! With simple i mean the code complexity, not your great altgorithm. I would be glad, if TAK was equally compact, but no chance.
I also like your simple to use code interfaces. I hope you don't mind, that i used them as a model for my new SDK...
I want to say thank you for your great work and i really hope, you will continue it. But please don't improve it too soon too much, please give TAK some time... Honestly i am always most curious if a new Wavpack version is beeing released, because regarding efficiency, WavPack is TAK's real (friendly) competitor.
Let's work on further improvements and have some more fun!
Thomas
Thanks for the test! It's interesting that the improvement was so modest, but it's comforting to see that none of the settings seemed to get slower.
I seem to remember that my corpus didn't fair so well with your improvements last time. I was kinda hoping someone else would post some results before mine to balance it out a bit! Anyway, considering that they are pure C amendments, and as you say none have worsened, it's most certainly better than a poke in the eye with a sharp stick. To try to make it sound slightly better: with the default encoding I shaved 15 seconds off a 202 second encode phase. I don't think that's all that bad.
Easier said than done.
True. I'm glad that you have made an effort though. David deserves some help from us.
Ok, I fetched your comparison-scripts-v1.2 (hope it's the latest version) pack and modified it for comparing 4.40 against 4.41...
that's two times 16 folders. What's the maximum disc requirement I can expect to be used? I hope the test en-/decodes will be deleted right after the processing so I won't end up with 8-16 times the size of the source folder, right?
MB used depends on the size of your corpus. If you run encodeall.bat and then decodeall.bat you will need room for the source, all encodes, and one decode set (1*wav + n*wv + 1*wav) before you start removing files with the decode scripts. If this is a concern write a short batch file to call a folder's encode.bat and decode.bat in turn - then you only need room for your corpus and one encode/decode set (1*wav + 1*wv + 1*wav).
Does it calculate the average times or do I have to do that myself (I think I've seen the word "spreadsheet" mentioned in the lossless comparison thread... I guess I could get that to work with OpenOffice Calc).
You have to collate the figures yourself. I used Excel for the report above. It's a simple/boring copy'n'paste job...
Another question: Can I skip the md5 part? I don't think it's necessary and will only make the test longer (and more likely that I will cancel it).
Remove:
@CALL "%~dp0md5.bat"
... from all decode.bat files.
BTW: All 1600 encodes (50 files encoded using 16 settings for 2 versions) matched for me.
NB: I have uploaded my spreadsheet and all script files (including the resulting txt files) here (http://www.synthetic-soul.co.uk/temp/wavpack-4.41.0b1-testing.zip).
Easier said than done.
Well, obviously I appreciate the incredible amount of work that Synthetic Soul has put into his tests and precedures. However, it is also possible to provide useful information on different systems with virtually no work. For example, here are results from my system using just a single track. I always do a couple test runs (not shown) to make sure the sources are in disk cache and I either write the output to "nul" for encoding or just use verify mode for decoding to make sure that no disk I/O is entering the timing. I get results like this:
Decoding test (sorry, you have to scroll a little to see the differences):
C:\devel\opt\trunk>wvunpack -v Track01*
WVUNPACK Hybrid Lossless Audio Decompressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
Track01hh.wv:
verified Track01hh.wv in 8.23 secs (lossless, 39.69%)
Track01f.wv:
verified Track01f.wv in 3.88 secs (lossless, 37.35%)
Track01.wv:
verified Track01.wv in 4.86 secs (lossless, 38.31%)
Track01ff.wv:
verified Track01ff.wv in 3.82 secs (lossless, 36.78%)
Track01hx.wv:
verified Track01hx.wv in 6.38 secs (lossless, 39.46%)
**** 5 files successfully processed ****
C:\devel\opt\trunk>release\wvunpack -v Track01*
WVUNPACK Hybrid Lossless Audio Decompressor Win32 Version 4.41.0-beta
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
Track01hh.wv:
verified Track01hh.wv in 7.35 secs (lossless, 39.69%)
Track01f.wv:
verified Track01f.wv in 3.34 secs (lossless, 37.35%)
Track01.wv:
verified Track01.wv in 4.14 secs (lossless, 38.31%)
Track01ff.wv:
verified Track01ff.wv in 3.33 secs (lossless, 36.78%)
Track01hx.wv:
verified Track01hx.wv in 5.66 secs (lossless, 39.46%)
**** 5 files successfully processed ****
Encoding test:
C:\devel\opt\trunk>wavpack -fy Track01 nul
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created nul.wv in 4.98 secs (lossless, 37.35%)
C:\devel\opt\trunk>release\wavpack -fy Track01 nul
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created nul.wv in 4.37 secs (lossless, 37.35%)
C:\devel\opt\trunk>wavpack -hhy Track01 nul
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created nul.wv in 13.43 secs (lossless, 39.55%)
C:\devel\opt\trunk>release\wavpack -hhy Track01 nul
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created nul.wv in 10.19 secs (lossless, 39.55%)
C:\devel\opt\trunk>wavpack -hhxy Track01 nul
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created nul.wv in 22.36 secs (lossless, 39.64%)
C:\devel\opt\trunk>release\wavpack -hhxy Track01 nul
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created nul.wv in 18.27 secs (lossless, 39.64%)
Finally, when I want to verify that I haven't broken anything I just pipe the output to md5:
C:\devel\opt\trunk>release\wavpack -hhy Track01 -q - | md5 -
3EA3537450C0189192174CAFAA6CA506 -
C:\devel\opt\trunk>wavpack -hhy Track01 -q - | md5 -
3EA3537450C0189192174CAFAA6CA506 -
Hopefully someone will get results somewhat like mine, otherwise I spent all that time optimizing for a single computer!
Thanks for the test! It's interesting that the improvement was so modest, but it's comforting to see that none of the settings seemed to get slower.
I seem to remember that my corpus didn't fair so well with your improvements last time. I was kinda hoping someone else would post some results before mine to balance it out a bit! Anyway, considering that they are pure C amendments, and as you say none have worsened, it's most certainly better than a poke in the eye with a sharp stick. To try to make it sound slightly better: with the default encoding I shaved 15 seconds off a 202 second encode phase. I don't think that's all that bad.
Yeah, the fast and normal modes show a nice improvement. I also saw less improvement on the 2 newer systems that I tried (although still more than you) so I suspect that somehow the newer CPU is simply able to more efficiently handle the things I am optimizing out.
BTW, how do these results compare to other results that you have for other codecs? The numbers don't seem to jive with the public version of your corpus that's here (http://synthetic-soul.co.uk/comparison/lossless/) so I suspect the procedure (or system) is different now.
BTW: For me WavPack is a great codec! Especially for it's simplicity. Please don't take me wrong, i want to say in my bad english: simple + efficient = ingenious! With simple i mean the code complexity, not your great altgorithm. I would be glad, if TAK was equally compact, but no chance.
Thanks! It's interesting that one of my goals for WavPack 4.0 was to make it as simple as possible. So, I worked hard on figuring out decorrelation and entropy encoding methods that would work well with both lossless and hybrid modes and work well with both fast and slow modes, and was very happy with the final result.
Of course, now, when I want to speed things up, I realize that some of the compromises I made for simplicity now are costing in efficiency. I could create a new mode specifically for lossless that would achieve the same compression at greater speed (and be more MMX'able) but that's what I was trying to get away from.
I also like your simple to use code interfaces. I hope you don't mind, that i used them as a model for my new SDK...
Of course I don't mind; I'm glad you found something useful there!
I want to say thank you for your great work and i really hope, you will continue it. But please don't improve it too soon too much, please give TAK some time... Honestly i am always most curious if a new Wavpack version is beeing released, because regarding efficiency, WavPack is TAK's real (friendly) competitor.
Don't worry, I don't think there's anything coming soon. I think the only way to make a significant improvement in WavPack's speed would be to start rewriting several large functions is pure assembler. Now, I would really like to do that, but unfortunately I just can't justify the time required.
BTW, I haven't been following all the TAK discussion closely enough to know whether you are using any hand-coded assmebler in there. I remember at one point you said that it was pure Delphi, but I was wondering if that was still the case (I know you must at least be using some intrinsics for MMX). And if not, do you think you'll be able to get further speed gains if you ever tried that?
Let's work on further improvements and have some more fun!
Absolutely!
BTW, if you'd like to send me your code I would be happy to look it over and let you know what I think. I bet Josh would even help me and I'm sure we could come up with some good ideas for you!
Very brief test on a P3-550MMX:
normal mode +6 % improvement
high + 8.7 %
v-high: + 10 %
fast: + 7 %
normal x1: + 15 %
normal x3: + 18 %
v-high x3: + 20 %
Good work David !
Very brief test on a P3-550MMX:
normal mode +6 % improvement
high + 8.7 %
v-high: + 10 %
fast: + 7 %
normal x1: + 15 %
normal x3: + 18 %
v-high x3: + 20 %
Good work David !
Thanks for letting me know. I'm glad that someone's getting results along these lines!
BTW, how do these results compare to other results that you have for other codecs? The numbers don't seem to jive with the public version of your corpus that's here (http://synthetic-soul.co.uk/comparison/lossless/) so I suspect the procedure (or system) is different now.
The results posted here are for CPU-only (my preferred format), while the comparison on my site reports CPU+IO (that's how it started, so for now, unfortunately, it must continue that way).
All the results on my site were produced with 512MB of RAM in the machine. I recently got given 1GB so I could upgrade. I am concerned about mixing results from both set-ups. I'm a bit dim when it comes to such things: would the additional RAM change results?
I did some testing for FLAC 1.1.4 recently. I re-did the 1.1.3 tests and compared the 512MB test and the 1GB test and the figures appeared to be very similar. My intention was to compare the 512MB and 1GB 4.40 results also.
Anyway, for now there are my FLAC 1.1.4 results (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=52430&view=findpost&p=469607), but I'll try to make a correlation with the others if I can.
Anyway, for now there are my FLAC 1.1.4 results (http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=52430&view=findpost&p=469607), but I'll try to make a correlation with the others if I can.
Ah, this is perfect! I just wanted to compare the results to
something else, and that will do fine.
Assuming you aren't running anything else on your PC during the tests I wouldn't believe that adding memory would make a difference (assuming it's the same speed memory). Certainly none of these programs are going to approach using even 100 megs, right? It would mean that there would be more disk cache available, but I wouldn't think that would make a difference either because you're not processing the same files over and over again. But I'm certainly no expert...
I checked the CPU-only results of both 4.40 runs (512MB and 1GB RAM) and they were suitably similar.
However, after adding the results for WavPack 4.41 and FLAC 1.1.4 to my CPU+IO comparison table I have seen that the 4.40 times are better than the 4.41!! Given the the CPU-only times reported I can only assume that this is down to my system - perhaps down to the disk cache that you mentioned? Anyway, in conclusion, the new CPU+IO results are worthless, until I can get the whole table up to tests run on my 1GB system.
I will try to run some tests on my laptop (AMD Turion 64 Mobile ML-36) to get you some more data.
However, after adding the results for WavPack 4.41 and FLAC 1.1.4 to my CPU+IO comparison table I have seen that the 4.40 times are better than the 4.41!! Given the the CPU-only times reported I can only assume that this is down to my system - perhaps down to the disk cache that you mentioned? Anyway, in conclusion, the new CPU+IO results are worthless, until I can get the whole table up to tests run on my 1GB system.
It occurred to me that it might be worth looking into the following items in order to make the CPU-only tests more precise and/or the CPU+IO tests more repeatable:
1. A high performance ramdisk driver for windows.
2. A high performance /dev/null equivalent for windows, if such a thing exists.
3. A means of preloading the system cache with your input file.
4. A means of ensuring the system cache does not have your input file preloaded at all.
Just throwing out some ideas...
-brendan
With a Pentium 4 2.6GHz speed increased like this:
switch encode decode
-hhx3 19,98% 12,18%
-hh 33,05% 14,72%
-hx3 18,44% 13,60%
-h 24,60% 16,14%
-x3 14,35% 15,76%
21,99% 16,63%
-fx3 8,48% 10,34%
-f 9,46% 18,43%
(based on process time according to timer.exe)
Wow, the improvements reported by shadowking and gaekwad2 are far more impressive! Superb.
Also, curious that mine was more apparent in the faster modes but theirs in the slower. Both their machines are Pentiums, while mine is an Athlon XP +2400. Unfortunately my laptop is an Athlon also. Had you seen results from an Athlon previously? Is it possible that compiler switches are somehow geared toward Intel chips?
Edit: I know it's a little early to be making conclusions, but hey.. I'm bored!
Edit: Bit late, but had to amend that hideous "there's"/"theirs" error...
Sorry if this question is stupid, but I didn't follow discussion about testing the speed - is there a special test suite available for download somewhare or do I have to feed the encoders with my own files?
Please forgive me for first testing the new beta now, and because of little time, then i have only tested one image consisting of 11 tracks(Evanescence - Fallen) and i have only used the -h switch.
For the test, then i used an Intel Celeron 1.7GHz with 256MB RAM(the Celeron 1.7GHz is identical to a P4 Willamete(first P4 generation), except with only a half L2 cache size).
(wavpack = v4.40)
timer wavpack -h test.wav > log.txt
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
Kernel Time = 6.699 = 00:00:06.699 = 5%
User Time = 101.525 = 00:01:41.525 = 83%
Process Time = 108.225 = 00:01:48.225 = 89%
Global Time = 121.204 = 00:02:01.204 = 100%
timer wavpack441b -h test.wav >>log.txt
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
Kernel Time = 6.839 = 00:00:06.839 = 5%
User Time = 91.351 = 00:01:31.351 = 75%
Process Time = 98.191 = 00:01:38.191 = 81%
Global Time = 120.473 = 00:02:00.473 = 100%
timer wvunpack test.wv >>log.txt
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
Kernel Time = 6.649 = 00:00:06.649 = 7%
User Time = 65.624 = 00:01:05.624 = 74%
Process Time = 72.273 = 00:01:12.273 = 81%
Global Time = 88.587 = 00:01:28.587 = 100%
timer wvunpack441b test.wv >>log.txt
Timer 3.01 Copyright © 2002-2003 Igor Pavlov 2003-07-10
Kernel Time = 6.919 = 00:00:06.919 = 8%
User Time = 55.289 = 00:00:55.289 = 70%
Process Time = 62.209 = 00:01:02.209 = 79%
Global Time = 78.383 = 00:01:18.383 = 100%
As always, very nice job, David
CU, Martin.
Had you seen results from an Athlon previously? Is it possible that compiler switches are somehow geared toward Intel chips?
I've found in the past WavPack had some strange behaviour from this point of view. I used to repeat my test on a PIV, a PIII and a Sempron. When compared to other encoders, WavPack usually showed bad behaviour on the PIII. I think I still should have some tables around...
Anyway, here's some results on an Athlon64 3500+:
4.40.0 4.41.0b Enhancement
f 121,9x 152,3x 132,1x 163,3x 8,4% 7,2%
fx1 70,0x 152,3x 77,0x 160,3x 10,0% 5,3%
fx2 50,5x 150,6x 57,5x 159,8x 13,9% 6,1%
fx3 30,7x 149,7x 35,7x 159,3x 16,3% 6,4%
fx4 13,0x 149,7x 15,1x 158,8x 16,2% 6,1%
fx5 10,5x 147,9x 12,2x 158,3x 16,2% 7,0%
fx6 9,0x 146,7x 10,4x 156,9x 15,6% 7,0%
100,8x 131,1x 111,0x 135,6x 10,1% 3,4%
x1 51,8x 129,5x 60,7x 134,9x 17,2% 4,2%
x2 33,8x 129,2x 41,1x 131,5x 21,6% 1,8%
x3 18,3x 127,9x 23,0x 131,1x 25,7% 2,5%
x4 4,8x 127,6x 5,8x 130,8x 20,8% 2,5%
x5 3,4x 126,7x 4,0x 130,5x 17,6% 3,0%
x6 1,7x 124,8x 2,0x 129,5x 17,6% 3,8%
h 74,5x 100,8x 83,4x 105,5x 11,9% 4,7%
hx1 36,5x 100,4x 43,8x 104,3x 20,0% 3,9%
hx2 21,7x 100,1x 27,0x 102,4x 24,4% 2,3%
hx3 10,9x 100,1x 13,8x 101,4x 26,6% 1,3%
hx4 2,9x 99,7x 3,5x 101,0x 20,7% 1,3%
hx5 2,2x 99,3x 2,7x 100,8x 22,7% 1,5%
hx6 1,5x 98,7x 1,8x 100,1x 20,0% 1,4%
hh 60,5x 80,6x 67,7x 81,8x 11,9% 1,5%
hhx1 26,8x 80,3x 33,5x 81,6x 25,0% 1,6%
hhx2 15,3x 79,9x 19,5x 81,5x 27,5% 2,0%
hhx3 7,4x 79,6x 9,6x 81,3x 29,7% 2,1%
While on the decoding side the improvements are not as dramatic as they are on the tests on the Intel chips above, nevertheless the encoding speed improvements seems very good also on this AMD chip test.
My diminutive test says improvement is tangible !
AMD Sempron(tm) Processor 3400+ (2GHz, Palermo core, 90 nm, socket 754)
C:\WINDOWS\Temp>wavpack -hx3m "02 - Epic.wav"
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
original md5 signature: 4321b3f5d474133547757fab9c7abd34
created 02 - Epic.wv in 31.02 secs (lossless, 32.13%)
C:\WINDOWS\Temp>wvunpack -m "02 - Epic.wv"
WVUNPACK Hybrid Lossless Audio Decompressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
original md5: 4321b3f5d474133547757fab9c7abd34
unpacked md5: 4321b3f5d474133547757fab9c7abd34
restored 02 - Epic.wav in 3.80 secs (lossless, 32.13%)
02 - Epic.wav 51.849.884 bytes
02 - Epic.wv 35.192.996 bytes
C:\WINDOWS\Temp>wavpack -hx3m "02 - Epic.wav"
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
original md5 signature: 4321b3f5d474133547757fab9c7abd34
created 02 - Epic.wv in 24.66 secs (lossless, 32.13%)
C:\WINDOWS\Temp>wvunpack -m "02 - Epic.wv"
WVUNPACK Hybrid Lossless Audio Decompressor Win32 Version 4.41.0-beta
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
original md5: 4321b3f5d474133547757fab9c7abd34
unpacked md5: 4321b3f5d474133547757fab9c7abd34
restored 02 - Epic.wav in 3.75 secs (lossless, 32.13%)
02 - Epic.wav 51.849.884 bytes
02 - Epic.wv 35.192.996 bytes
Of course, now, when I want to speed things up, I realize that some of the compromises I made for simplicity now are costing in efficiency. I could create a new mode specifically for lossless that would achieve the same compression at greater speed (and be more MMX'able) but that's what I was trying to get away from.
That's also my experience: Highly optimized code and a good neat design don't go together...
BTW, I haven't been following all the TAK discussion closely enough to know whether you are using any hand-coded assmebler in there. I remember at one point you said that it was pure Delphi, but I was wondering if that was still the case (I know you must at least be using some intrinsics for MMX). And if not, do you think you'll be able to get further speed gains if you ever tried that?
TAK contains highly optimized MMX-assembler code. I wrote it in 1997 when i got my first Pentium 200 MMX. Without the speedup achieved from the MMX-code, my tests and evaluations would have lasted intolerably long...
But there is also a switch to disable the MMX-optimizations. Presets Turbo and Fast are performing quite well without MMX (about 40 percent less encoding speed on my P3), but the stronger presets with higher predictor orders are getting much slower without MMX.
Unfortunately the MMX-usage also makes some transformations of the data neccessary, which probably are eating up the advantage for at least the turbo preset.
And for cpu dependency: My MMX-code is optimized for my P3 and should also work very well on the newer Intel cpus, which are quite similar to the P3. But the P4 is a strange beast (i never liked it...): My code is using many register moves and they are incredibly slow on a P4 (can be different for the latest revisions of the P4). A latency of 6 clock cycles for a simple Movq! A Pmaddwd doesn't take longer...
BTW, if you'd like to send me your code I would be happy to look it over and let you know what I think. I bet Josh would even help me and I'm sure we could come up with some good ideas for you!
Thank you!
Thomas
Also, curious that mine was more apparent in the faster modes but there's in the slower. Both their machines are Pentiums, while mine is an Athlon XP +2400. Unfortunately my laptop is an Athlon also. Had you seen results from an Athlon previously? Is it possible that compiler switches are somehow geared toward Intel chips?
It turns out that the faster modes vs. slower modes difference makes some sense. There are two components to WavPack (and other lossless audio for that matter) compression. The first is decorrelation where the redundancy in the audio samples is [almost] removed so you only have [ideally] noise of varying amplitude. Then the noise (sometimes called the residuals) is sent to the entropy encoder that turns it into a bitstream by [essentially] removing all the unneeded bits (because these numbers tend to be quite small).
The higher modes of WavPack are created by having the decorrelator make more passes over the data (2 for fast mode, 16 for very high mode). The entropy encoder (and some other stuff like JS and CRC) is always exactly the same no matter what the mode. So, in fast mode the entropy encoder (and other constant stuff) is most of the time, while in the higher modes more and more of the total time is in the decorrelator.
In this round of optimization I made improvements in both sections. But, if in your system for some reason only the constant stuff saw the improvement then the fast modes would see the big improvement, whereas in other systems the decorrelation improvements would help the higher modes more. Not a mystery at all!
My wife's laptop is also a Turion 64 and I saw improvement there also. Not as much as my P4, but more than your AMD. In any event, I am beginning to think the improvements are certainly worth it based on the other results. Yours are of course still greatly appreciated...
Sorry if this question is stupid, but I didn't follow discussion about testing the speed - is there a special test suite available for download somewhare or do I have to feed the encoders with my own files?
With some encoders it might make a difference, but WavPack should pretty much always run the same regardless of the source material. Use whatever you need to encode anyway...
TAK contains highly optimized MMX-assembler code. I wrote it in 1997 when i got my first Pentium 200 MMX. Without the speedup achieved from the MMX-code, my tests and evaluations would have lasted intolerably long...
But there is also a switch to disable the MMX-optimizations. Presets Turbo and Fast are performing quite well without MMX (about 40 percent less encoding speed on my P3), but the stronger presets with higher predictor orders are getting much slower without MMX.
Unfortunately the MMX-usage also makes some transformations of the data neccessary, which probably are eating up the advantage for at least the turbo preset.
I see. I imagine that the faster modes benefit less from MMX because more time is spent in the entropy coder (which I assume is not easily MMX-able, although I haven't thought about it too much).
Anyway, however you're doing it is certainly working incredibly well!
And for cpu dependency: My MMX-code is optimized for my P3 and should also work very well on the newer Intel cpus, which are quite similar to the P3. But the P4 is a strange beast (i never liked it...): My code is using many register moves and they are incredibly slow on a P4 (can be different for the latest revisions of the P4). A latency of 6 clock cycles for a simple Movq! A Pmaddwd doesn't take longer...
Yeah, I've heard other places that the P4 is an admitted Intel blunder (especially the early ones). Maybe it's not the best for me to be optimizing for! I also notice that mine seems to slow down significantly after just a minute or so of running WavPack, which makes benchmarking frustrating! Maybe I should make sure the fan is clean...
BTW, if you'd like to send me your code I would be happy to look it over and let you know what I think. I bet Josh would even help me and I'm sure we could come up with some good ideas for you!
Thank you!
I hope you know that this was my poor attempt at a joke...
I too am interested to see what happens on other systems. This optimization business can be kind of a circus with all the different CPUs and compilers and their switches. Sometimes I'd come up with an improvement and it would be 20% faster on my primary system, then I'd try it at work and it would be much slower! What I finally came up with was faster on all three systems I tried, so my fingers are crossed...
In next rough and absolutely not ultimate table, you can see a few random encoders performances on a PIV2.8HT, a Tualatin1.56ghz and a Sempron1.83ghz.
In the last columns you can see PIV to PIII performance comparison and PIV to Sempron performance comparison.
On the average row you can see the PIV being 41% (encoding) and 29% (decoding) faster than the PIII (on an average base) and 6% faster (encoding) and 1% slower (decoding) than the Sempron (on an average base).
PIV Speed PIII Speed Sempron Speed PIV to PIII PIV to Sempron
OptimFrog Normal 17,8x 22,9x 11,2x 15,8x 15,5x 22,2x 59,0% 45,5% 14,7% 3,3%
Flac 1.1.2 -8 9,5x 99,1x 7,5x 77,1x 9,5x 115,7x 26,5% 28,6% 0,7% -14,3%
Flac 1.1.2 -5 49,6x 99,1x 29,2x 81,6x 46,3x 115,7x 69,6% 21,4% 7,1% -14,3%
Monkey High 35,6x 31,5x 27,5x 25,7x 38,0x 34,7x 29,5% 22,7% -6,4% -9,1%
Monkey Normal 40,2x 33,0x 32,7x 30,2x 42,7x 38,6x 23,2% 9,5% -5,8% -14,3%
MP4ALSGarf -a-o32 38,6x 59,1x 24,8x 44,1x 34,3x 57,8x 55,6% 34,0% 12,5% 2,1%
MP4ALSGarf -a-o4 89,5x 111,0x 61,7x 86,8x 75,0x 95,7x 45,2% 28,0% 19,4% 16,0%
Yalac0.05 Normal 23,5x 95,7x 16,9x 71,2x 21,9x 86,8x 39,0% 34,5% 7,6% 10,3%
Yalac0.05 High 9,2x 99,1x 7,6x 71,2x 9,1x 86,8x 19,8% 39,3% 1,0% 14,3%
Average 40,8% 29,3% 5,6% -0,7%
Now, in next table you can see the same numbers on a few WavPack tests. As you can see the PIV is not less than 147% faster (encoding) and 50% faster (decoding) than the PIII (WavPack based), and 42% faster (encoding) and 14% faster (decoding) than the Sempron (WavPack based).
Again, those are not enough numbers to be considered ultimate, I only mean to show a possible behaviour.
When confirmed, this should mean that, in a comparison with other codecs, WavPack likes the PIV better than the Sempron and it likes the PIII much less than the other two chips. The PIV is encoding -hx3 over three times faster than the PIII and over 50% faster than the Sempron!
PIV Speed PIII Speed Sempron Speed PIV to PIII PIV to Sempron
WavPack 4.3 -hx3 1,9x 60,3x 0,6x 42,1x 1,2x 52,4x 191,7% 43,5% 51,3% 15,2%
WavPack 4.3 Default 81,6x 99,1x 43,4x 63,1x 64,6x 89,5x 88,2% 57,1% 26,5% 10,7%
WavPack 4.3 -fx4 15,0x 115,7x 5,7x 77,1x 10,2x 99,1x 161,1% 50,0% 47,6% 16,7%
Average 147,0% 50,2% 41,8% 14,2%
Finally, here's the results for my laptop (AMD Turion 64 Mobile ML-36):
| | 4.40.0 | 4.41 | Benefit
Setting | Comp % | Enc Dec | Enc Dec | Enc Dec
======================================================================
f | 61.020% | 101x 119x | 115x 128x | 114% 107%
x | 60.485% | 68x | 79x | 115%
x2 | 60.369% | 50x | 59x | 118%
x3 | 60.303% | 30x | 35x | 119%
Default | 59.715% | 89x 104x | 100x 108x | 113% 104%
x | 59.470% | 50x | 61x | 120%
x2 | 59.398% | 33x | 41x | 124%
x3 | 59.369% | 17x | 22x | 127%
h | 59.058% | 66x 83x | 75x 85x | 114% 102%
x | 58.918% | 34x | 43x | 125%
x2 | 58.841% | 20x | 26x | 129%
x3 | 58.841% | 10x | 13x | 132%
hh | 58.760% | 54x 68x | 62x 69x | 115% 102%
x | 58.674% | 25x | 33x | 129%
x2 | 58.639% | 14x | 19x | 132%
x3 | 58.627% | 7x | 9x | 135%
NB: Different corpus to other test - no comparitive results (these are the first tests that I have run on my laptop).
Pleasing figures though.
BTW, if you'd like to send me your code I would be happy to look it over and let you know what I think. I bet Josh would even help me and I'm sure we could come up with some good ideas for you!
Thank you!
I hope you know that this was my poor attempt at a joke...
Huch... ...... Ahh... .... I've got it!
When confirmed, this should mean that, in a comparison with other codecs, WavPack likes the PIV better than the Sempron and it likes the PIII much less than the other two chips. The PIV is encoding -hx3 over three times faster than the PIII and over 50% faster than the Sempron!
This is very interesting, thanks for presenting it!
I guess there are just two possibilities. The first is that because I have done my optimization on a P4 (although a much older one than yours) the decisions have all been skewed towards that processor. The second is that something about the WavPack algorithm itself is more suited for a P4, like maybe the multiple passes over the samples. I don't know enough about the CPU differences to say, but I'd bet on the first guess.
It seems like the P3 decoding speed results are close to what I normally see in comparisons with WavPack "fast" just about matching FLAC. But the P4 results show the
default WavPack mode matching FLAC and WavPack "fast" significantly faster than anything else!
Oh boy, more work to do...
Finally, here's the results for my laptop (AMD Turion 64 Mobile ML-36):
[..snip..]
NB: Different corpus to other test - no comparitive results (these are the first tests that I have run on my laptop).
Pleasing figures though.
Yes, I am perfectly happy with these results!
In fact, after looking over all the results I believe that this is certainly worth a release, especially considering that there doesn't seem to be regression anywhere.
Thanks again to everyone for their great help!
Yup no regression my side with hardcore punk, metal and ska so far (about 1500 files).
Silly question: any probs using your copytags.exe little proggy with this release too?
Thanks for your superb work, appreciated!
my short test
test on 70 short files (1cd):
-------------------------------------------------------
type.........................size....... | render time
-------------------------------------------------------
!orig\.....................1.502.251.352 | x
flac114_5\...................521.737.686 | ~3min
flac114_8\...................516.297.252 | ~7min
frog46ex_default\............480.359.971 | ~9min
wavpack441beta_default\......536.306.866 | ~2min
wavpack441beta_h\............528.509.776 | ~2min
wavpack441beta_hx\...........518.291.598 | ~5min
encoding times are aprox., decoding times not tested.
p.s. none of the encoders seems to use my two cpus.
p.s.2. sysinfo:
Property Value
Number of CPU(s) 2 Physical Processors / One Core / 2 Logical Processors
Vendor GenuineIntel
CPU Full Name Intel Xeon
CPU Name Intel® Xeon CPU 2.80GHz
CPU Code Name Prestonia
Technology 0.13µ
Platform Name Socket 603/604
Here's my quick test using the 'Finding Nemo' soundtrack on my P4 3GHz, 1MB RAM, XP Pro, default compression levels for all:
Original File: 638,775,020 bytes test_image.wav
flac 1.1.3
test_image.wav: wrote 329,487,790 bytes, ratio=0.516
Kernel Time = 3.546 = 00:00:03.546 = 2%
User Time = 62.890 = 00:01:02.890 = 48%
Process Time = 66.437 = 00:01:06.437 = 50%
Global Time = 130.422 = 00:02:10.422 = 100%
flac 1.1.4
test_image.wav: wrote 328,246,889 bytes, ratio=0.514
Kernel Time = 3.625 = 00:00:03.625 = 2%
User Time = 49.406 = 00:00:49.406 = 40%
Process Time = 53.031 = 00:00:53.031 = 43%
Global Time = 123.281 = 00:02:03.281 = 100%
wavpack-4.40.0
created test_image.wv (336,278,676 bytes) in 66.75 secs (lossless, 47.36%)
Kernel Time = 1.953 = 00:00:01.953 = 2%
User Time = 35.265 = 00:00:35.265 = 52%
Process Time = 37.218 = 00:00:37.218 = 55%
Global Time = 66.828 = 00:01:06.828 = 100%
wavpack-4.41.0-beta
created test_image.wv (336,278,676 bytes) in 64.36 secs (lossless, 47.36%)
Kernel Time = 2.187 = 00:00:02.187 = 3%
User Time = 33.156 = 00:00:33.156 = 51%
Process Time = 35.343 = 00:00:35.343 = 54%
Global Time = 64.422 = 00:01:04.422 = 100%
IMO, wavpack has
very impressive performance on my system. Josh, if you want to see any other tests, just yell.
John
If it's of any interest I have now updated my online comparison.
As I had to redo all encoders (I dropped some, like LA, and MA Insane) I am now reporting CPU-only times, which I'm pleased about.
Append ?all=1 to view the 4.41b results
http://www.synthetic-soul.co.uk/comparison...index.asp?all=1 (http://www.synthetic-soul.co.uk/comparison/lossless/index.asp?all=1)
If it's of any interest I have now updated my online comparison.
As I had to redo all encoders (I dropped some, like LA, and MA Insane) I am now reporting CPU-only times, which I'm pleased about.
Thank you very much!
And fine that you are reporting CPU-only times!
Hopefully preset Turbo will be about 5 % faster in the next TAK version (No, this isn't the new dedicated Turbo codec...). This tiny improvement would hardly be noticeable with disk io times...
Thomas
Kudos to Synthetic Sould and both of you TBeck and bryant and the FLAC developers, too, of course.
This chart really shows very well how exciting the lossless codec developments are at the moment.
If it's of any interest I have now updated my online comparison.
As I had to redo all encoders (I dropped some, like LA, and MA Insane) I am now reporting CPU-only times, which I'm pleased about.
Append ?all=1 to view the 4.41b results
http://www.synthetic-soul.co.uk/comparison...index.asp?all=1 (http://www.synthetic-soul.co.uk/comparison/lossless/index.asp?all=1)
Thanks again for your work and results, and I also agree that CPU-only speeds make the most sense.
I just wish I could get a few more percent on the decoding speed to break 100x. The way it looks now it's easy for people to get WavPack mixed up with the riff-raff!
LOL @ riff-raff. Considering WavPack is the second favourite encoder on this board (according to the poll) I don't think you need to worry about that.
I don't really understand the MMX, SSE stuff; however would I be right in saying that, due to the integer arithmetic nature of WavPack, SSE code is pointless? Anything else up your sleeve?
-f is so close to 100x decoding speed with this beta. Although, of course, not too much credance should be given to my single comparison - these tests kinda hit that home to me.
Anyway, I'm sure all your users will agree, the extra benefit provided by this beta with no loss in compression is extremely welcome. Many thanks for your continued efforts.
I don't really understand the MMX, SSE stuff; however would I be right in saying that, due to the integer arithmetic nature of WavPack, SSE code is pointless? Anything else up your sleeve?
Right. Because WavPack encoding is integer-only (and generally not parallelizable) there's little or nothing to be gained by SSE and his brothers. MMX
should be able to make a nice improvement because the left and right channels can be done together, but because of some optimization available for 16-bit data that doesn't work with MMX, the MMX stuff only really helps with 24-bit (and float) data.
I did get an idea from one of the FLAC 1.1.4 improvements that might be applicable to WavPack and I'll give that a try. You never know when something's going to make a big difference...
Anyway, I'm sure all your users will agree, the extra benefit provided by this beta with no loss in compression is extremely welcome. Many thanks for your continued efforts.
Ah, well it's my pleasure...
[...]Anyway, I'm sure all your users will agree, the extra benefit provided by this beta with no loss in compression is extremely welcome. Many thanks for your continued efforts.
Definitely extremely welcome and surely worth thanking whole heartedly for.
Haven't begun re-encoding from 4.3x to 4.4x for better DAP support yet, so the added speed is greatly appreciated.
[...]I did get an idea from one of the FLAC 1.1.4 improvements that might be applicable to WavPack and I'll give that a try. You never know when something's going to make a big difference...[...]
Are you going to try and include the above into the 4.41 changes or can we expect a separate release please?
Thanks a bunch once more for your time and brilliant work.
[...]I did get an idea from one of the FLAC 1.1.4 improvements that might be applicable to WavPack and I'll give that a try. You never know when something's going to make a big difference...[...]
Are you going to try and include the above into the 4.41 changes or can we expect a separate release please?
Thanks a bunch once more for your time and brilliant work.
I tried it the other day but wasn't able to make a big enough improvement for how ugly the change is. I'll work at a little more, but I'm not sure it's going to work out.
I have created a second beta that I believe makes another worthwhile improvement in performance. This involves accessing the bitstream in 16-bit chunks instead of 8-bit chunks, so it should affect the faster modes more (at least on a percentage basis).
Like the first beta, it should be bit identical to 4.40.0 in all circumstances.
Thanks in advance for any testing...
Here it is. (http://www.wavpack.com/wavpack-4.41.0-beta2.zip)
Some sourcecode would be nice for us non-windows users.
the svn server is at http://svn.slomosnail.de/wavpack (http://svn.slomosnail.de/wavpack)
grab subversion(if you don't already have it) and get it.
i'm just wondering if there is any harm having the --optimize-mono to be on by default?
IIRC, --optimize-mono only breaks compatability with any version of WavPack below 4.31
--optimize-mono is also buggy in lossy mode - bitrate drops , though only on dual-mono content.
i did a quick comparison using the --optimize-mono switch and without it in lossless mode, and the file sizes are the same, this is on normal stereo audio.
It only kicks in on dual-mono content or similar, so its safe to use in lossless if your player doesn't need < 4.31 decoder. From what I understand is that there is a new decoder that fixes this without the hack and will be used at some point in the future.. or maybe the --optimize hack will hardcoded instead ?
--optimize-mono is also buggy in lossy mode - bitrate drops , though only on dual-mono content.
Have you actually seen this lately? I
thought I fixed it before I released 4.40 but I never mentioned it in the changelog because it (the bug) never made it into a real release.
Now that the DirectShow filter is based on the latest library, the only decoder that can't handle the new mode is the XMMS plugin that I provide. Once I figure out how to build that with the latest library I could make the --optimize-mono the default, with perhaps a new switch to turn it off (for maximum compatibility).
Unless you're using the XMMS plugin (or are encoding files for a large audience), it's perfectly safe to use all the time.
Encoding speeds are amazing compared to flac, wavpack -hh seems much faster than flac -8 (order of 4 difference), decoding is about half the speed (only 40% difference with mode -h compared to the 100% difference of -hh). Good work, wavpack has the potential to beat flac. Keep up the good work.
--optimize-mono is also buggy in lossy mode - bitrate drops , though only on dual-mono content.
Have you actually seen this lately? I thought I fixed it before I released 4.40 but I never mentioned it in the changelog because it (the bug) never made it into a real release.
Now that the DirectShow filter is based on the latest library, the only decoder that can't handle the new mode is the XMMS plugin that I provide. Once I figure out how to build that with the latest library I could make the --optimize-mono the default, with perhaps a new switch to turn it off (for maximum compatibility).
Unless you're using the XMMS plugin (or are encoding files for a large audience), it's perfectly safe to use all the time.
http://www.hydrogenaudio.org/forums/index....ost&id=2888 (http://www.hydrogenaudio.org/forums/index.php?act=Attach&type=post&id=2888)
When encoding that sample with optimize-mono bitrate drops 60k or so. SNR is similar with or without the switch, but really should be much higher. If I add the missing 60k - i.e. change b320 to b380 I'll get what should be , I think..
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta2
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
ave noise = -59.18 dB, peak noise = -51.82 dB
created 001__08__Help_Me_Rhonda___Beach_Boys_cut___dual_mono.wv in 1.40 secs (lo
ssy, 301 kbps)
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta2
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
ave noise = -60.85 dB, peak noise = -53.20 dB
created 001__08__Help_Me_Rhonda___Beach_Boys_cut___dual_mono.wv in 1.40 secs (lo
ssy, 243 kbps)
When encoding that sample with optimize-mono bitrate drops 60k or so. SNR is similar with or without the switch, but really should be much higher. If I add the missing 60k - i.e. change b320 to b380 I'll get what should be , I think..
That's actually the intended behavior. In the original release that supported --mono-optimize, the bitrate would drop to about half the specified bitrate and the noise would go up significantly compared to the stereo encoding, which was obviously wrong. In the newer versions the target was to keep the noise constant and let the bitrate drop (but not as much as before).
My thinking was that a worst-case scenario for this would be a track that was basically pure mono, but with an occasional sample different (in one channel) by one LSB. In this case, most blocks would be encoded in the new special mono mode, but the blocks with the single sample difference would have to be encoded in the regular stereo mode. In this case I think the ideal situation would be to have the noise remain about the same, which would mean that the bitrate would jump up in the stereo blocks because the "mono" mode is more efficient (the whole reason it exists). I think the bitrate jumping around would not be an issue, but the noise jumping around might be audible (and if it wasn't audible then why waste bits to make it
even lower in the "mono" blocks?)
This way I can say that the --optimize-mono option can reduce the bitrate (in either lossless or lossy mode) with tracks that contain mono audio, but should never have any audible effect in lossy mode.
David
I have created a second beta... Thanks in advance for any testing...
Here is some evaluation of latest WavPack binaries conducted on my PIV Prescott 2.8ghz.
Ratio is calculated on my SetF.
| | Ratio | 4.40.00 | 4.41.0b1 | 4.41.0b2 |
| f | 65,66 | 96,7 115,6 | 100,4 124,4 | 102,4 137,6 |
| fx1 | 65,24 | 59,9 116,1 | 64,2 123,2 | 65,0 137,6 |
| fx2 | 65,17 | 45,1 117,1 | 49,1 123,8 | 49,4 137,6 |
| fx3 | 65,13 | 27,8 117,1 | 31,1 125,6 | 31,2 138,3 |
| fx4 | 65,08 | 12,0 114,5 | 13,5 124,4 | 13,5 139,8 |
| fx5 | 65,07 | 9,7 115,0 | 10,9 123,8 | 10,9 139,0 |
| fx6 | 65,07 | 8,3 115,6 | 9,3 123,2 | 9,3 139,8 |
| | 64,00 | 82,0 98,1 | 85,8 105,7 | 87,0 115,0 |
| x1 | 63,56 | 44,4 95,9 | 49,5 102,0 | 50,1 110,6 |
| x2 | 63,51 | 29,4 95,6 | 34,0 101,6 | 33,9 111,6 |
| x3 | 63,52 | 15,9 95,6 | 19,0 102,0 | 18,7 112,6 |
| x4 | 63,27 | 4,3 95,9 | 5,1 100,8 | 5,1 110,2 |
| x5 | 63,25 | 3,0 95,6 | 3,5 100,0 | 3,5 109,7 |
| x6 | 63,20 | 1,5 91,5 | 1,8 100,4 | 1,8 105,3 |
| h | 62,88 | 57,1 75,4 | 64,8 74,7 | 64,8 78,8 |
| hx1 | 62,78 | 30,0 76,5 | 34,7 78,3 | 34,4 83,1 |
| hx2 | 62,70 | 18,3 76,2 | 21,9 77,8 | 21,7 82,5 |
| hx3 | 62,66 | 9,2 76,2 | 11,4 77,8 | 11,3 82,3 |
| hx4 | 62,43 | 2,6 75,4 | 3,1 77,8 | 3,1 83,1 |
| hx5 | 62,39 | 1,9 76,2 | 2,3 78,3 | 2,3 82,8 |
| hx6 | 62,38 | 1,3 76,5 | 1,6 78,3 | 1,6 83,9 |
| hh | 62,47 | 49,7 60,9 | 52,5 62,7 | 53,0 65,8 |
| hhx1 | 62,28 | 22,2 61,2 | 26,6 61,8 | 26,7 64,8 |
| hhx2 | 62,22 | 12,9 61,8 | 15,8 61,9 | 15,8 65,0 |
| hhx3 | 62,17 | 6,3 60,9 | 7,9 61,8 | 7,9 65,0 |
| hhx4 | 62,01 | 1,7 62,2 | 2,0 61,5 | 2,0 65,3 |
| hhx5 | 61,96 | 1,0 62,1 | 1,2 61,8 | 1,2 65,5 |
| hhx6 | 61,96 | 0,7 62,4 | 0,9 61,8 | 0,9 66,0 |
A nice 16% encoding speed improvement and 12% decoding speed improvement, when going from 4.40 to 4.41b2.
Most encoding speed improvement is achieved with 4.41b1, most decoding speed improvement with 4.41b2.
Encoding speed improvement is slightly more focused on slower modes, while decoding speed improvement is the other way round.
When encoding that sample with optimize-mono bitrate drops 60k or so. SNR is similar with or without the switch, but really should be much higher. If I add the missing 60k - i.e. change b320 to b380 I'll get what should be , I think..
That's actually the intended behavior. In the original release that supported --mono-optimize, the bitrate would drop to about half the specified bitrate and the noise would go up significantly compared to the stereo encoding, which was obviously wrong. In the newer versions the target was to keep the noise constant and let the bitrate drop (but not as much as before).
This way I can say that the --optimize-mono option can reduce the bitrate (in either lossless or lossy mode) with tracks that contain mono audio, but should never have any audible effect in lossy mode.
David
Thanks for the explanation.
1 image file ~ 11 tracks ~ 300MB | Intel Celeron 1.7GHz. :
D:\Temp>wavpack -h *.wav
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created Evanescence - Fallen.wv in 129.54 secs (lossless, 32.76%)
D:\Temp>wavpack441b2 -h *.wav
WAVPACK Hybrid Lossless Audio Compressor Win32 Version 4.41.0-beta2
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
created Evanescence - Fallen.wv in 118.27 secs (lossless, 32.76%)
D:\Temp>wvunpack *.wv
WVUNPACK Hybrid Lossless Audio Decompressor Win32 Version 4.40.0
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
restored Evanescence - Fallen.wav in 91.87 secs (lossless, 32.76%)
D:\Temp>wvunpack441b2 *.wv
WVUNPACK Hybrid Lossless Audio Decompressor Win32 Version 4.41.0-beta2
Copyright © 1998 - 2006 Conifer Software. All Rights Reserved.
restored Evanescence - Fallen.wav in 80.67 secs (lossless, 32.76%)
Result :
WavPack v4.41b2 encoded the image file 11.27 seconds faster than v4.40.
WvUnpack v4.41b2 decoded the image file 11.20 seconds faster than v4.40.
Nice work, David - and many thanks for your continued efforts, my friend
(http://img89.imageshack.us/img89/1228/wv4410b2testxlsmh5.th.png) (http://img89.imageshack.us/my.php?image=wv4410b2testxlsmh5.png)
Nice improvement indeed!
Many thanks, David!
Thanks Josef, Martin, and DARcode for the tests!
These numbers are very much in line with what I am seeing here and I think a new release is certainly justified at this point.
Right. Because WavPack encoding is integer-only (and generally not parallelizable) there's little or nothing to be gained by SSE and his brothers. MMX should be able to make a nice improvement because the left and right channels can be done together, but because of some optimization available for 16-bit data that doesn't work with MMX, the MMX stuff only really helps with 24-bit (and float) data.
what about for PCM Float? SSE should be able to make a nice improvement there, shouldn't it?
Right. Because WavPack encoding is integer-only (and generally not parallelizable) there's little or nothing to be gained by SSE and his brothers. MMX should be able to make a nice improvement because the left and right channels can be done together, but because of some optimization available for 16-bit data that doesn't work with MMX, the MMX stuff only really helps with 24-bit (and float) data.
what about for PCM Float? SSE should be able to make a nice improvement there, shouldn't it?
Well, no. WavPack actually uses no floating-point operations even when operating on IEEE float data. I did this to ensure lossless operation even with slight variations in rounding behavior and to allow integer-only platforms (like Rockbox) to play IEEE float files even without an FPU.
Hi David,
there has been an idea in my mind for a while, and now I'd like to ask you, if you think this would be possible
As I read it from your docs, each block in the encoded data stream is independent from the previous one. So, if two blocks have the same parameters (which should be almost always the case in your current implementation - except for –optimize-mono), we could decorrelate them at once with SSE2 or AltiVec, couldn't we?
Regards,
Jo.
there has been an idea in my mind for a while, and now I'd like to ask you, if you think this would be possible
As I read it from your docs, each block in the encoded data stream is independent from the previous one. So, if two blocks have the same parameters (which should be almost always the case in your current implementation - except for –optimize-mono), we could decorrelate them at once with SSE2 or AltiVec, couldn't we?
I'm not completely up to speed (so to speak) on SSE2 or AltiVec, but what you suggest sounds reasonable. Unless the user specifies a -x mode, all blocks will have exactly the same sequence of terms and deltas.
I have also thought that the existing MMX code could also be used for 24-bit mono data by doing multiple blocks in parallel.
Of course, these changes would involve a lot more internal shuffling than the current stuff did.
BTW, please feel free to e-mail me directly on your ideas. I really do appreciate your input (and work so far).
David
edit: spelling
Of course, these changes would involve a lot more internal shuffling than the current stuff did.
Yes That's why I hoped to get some help from you I need to think a bit more about it. You know, absence of spare time is the biggest problem...
BTW, please feel free to e-mail me directly on your ideas. I really do appreciate your input (and work so far).
Thanks for the invitation! I think the developer mailing list would probably be a good place for discussion.
Is it release time yet please ?
Is it release time yet please ?
Well, there's going to be a beta3 to fix the issues raised in this (http://www.hydrogenaudio.org/forums/index.php?showtopic=53482) thread, and I'm also looking at quickly throwing in the --skip and --until commands from the FLAC decoder. So It'll be at least a few weeks before an official release...
David
Is it release time yet please ?
Well, there's going to be a beta3 to fix the issues raised in this (http://www.hydrogenaudio.org/forums/index.php?showtopic=53482) thread, and I'm also looking at quickly throwing in the --skip and --until commands from the FLAC decoder. So It'll be at least a few weeks before an official release...
David
Fantastic! Thanks a lot!
I just wish I could get a few more percent on the decoding speed to break 100x. The way it looks now it's easy for people to get WavPack mixed up with the riff-raff!
I've
finally got around to uploading my test results (http://www.synthetic-soul.co.uk/comparison/lossless/) for 4.41 final.
You may be pleased to see -f(x) achieving 104x when decoding.
It's just a shame (for WavPack marketing) that my CPU doesn't benefit as well as many.
I just wish I could get a few more percent on the decoding speed to break 100x. The way it looks now it's easy for people to get WavPack mixed up with the riff-raff!
I've finally got around to uploading my test results (http://www.synthetic-soul.co.uk/comparison/lossless/) for 4.41 final.
You may be pleased to see -f(x) achieving 104x when decoding.
It's just a shame (for WavPack marketing) that my CPU doesn't benefit as well as many.
Just checked, nice tests. I'd like to see this test run under a dual-core CPU, to see how the codecs would scale. I don't know how many (if any) support it. We would see a huge speed improvement. AFAIK, WavPack doesn't support it (yet)...
Maybe I'll run that tests in some time, maybe broading the test corpus to include other types of music!
Regards!
... run under a dual-core CPU, to see how the codecs would scale. I don't know how many (if any) support it. We would see a huge speed improvement.
You don't understand. On multi core machines you should run multi instances of the wavpack executable. Your hard disk will become the next bottleneck quite soon.