Skip to main content
Topic: WavPack 4.41 beta available (Read 58647 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

WavPack 4.41 beta available

I have been working on speed enhancements for WavPack and think I have made enough progress to justify a release. The improvement seems to vary quite a bit on different systems, but I couldn't find a system where this didn't make some improvement (even a few percent) and on some systems (like my old 1.7 GHz P4) it is significant. The improvements apply to both packing and unpacking stereo files. Note that this is still pure C; there is no hand-coded assembler.

This also includes the MMX code (via intrinsics) from Joachim Henke. Unfortunately, this only seems to provide an improvement for bitdepths over 16, so it's restricted to that for now (although it does make a nice improvement for 24-bit or IEEE float data). Regarding MMX support, is that ubiquitous enough now that programs simply use it without checking for it? Since WavPack now requires Windows 98, it is reasonable to assume that any computer running Windows 98 also has MMX?

A bug has been fixed that sometimes caused a crash if a metadata source file specified on the command-line was not found, but otherwise this release only involves optimization. The output files should always be bit-identical to 4.40 regardless of the mode. This code hasn't been checked in to SVN and I haven't tested it on Linux, but that will come shortly.

Many thanks to the WavPack users and also thanks in advance for any comparison testing done on this new version, which can be found here.

David


WavPack 4.41 beta available

Reply #2
Thank you very much for this new beta release, David  Your work is very much appreciated

I sometimes have some periods where i totally change my mind about something and this happened a little while ago, where i changed from WavPack to FLAC. Now their has gone about 14 days and now i'm beginning to come back to my old way of thinking again, and so i again changed my mind back and am now in the process of transcoding my FLAC images back to WavPack 

Again, many thank's for your great work David and i humbly apologise for my last 14 days of "un-loyalty" to the great WavPack format, and i hope that you will forgive me from changing "sides" momentarily and to let me pass into the WavPack community again

CU, Martin.

WavPack 4.41 beta available

Reply #3
Again, many thank's for your great work David and i humbly apologise for my last 14 days of "un-loyalty" to the great WavPack format, and i hope that you will forgive me from changing "sides" momentarily and to let me pass into the WavPack community again

Hi Martin! 

I actually almost thanked all the faithful WavPack users in my original post as a subtle reference to your very visible defection, but my good judgement prevailed. Glad you're back! 

After what sometimes seems like daily announcements for FLAC support in various software and hardware, it doesn't surprise me that WavPack's popularity has slipped over time (at least with HA members). I have done everything I possibly can to make incorporating WavPack support in a product painless, but I suspect that if a vendor has to choose a single (or first) format to support then FLAC is the obvious choice simply because it already has the widest support. My hope is that some WavPack fan in the right position will steer a decision my way, or I will get a job with a company making audio devices and do the same (I interviewed at a well-known music server maker, but never got a call back...  ).

In any event, WavPack can continue to provide a viable alternative, especially in applications where one of its features helps fill a niche (like the hybrid/lossy mode, float support, RIFF chunks, integer-based encoding, etc.), and can also provide Josh with incentive to continue pushing the envelope with FLAC.

That is, of course, until everyone switches to Tak! 



Well, AFAIK, you can run Windows 98 on a 486DX without MMX, but I doubt anyone is doing that. Here's a list of CPUs that support MMX (German, but you should easily get the point):

http://de.wikipedia.org/wiki/MMX#CPUs_mit_MMX

Thanks for the info. I guess I'll not worry about it for now except maybe a notice somewhere that audio resolutions greater than 16-bit require an MMX-capable CPU.

WavPack 4.41 beta available

Reply #4
You sound like being a bit sad because of hardware support which is bigger for FLAC.

Hope you're not really dissappointed. You do such a great work.
And may be when lossless encoding becomes more widespread than now also top quality lossy encoding becomes more widespread cause it saves some 50% of file size or more compared to lossless while keeping extremely good quality.

As for lossy/hybrid mode wavPack has the best hardware suppport. When we'll have 16 or 32 GB flash players (expected to arrive soon) with rockbox support (hopefully) or maybe even natively (even better), wavPack lossy will be one of the most adequate fomats to many people.

Keep up the good work !
lame3995o -Q1.7
opus --bitrate 140

WavPack 4.41 beta available

Reply #5
I'm not sure Martin.  I'm not sure that I want to hang around with a "FLAC user".  As long as you can promise that you have kicked the habit completely, and will not "use" again then perhaps we can begin your rehabilitation one step at a time...

Hey, who am I kidding: you're a very useful person to have onside.  Welcome back brother!

I'm a bit gutted actually. I've been running tests all day, on various settings using both 4.40 and 4.41, and the testing had about 10-15 minutes to go and I had to shut down the PC so that we could put my son to bed.  Wife's orders.

Looks like I will have to finish the decoding tomorrow (20:00pm here) and post then - around 15 hours' time.  Shame, as I was really looking forward to seeing the figures

Anyways, thanks for the update David.
I'm on a horse.

WavPack 4.41 beta available

Reply #6
Just wanted to add my 2 cents.  I've been using WavPack for quite some time now.  I admit I still use FLAC, but WavPack is my favorite.  I find it particularly good for my own music, since my masters are 32-bit audio.  I'm not sure if FLAC even supports that bit depth.  but 32/96 and 32/48 audio takes up some space, and WavPack compresses these files quite nicely (-hhxm).  One thing I would like to see is support for WavPack in the Squeezebox, at least in the software.  I wouldn't worry about hardware support too much, I really don't think that many people put lossless audio on portable players.  Might be something to make a poll for.  So keep up the development.  Been doing a great job so far.  Fact is WavPack was basically unknown a few years ago and now seems to be in the top 3.

WavPack 4.41 beta available

Reply #7
Hi Martin! 

I actually almost thanked all the faithful WavPack users in my original post as a subtle reference to your very visible defection, but my good judgement prevailed. Glad you're back! 

Thank you very much David, i really appreciate it  Btw, i also changed to using only GNU/Linux(Debian) instead of win32, besides changing to FLAC, but now i'm back for good with WavPack and on win32

Thank's again, my friend

CU, Martin.

I'm not sure Martin.  I'm not sure that I want to hang around with a "FLAC user".  As long as you can promise that you have kicked the habit completely, and will not "use" again then perhaps we can begin your rehabilitation one step at a time...

ROFLMAO  You are hillarious, Synthetic Soul 
Quote
Hey, who am I kidding: you're a very useful person to have onside.  Welcome back brother!

Thank you very much for the kind words, my friend

CU, Martin.

WavPack 4.41 beta available

Reply #8
sweet. changes are now in SVN . now if only Visual C++ didn't crash everytime i started compiling


WavPack 4.41 beta available

Reply #10
right now, that's what i'm using(although i havent' encoded any songs yet). but for some reason, if i compile my own versions, then i get a faster binary which is faster by 10 seconds. it's not much, but i guess it's worth it.

i personally want to use MinGW to compile it and compare differences between VC++ and GCC since VC++ usually makes bloated exes.

WavPack 4.41 beta available

Reply #11
I'm a bit gutted actually. I've been running tests all day, on various settings using both 4.40 and 4.41, and the testing had about 10-15 minutes to go and I had to shut down the PC so that we could put my son to bed.  Wife's orders.

Looks like I will have to finish the decoding tomorrow (20:00pm here) and post then - around 15 hours' time.  Shame, as I was really looking forward to seeing the figures

I too am interested to see what happens on other systems. This optimization business can be kind of a circus with all the different CPUs and compilers and their switches. Sometimes I'd come up with an improvement and it would be 20% faster on my primary system, then I'd try it at work and it would be much slower! What I finally came up with was faster on all three systems I tried, so my fingers are crossed... 

I just built it on gcc for Linux and most of the improvements seem to work there as well, especially when I tell gcc to optimize for my exact CPU (a feature which Microsoft seems to have decided was more trouble than it was worth and deleted from VC8).

Anyway, Neil, thanks for giving it a go and thanks to everyone for the kind words and support! 

Now my wife is trying to put me to bed, so I'm going to have to wait also... 

David

WavPack 4.41 beta available

Reply #12
OK, finally, here they are:

Code: [Select]
         |           |    4.40.0    |    4.41
Setting  |  Comp %   |  Enc    Dec  |  Enc    Dec
=========|===========|==============|============
f        |  66.741%  |  68x    92x  |  73x    97x
  x      |  66.539%  |  38x         |  40x
  x2     |  66.445%  |  28x         |  29x
  x3     |  66.386%  |  16x         |  17x
default  |  65.582%  |  59x    80x  |  64x    82x
  x      |  65.370%  |  29x         |  30x
  x2     |  65.312%  |  18x         |  20x
  x3     |  65.285%  |  10x         |  10x
h        |  64.877%  |  40x    65x  |  43x    67x
  x      |  64.764%  |  20x         |  21x
  x2     |  64.679%  |  12x         |  12x
  x3     |  64.679%  |   6x         |   6x
hh       |  64.487%  |  32x    53x  |  34x    54x
  x      |  64.422%  |  14x         |  16x
  x2     |  64.394%  |   8x         |   9x
  x3     |  64.378%  |   4x         |   4x

There is definate improvement, most noticeable when encoding when the x switches are not used, and more prominent with the faster settings.

All 16 bit files I'm afraid, so no test for the MMX code.


NB: Given that 144 people so far have stated that WavPack is their codec of choice in the 2007 poll, it would be nice if more than three people provided some data.

C'mon kiddies: time to show the nice man your appreciation...
I'm on a horse.

WavPack 4.41 beta available

Reply #13
Hi Bryant,

A quick note to add my thanks for your efforts.

I've been using it for a couple of years now and, coming from an audio (rather than software) background, WavPack has been one of the things that made me a convert to PC-based audio.

Best

Midas

WavPack 4.41 beta available

Reply #14
You are right Synthetic Soul.
I was using FLAC/REACT to backup my collection, then I had a HD crash (after about 100 CDs).
Anyhow, after I got the computer back and running, I seriously looked at WV versus FLAC.  I made the switch to wavpack images to backup my collection.  I love it.
I also tried 3 CD rips using REACT/wavpack beta, and it works great.  I don't have the numbers to back up my claim, but the speed increase is great.
Thank you for such a great program.
And if there is any concern, it works great on Vista.

 

WavPack 4.41 beta available

Reply #15
There is definate improvement, most noticeable when encoding when the x switches are not used, and more prominent with the faster settings.

Thanks for the test! It's interesting that the improvement was so modest, but it's comforting to see that none of the settings seemed to get slower. 

On my older system the decoding improvement was about 15% across all modes and the encoding improvement was as much as 30% in the higher modes. Hopefully some other results will show more significance; I can't imagine this alone being worth its own release (although it's obviously worth keeping).

WavPack 4.41 beta available

Reply #16
NB: Given that 144 people so far have stated that WavPack is their codec of choice in the 2007 poll, it would be nice if more than three people provided some data.

C'mon kiddies: time to show the nice man your appreciation...

Easier said than done. 

Ok, I fetched your comparison-scripts-v1.2 (hope it's the latest version) pack and modified it for comparing 4.40 against 4.41...
that's two times 16 folders. What's the maximum disc requirement I can expect to be used? I hope the test en-/decodes will be deleted right after the processing so I won't end up with 8-16 times the size of the source folder, right?

Does it calculate the average times or do I have to do that myself (I think I've seen the word "spreadsheet" mentioned in the lossless comparison thread... I guess I could get that to work with OpenOffice Calc).

Another question: Can I skip the md5 part? I don't think it's necessary and will only make the test longer (and more likely that I will cancel it).

Currently fb2k is still decoding the originals.

WavPack 4.41 beta available

Reply #17
Hi David,
In any event, WavPack can continue to provide a viable alternative, especially in applications where one of its features helps fill a niche (like the hybrid/lossy mode, float support, RIFF chunks, integer-based encoding, etc.), and can also provide Josh with incentive to continue pushing the envelope with FLAC.

That is, of course, until everyone switches to Tak! 

I will be happy, if TAK will find it's niche, at least big enough to justify all the work put into it's release... But thank you very much for the encouragement! 

BTW: For me WavPack is a great codec! Especially for it's simplicity. Please don't take me wrong, i want to say in my bad english: simple + efficient = ingenious! With simple i mean the code complexity, not your great altgorithm. I would be glad, if TAK was equally compact, but no chance.

I also like your simple to use code interfaces. I hope you don't mind, that i used them as a model for my new SDK...

I want to say thank you for your great work and i really hope, you will continue it. But please don't improve it too soon too much, please give TAK some time... Honestly i am always most curious if a new Wavpack version is beeing released, because regarding efficiency, WavPack is TAK's real (friendly) competitor.

Let's work on further improvements and have some more fun! 

  Thomas

WavPack 4.41 beta available

Reply #18
Thanks for the test! It's interesting that the improvement was so modest, but it's comforting to see that none of the settings seemed to get slower. 
I seem to remember that my corpus didn't fair so well with your improvements last time.  I was kinda hoping someone else would post some results before mine to balance it out a bit!  Anyway, considering that they are pure C amendments, and as you say none have worsened, it's most certainly better than a poke in the eye with a sharp stick.  To try to make it sound slightly better: with the default encoding I shaved 15 seconds off a 202 second encode phase.  I don't think that's all that bad.

Easier said than done.
True.  I'm glad that you have made an effort though.  David deserves some help from us.

Ok, I fetched your comparison-scripts-v1.2 (hope it's the latest version) pack and modified it for comparing 4.40 against 4.41...
that's two times 16 folders. What's the maximum disc requirement I can expect to be used? I hope the test en-/decodes will be deleted right after the processing so I won't end up with 8-16 times the size of the source folder, right?
MB used depends on the size of your corpus.  If you run encodeall.bat and then decodeall.bat you will need room for the source, all encodes, and one decode set (1*wav + n*wv + 1*wav) before you start removing files with the decode scripts.  If this is a concern write a short batch file to call a folder's encode.bat and decode.bat in turn - then you only need room for your corpus and one encode/decode set (1*wav + 1*wv + 1*wav).

Does it calculate the average times or do I have to do that myself (I think I've seen the word "spreadsheet" mentioned in the lossless comparison thread... I guess I could get that to work with OpenOffice Calc).
You have to collate the figures yourself.  I used Excel for the report above.  It's a simple/boring copy'n'paste job...

Another question: Can I skip the md5 part? I don't think it's necessary and will only make the test longer (and more likely that I will cancel it).
Remove:

Code: [Select]
@CALL "%~dp0md5.bat"

... from all decode.bat files.

BTW:  All 1600 encodes (50 files encoded using 16 settings for 2 versions) matched for me.

NB: I have uploaded my spreadsheet and all script files (including the resulting txt files) here.
I'm on a horse.

WavPack 4.41 beta available

Reply #19
Easier said than done.
Well, obviously I appreciate the incredible amount of work that Synthetic Soul has put into his tests and precedures. However, it is also possible to provide useful information on different systems with virtually no work. For example, here are results from my system using just a single track. I always do a couple test runs (not shown) to make sure the sources are in disk cache and I either write the output to "nul" for encoding or just use verify mode for decoding to make sure that no disk I/O is entering the timing. I get results like this:

Decoding test (sorry, you have to scroll a little to see the differences):
Code: [Select]
C:\devel\opt\trunk>wvunpack -v Track01*

 WVUNPACK  Hybrid Lossless Audio Decompressor  Win32 Version 4.40.0
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.


Track01hh.wv:
verified Track01hh.wv in 8.23 secs (lossless, 39.69%)

Track01f.wv:
verified Track01f.wv in 3.88 secs (lossless, 37.35%)

Track01.wv:
verified Track01.wv in 4.86 secs (lossless, 38.31%)

Track01ff.wv:
verified Track01ff.wv in 3.82 secs (lossless, 36.78%)

Track01hx.wv:
verified Track01hx.wv in 6.38 secs (lossless, 39.46%)

 **** 5 files successfully processed ****

C:\devel\opt\trunk>release\wvunpack -v Track01*

 WVUNPACK  Hybrid Lossless Audio Decompressor  Win32 Version 4.41.0-beta
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.


Track01hh.wv:
verified Track01hh.wv in 7.35 secs (lossless, 39.69%)

Track01f.wv:
verified Track01f.wv in 3.34 secs (lossless, 37.35%)

Track01.wv:
verified Track01.wv in 4.14 secs (lossless, 38.31%)

Track01ff.wv:
verified Track01ff.wv in 3.33 secs (lossless, 36.78%)

Track01hx.wv:
verified Track01hx.wv in 5.66 secs (lossless, 39.46%)

 **** 5 files successfully processed ****
Encoding test:
Code: [Select]
C:\devel\opt\trunk>wavpack -fy Track01 nul

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.40.0
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created nul.wv in 4.98 secs (lossless, 37.35%)

C:\devel\opt\trunk>release\wavpack -fy Track01 nul

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.41.0-beta
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created nul.wv in 4.37 secs (lossless, 37.35%)

C:\devel\opt\trunk>wavpack -hhy Track01 nul

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.40.0
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created nul.wv in 13.43 secs (lossless, 39.55%)

C:\devel\opt\trunk>release\wavpack -hhy Track01 nul

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.41.0-beta
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created nul.wv in 10.19 secs (lossless, 39.55%)

C:\devel\opt\trunk>wavpack -hhxy Track01 nul

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.40.0
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created nul.wv in 22.36 secs (lossless, 39.64%)

C:\devel\opt\trunk>release\wavpack -hhxy Track01 nul

 WAVPACK  Hybrid Lossless Audio Compressor  Win32 Version 4.41.0-beta
 Copyright © 1998 - 2006 Conifer Software.  All Rights Reserved.

created nul.wv in 18.27 secs (lossless, 39.64%)

Finally, when I want to verify that I haven't broken anything I just pipe the output to md5:
Code: [Select]
C:\devel\opt\trunk>release\wavpack -hhy Track01 -q - | md5 -
3EA3537450C0189192174CAFAA6CA506  -

C:\devel\opt\trunk>wavpack -hhy Track01 -q - | md5 -
3EA3537450C0189192174CAFAA6CA506  -

Hopefully someone will get results somewhat like mine, otherwise I spent all that time optimizing for a single computer! 

WavPack 4.41 beta available

Reply #20
Thanks for the test! It's interesting that the improvement was so modest, but it's comforting to see that none of the settings seemed to get slower. 
I seem to remember that my corpus didn't fair so well with your improvements last time.  I was kinda hoping someone else would post some results before mine to balance it out a bit!  Anyway, considering that they are pure C amendments, and as you say none have worsened, it's most certainly better than a poke in the eye with a sharp stick.  To try to make it sound slightly better: with the default encoding I shaved 15 seconds off a 202 second encode phase.  I don't think that's all that bad.

Yeah, the fast and normal modes show a nice improvement. I also saw less improvement on the 2 newer systems that I tried (although still more than you) so I suspect that somehow the newer CPU is simply able to more efficiently handle the things I am optimizing out.

BTW, how do these results compare to other results that you have for other codecs? The numbers don't seem to jive with the public version of your corpus that's here so I suspect the procedure (or system) is different now.

WavPack 4.41 beta available

Reply #21
BTW: For me WavPack is a great codec! Especially for it's simplicity. Please don't take me wrong, i want to say in my bad english: simple + efficient = ingenious! With simple i mean the code complexity, not your great altgorithm. I would be glad, if TAK was equally compact, but no chance.
Thanks! It's interesting that one of my goals for WavPack 4.0 was to make it as simple as possible. So, I worked hard on figuring out decorrelation and entropy encoding methods that would work well with both lossless and hybrid modes and work well with both fast and slow modes, and was very happy with the final result.

Of course, now, when I want to speed things up, I realize that some of the compromises I made for simplicity now are costing in efficiency. I could create a new mode specifically for lossless that would achieve the same compression at greater speed (and be more MMX'able) but that's what I was trying to get away from.

Quote
I also like your simple to use code interfaces. I hope you don't mind, that i used them as a model for my new SDK...
Of course I don't mind; I'm glad you found something useful there!

Quote
I want to say thank you for your great work and i really hope, you will continue it. But please don't improve it too soon too much, please give TAK some time... Honestly i am always most curious if a new Wavpack version is beeing released, because regarding efficiency, WavPack is TAK's real (friendly) competitor.
Don't worry, I don't think there's anything coming soon. I think the only way to make a significant improvement in WavPack's speed would be to start rewriting several large functions is pure assembler. Now, I would really like to do that, but unfortunately I just can't justify the time required.

BTW, I haven't been following all the TAK discussion closely enough to know whether you are using any hand-coded assmebler in there. I remember at one point you said that it was pure Delphi, but I was wondering if that was still the case (I know you must at least be using some intrinsics for MMX). And if not, do you think you'll be able to get further speed gains if you ever tried that?

Quote
Let's work on further improvements and have some more fun! 
Absolutely!

BTW, if you'd like to send me your code I would be happy to look it over and let you know what I think. I bet Josh would even help me and I'm sure we could come up with some good ideas for you! 

WavPack 4.41 beta available

Reply #22
Very brief test on a P3-550MMX:

normal mode +6 % improvement

high + 8.7 %

v-high: + 10 %

fast: + 7 %

normal x1: + 15 %

normal x3: + 18 %

v-high x3: + 20 %


Good work David !
wavpack 4.8 -b3x6c

WavPack 4.41 beta available

Reply #23
Very brief test on a P3-550MMX:

normal mode +6 % improvement

high + 8.7 %

v-high: + 10 %

fast: + 7 %

normal x1: + 15 %

normal x3: + 18 %

v-high x3: + 20 %


Good work David !

Thanks for letting me know. I'm glad that someone's getting results along these lines! 

WavPack 4.41 beta available

Reply #24
BTW, how do these results compare to other results that you have for other codecs? The numbers don't seem to jive with the public version of your corpus that's here so I suspect the procedure (or system) is different now.
The results posted here are for CPU-only (my preferred format), while the comparison on my site reports CPU+IO (that's how it started, so for now, unfortunately, it must continue that way).

All the results on my site were produced with 512MB of RAM in the machine.  I recently got given 1GB so I could upgrade.  I am concerned about mixing results from both set-ups.  I'm a bit dim when it comes to such things: would the additional RAM change results?

I did some testing for FLAC 1.1.4 recently.  I re-did the 1.1.3 tests and compared the 512MB test and the 1GB test and the figures appeared to be very similar.  My intention was to compare the 512MB and 1GB 4.40 results also.

Anyway, for now there are my FLAC 1.1.4 results, but I'll try to make a correlation with the others if I can.
I'm on a horse.

 
SimplePortal 1.0.0 RC1 © 2008-2019