Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: WavPack 4.75 Release Candidate (Read 23371 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

WavPack 4.75 Release Candidate

I have been working on a new release of WavPack for a few months, and I am ready with a release candidate. There are no new features, and in fact the new code should generate the same binary output as 4.70, however there have been many internal changes and optimizations.

The biggest change is that I took out the MMX intrinsics and replaced them with assembly (although still using MMX instructions when they were applicable). This means the optimizations have been extended to the decoding side and to mono files (previously they were only for stereo encoding), and I can now provide 64-bit executables with the optimizations (the Visual Studio compiler dropped support for MMX intrinsics when they went to 64-bit). Basically there is a nice speed boost for most operations on most processors (although the “higher” modes and 24-bit audio benefit more).

The code has also been reorganized to be more modular, which results in smaller images when linking to projects that do not require all the functionality, and it's also easier to understand. On Linux (and related systems) the configure script chooses the appropriate assembler optimizations based on the host processor (x86, x64, or ARM) unless explicitly disabled with --disable-asm.

Any testing or feedback is greatly appreciated; I hope there are still some WavPack users around! With any luck I should be ready for a full release in a week or so.

32-bit Windows Executables

64-bit Windows Executables

Linux Distribution

Thanks!

--David

WavPack 4.75 Release Candidate

Reply #1
I hope there are still some WavPack users around!

I feel guilty if I don't post a reply after seeing this. I've been using wavpack since 2007, and love how it handles floating point and non-standard sample rate files nicely. I didn't use flac until I know about flaccl.

Just tried 64-bit wavpack.exe as foobar's encoding plugin. Couldn't see encoding speed improvements, may be my harddisk is the bottleneck. It produced identical files as the (current?) version dated in 14/12/2014 so I think I don't need to worry about compatibility.

WavPack 4.75 Release Candidate

Reply #2
32-bit wavpack crashes here (as well as wvunpack). 64-bit seems to be fine.

Unhandled exception at 0x00442485 in wavpack.exe: 0xC0000005: Access violation executing location 0x00442485.

[disassembly removed as useless]

Looks like the beginning of pack_decorr_stereo_pass_cont_rev_x86. Also decoding crashes at the beginning of unpack_decorr_stereo_pass_cont_x86.

Added: OS is Win7SP1 x64. No problems in VM with WinXPSP3.

Added2: the problem disappear if I compress wavpack.exe with UPX. Strange problem indeed.

Added3: found the problem, see post #9.

WavPack 4.75 Release Candidate

Reply #3
Not really seeing any speed improvements (win x64), I'm guessing my Phenom II does not fall under "most processors" ^^;

WavPack 4.75 Release Candidate

Reply #4
Also, I downloaded wavpack-4.75.0-rc.tar.bz2 and tried to build it under MSYS2+MinGW from http://xhmikosr.1f0.de/tools/msys/ (I typed ./configure, then make)

make fails with the following error:
Code: [Select]
.libs/libwavpack_la-extra1.o: In function `recurse_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:228: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `sort_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:362: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `analyze_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:416: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `delta_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:293: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `analyze_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:421: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o:E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:318: more undefined references to `log2buffer_x86' follow
.libs/libwavpack_la-extra2.o: In function `decorr_stereo_pass':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:266: undefined reference to `pack_decorr_stereo_pass_cont_x86'
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:264: undefined reference to `pack_decorr_stereo_pass_cont_rev_x86'
.libs/libwavpack_la-extra2.o: In function `recurse_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:414: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `sort_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:549: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `analyze_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:603: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `delta_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:480: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `analyze_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:608: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o:E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:505: more undefined references to `log2buffer_x86' follow
.libs/libwavpack_la-pack.o: In function `pack_samples':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/pack.c:976: undefined reference to `pack_decorr_stereo_pass_x86'
E:\MSYS\home\user\wavpack-4.75.0-rc\src/pack.c:957: undefined reference to `pack_decorr_mono_buffer_x86'
.libs/libwavpack_la-unpack.o: In function `unpack_samples':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/unpack.c:122: undefined reference to `unpack_decorr_mono_pass_cont_x86'
E:\MSYS\home\user\wavpack-4.75.0-rc\src/unpack.c:180: undefined reference to `unpack_decorr_stereo_pass_cont_x86'
collect2.exe: error: ld returned 1 exit status
make[1]: *** [libwavpack.la] Error 1
make[1]: Leaving directory `/home/user/wavpack-4.75.0-rc/src'
make: *** [all-recursive] Error 1

Added: I managed to compile it when I added leading underscores to all .globl symbols in pack_x86.s and unpack_x86.s (log2buffer_x86 -> _log2buffer_x86, etc). It seems that underscores are necessary for Win32 build.

WavPack 4.75 Release Candidate

Reply #5
Thanks David. This one appears 3 .. 4 %  faster than 4.60.1 [default mode - win 7 - 32bit]

WavPack 4.75 Release Candidate

Reply #6
Here's a detailed comparison between 4.70 and 4.75rc on the test rig I use for lossless audio codec comparison I publish. Data was acquired with a AMD A4-3400 running Windows 7 with 4GB of RAM. Files were loaded and processed in a RAMdisk. CPU frequency was locked to the maximum. The encoding swiches tested are, in order, -f, [none], -h, -h -x, -hh, -hh -x.




PDF with data per track can be found here.

The results indeed seem to indicate a speed boost dependent of encoding mode. The speed boost is indeed bigger for 24-bit files (see the track-01-01[1]-03-[Lf-Rf-C-LFE-Ls-Rs]-24-48000 file). There is one rather strange case in the per-track PDF, ZOOM0004, which is a extremely quiet recording of a acoustic guitar. For some reason, -h compressed less than the normal mode and the fast mode, and -hh does even worse. Maybe you want to look into that?


Music: sounds arranged such that they construct feelings.

WavPack 4.75 Release Candidate

Reply #7
To whom it may concern: WavPack 4.75rc compiled with GCC 4.9.2. It should be a bit faster than MSVC.

System Requirements: at least i686 CPU. Also I'm not sure that MinGW-w64 supports Win2000 (should work on WinXP without problems).

WavPack 4.75 Release Candidate

Reply #8
I ran "dumpbin /headers wavpack.exe" and noticed that asmcode section in 32-bit executables has flags=0xC0000040 (Initialized Data, Read, Write) and it probably should have flags=0x60000020 (Code, Execute, Read).

(IOW wavpack crashes because it tries to execute code from data section)

So, TWIMC-2: 32-bit executables from bryant fixed with "editbin /section:asmcode,ce!i!w *.exe".


WavPack 4.75 Release Candidate

Reply #10
...

Any testing or feedback is greatly appreciated; I hope there are still some WavPack users around!

...


Hi,

I am a happy WavPack user for several years, and thanks for updating it, but please consider reverting this change in 4.70 :
Quote
added: several Windows features to Linux (clean ^C handling, console title)


On Ubuntu, all terminals that start a wavpack process without the -z switch are left with a "WavPack completed" title even after the process has died.

Changing the terminal title on Unix is considered intrusive and should be left to a very few special programs like SSH that actually change the state of the terminal session, and need to reflect that in the title.
Opus 96 kb/s (Android) / Vorbis -q5 (PC) / WavPack -hhx6m (Archive)

WavPack 4.75 Release Candidate

Reply #11
Thanks so much to everyone for the feedback and testing; it is greatly appreciated!!

I have made a few updates at GitHub based on these findings and comments. Unless something else comes up, I don't think another release candidate will be needed.

The results indeed seem to indicate a speed boost dependent of encoding mode. The speed boost is indeed bigger for 24-bit files (see the track-01-01[1]-03-[Lf-Rf-C-LFE-Ls-Rs]-24-48000 file). There is one rather strange case in the per-track PDF, ZOOM0004, which is a extremely quiet recording of a acoustic guitar. For some reason, -h compressed less than the normal mode and the fast mode, and -hh does even worse. Maybe you want to look into that?

Thanks for the exhaustive testing (I have looked over your compilations with interest in the past). Yes, I understand why some very quiet tracks might compress better in faster modes. The problem is that the block headers for the higher modes get bigger and bigger. Normally the improved compression swamps these few extra bytes, but I guess in some cases it doesn't (although I think I've only seen this in experimental files). Unfortunately there's not a lot I can do that wouldn't spend extra time (I'd have to try re-encoding blocks that seemed to show this). Fortunately, I don't think this is very common... 

To whom it may concern: WavPack 4.75rc compiled with GCC 4.9.2. It should be a bit faster than MSVC.

Thanks for creating these, and they do seem somewhat faster than the MSVC versions, especially the x64 version (and packing vs. unpacking). I had actually noticed this myself because my Linux versions and my Cygwin64 versions on Windows both run faster than the MSVC versions. Do you happen to know if this is something that will be better after an upgrade to Visual Studio 2013?

Thanks for the other catches too!

hi, I will do some testing when I get home, just wondering if the 4gb+ support is in this version (http://www.hydrogenaud.io/forums/index.php?showtopic=108891)

Sorry, the 4GB stuff is for the next release. I wanted to isolate these speed improvements and reorganization from the slew of new features coming... 

Changing the terminal title on Unix is considered intrusive and should be left to a very few special programs like SSH that actually change the state of the terminal session, and need to reflect that in the title.

Thanks for the heads-up on this. I really like the console status because I can easily see when my WavPack operations are done by just glancing at the panel, but I never could figure out how to portably restore the previous title (and I don't think I'm alone because Vim leaves that "Thanks for flying Vim"). Anyway, I've changed the console title update to disabled by default on Linux, but it can be enabled with "-z1" (and -z still leaves it off).

WavPack 4.75 Release Candidate

Reply #12
Thanks for creating these, and they do seem somewhat faster than the MSVC versions, especially the x64 version (and packing vs. unpacking). I had actually noticed this myself because my Linux versions and my Cygwin64 versions on Windows both run faster than the MSVC versions. Do you happen to know if this is something that will be better after an upgrade to Visual Studio 2013?


I realize that FLAC != WavPack, but the FLAC 1.3.1 thread seemed to indicate that the official GCC compile beat both MSVC and ICL compiles, at least for recent processors.  It's possible GCC has just gotten that good.

WavPack 4.75 Release Candidate

Reply #13
Here's a detailed comparison between 4.70 and 4.75rc on the test rig I use for lossless audio codec comparison I publish. Data was acquired with a AMD A4-3400 running Windows 7 with 4GB of RAM. Files were loaded and processed in a RAMdisk. CPU frequency was locked to the maximum. The encoding swiches tested are, in order, -f, [none], -h, -h -x, -hh, -hh -x.

on my CPU (Intel Core2) encoding with -x4 becomes 1.4x faster. Maybe it makes sense to test -hh -x4? It's very slow, but usually there's a noticeable increase in compression ratio between -x3 and -x4.

PDF with data per track can be found here.

I don't understand why on some tracks -h -x encodes with zero speed: (01 - Autobahn and 01 - Part One).

WavPack 4.75 Release Candidate

Reply #14
Thanks for creating these, and they do seem somewhat faster than the MSVC versions, especially the x64 version (and packing vs. unpacking). I had actually noticed this myself because my Linux versions and my Cygwin64 versions on Windows both run faster than the MSVC versions. Do you happen to know if this is something that will be better after an upgrade to Visual Studio 2013?

On my computer GCC 4.9.2 compiles are slightly faster than MSVC 2013.

WavPack 4.75 Release Candidate

Reply #15
Here's a detailed comparison between 4.70 and 4.75rc on the test rig I use for lossless audio codec comparison I publish. Data was acquired with a AMD A4-3400 running Windows 7 with 4GB of RAM. Files were loaded and processed in a RAMdisk. CPU frequency was locked to the maximum. The encoding swiches tested are, in order, -f, [none], -h, -h -x, -hh, -hh -x.

on my CPU (Intel Core2) encoding with -x4 becomes 1.4x faster. Maybe it makes sense to test -hh -x4? It's very slow, but usually there's a noticeable increase in compression ratio between -x3 and -x4.

This is actually something that has always made the WavPack results a little difficult for me to understand: the mixing of -x modes with the non-x modes on the same line. I know other codecs work this way to some extent (and none have more than one line), but WavPack is sort of two codecs with two different purposes.

For realtime encoding (like with Reaper), the -x switches makes no sense because they slow encoding down a lot with almost no gain in compression. On a compression vs. encoding speed chart they look silly. However, since the -x switches shouldn't affect decoding speed at all, they make perfect sense for archiving of material that will be decoded many times (like a new CD).

In a perfect world, I think that the only scenarios that make sense are the four basic modes without -x, and the four basic mode with -x6. Of course, if someone is ripping a lot of CDs then that might be prohibitive if it ended up taking longer than the ripping, and a test like yours might stretch to a long time (not considering your electricity bill!) but in the latest version I can encode an average CD in 5 minutes with -hx4, and that's with just one core.

I guess what I'm saying is that it might make your charts clearer to either just do the four basic modes without -x, or, if you want to include the -x switch in a comprehensible way, make a separate line containing the four basic modes with some constant -x mode, like -x4 (which lvqcl suggests is a good compromise).

That way, on the compression vs. encoding chart the two lines would be completely separated (and WavPack -xn line would not fare too well, obviously), but at least someone could look at the compression vs. decoding chart and see that extra bump that the -x4 (or whatever) gives for each mode.

I realize that these are your charts and I would not want to be accused of suggesting things to make WavPack look better than it is, but I have to admit that every time I look at them I get very confused trying to figure out the WavPack results (it makes me wish it was two different codecs!)

WavPack 4.75 Release Candidate

Reply #16
PDF with data per track can be found here.

I don't understand why on some tracks -h -x encodes with zero speed: (01 - Autobahn and 01 - Part One).

I missed that... there must be some bug in my scripts causing it. I haven't got a clue as to why this happens.

I realize that these are your charts and I would not want to be accused of suggesting things to make WavPack look better than it is, but I have to admit that every time I look at them I get very confused trying to figure out the WavPack results (it makes me wish it was two different codecs!)

Well, actually, I really appreciate feedback like this. I have similar problems with OptimFROG, which has way too many switches to test. (would it be better to use a higher base mode, or use one step more optimization? Difficult decisions  ) Likewise, I've always wondered which x setting(s) I should test for WavPack. I'll stick with x4 then.

About having two lines, I'll keep it in mind. I agree that the charts aren't really clear this way.
Music: sounds arranged such that they construct feelings.

WavPack 4.75 Release Candidate

Reply #17
Changing the terminal title on Unix is considered intrusive and should be left to a very few special programs like SSH that actually change the state of the terminal session, and need to reflect that in the title.

Thanks for the heads-up on this. I really like the console status because I can easily see when my WavPack operations are done by just glancing at the panel, but I never could figure out how to portably restore the previous title (and I don't think I'm alone because Vim leaves that "Thanks for flying Vim"). Anyway, I've changed the console title update to disabled by default on Linux, but it can be enabled with "-z1" (and -z still leaves it off).

Thank you, this would allow having a sane (not too surprising) behavior by default and keeping compatibility with the 4.70 -z switch.

Also I'm surprised because I have never seen vim change my terminal title.
Opus 96 kb/s (Android) / Vorbis -q5 (PC) / WavPack -hhx6m (Archive)

WavPack 4.75 Release Candidate

Reply #18
I realize that FLAC != WavPack, but the FLAC 1.3.1 thread seemed to indicate that the official GCC compile beat both MSVC and ICL compiles, at least for recent processors.  It's possible GCC has just gotten that good.

On my computer GCC 4.9.2 compiles are slightly faster than MSVC 2013.

Thanks for the info. I may consider going to this for my releases in the future. And the binaries are significantly smaller too!

Well, actually, I really appreciate feedback like this. I have similar problems with OptimFROG, which has way too many switches to test. (would it be better to use a higher base mode, or use one step more optimization? Difficult decisions  ) Likewise, I've always wondered which x setting(s) I should test for WavPack. I'll stick with x4 then.

About having two lines, I'll keep it in mind. I agree that the charts aren't really clear this way.

Great, thanks!

Also I'm surprised because I have never seen vim change my terminal title.

They changed the default too, at version 4.0!

You can use ":set title" to see it now, although somehow I don't think you'll be leaving it on... 

WavPack 4.75 Release Candidate

Reply #19
Using the 64-bit executable, I've encoded a 48kHz 24-bit album with -h -x4. Turns out there is a difference between the new and old build for me as well:
Code: [Select]
wv old: 4:52  / 919MB
wv new: 4:43 / 919MB
tak:    0:55 / 899MB
flac:   0:26 / 921MB

tak -p4m and flac -8 for comparison. This is with 4x threads using foobar2000, encoding from wav on a ramdisk.

Going -x6 slows things down considerably for almost no gain, and takes about the same time to encode.

Code: [Select]
wv old: 9:34 / 918MB
wv new: 9:27 / 918MB

WavPack 4.75 Release Candidate

Reply #20
Using the 64-bit executable, I've encoded a 48kHz 24-bit album with -h -x4. Turns out there is a difference between the new and old build for me as well:

Thanks for sharing your results. This is not the kind of improvement for this mode that I have been seeing, which is more like 30% - 50% faster for the new version (Intel i7 and Core2 Duo). I have an old Turion64 laptop that shows a similar [lack of] improvement to what you're seeing, but I chalked that up to something else being a bottleneck (like memory bus). Perhaps it's something I'm doing that AMD just doesn't like... 

WavPack 4.75 Release Candidate

Reply #21
Or maybe the msvc compiler just hates AMD CPUs. The same album encoded on my Core2Duo takes 14m54.729s with the old build and 8m40.118s with the new on linux. That's a nice improvement right there.
Just to make sure, I also built it both the old and the new build with march=native on my AMD Phenom II X4 CPU on linux and it takes 4m38.738s with the old build and 3m9.274s with the new!

This is with caudec at 2 and 4 threads respectively.

Code: [Select]
wv old msvc: 4:52
wv old unix: 4:38
wv new msvc: 4:43
wv new unix: 3:09
tak:         0:55
flac:        0:26

This is some nice improvement. Now to get that on the windows side....

edit: question about linux compile, am I getting all optimizations by simply running ./configure? Do I need any special tools for asm optimizations to work?

WavPack 4.75 Release Candidate

Reply #22
That would make some sense, but almost all the CPU time for an -hx4 operation is spent in 2 functions (log2buffer() and pack_decorr_stereo_pass_cont()) and I rewrote both those in assembly language, so the compiler should not make a difference here (I am assuming that my assembly is not being optimized).

It would be interesting to try the gcc compiles that lvqcl created up in post #8. Maybe those will show the same improvement on Windows that your gcc compiles show in Linux.

edit: question about linux compile, am I getting all optimizations by simply running ./configure? Do I need any special tools for asm optimizations to work?

Yes, you get the optimizations by default (assuming the script is working). I changed the default so now --disable-asm forces the pure C code. You should see this near the bottom of the configure output:

Code: [Select]
checking if assembly optimizations are available... x64

Thanks again for trying this out!

WavPack 4.75 Release Candidate

Reply #23
It would be interesting to try the gcc compiles that lvqcl created up in post #8. Maybe those will show the same improvement on Windows that your gcc compiles show in Linux.
The gcc compile by lvqcl is as fast as the msvc for me =/

I made my own using msys2, march=native again, and it does not make any difference. Maybe linux is just more efficient at file operations, or caudec is more efficient than foobar at multithreading. Both encode from and to a ramdisk though, so I'd expect them to be on par.

Quote
Code: [Select]
checking if assembly optimizations are available... x64

Thanks again for trying this out!
Yes, I saw this line. I guess the arch AUR git package should be working correctly then. And you're welcome.

edit: I just noticed that my msys2 compile reports itself as ver 4.70 even though the code was checked out from github and the compilation process shows stuff like
Code: [Select]
-DPACKAGE_TARNAME=\"wavpack\" -DPACKAGE_VERSION=\"4.75.0-rc\" -DPACKAGE_STRING=\"wavpack\ 4.75.0-rc\"

WavPack 4.75 Release Candidate

Reply #24
I made my own using msys2, march=native again, and it does not make any difference. Maybe linux is just more efficient at file operations, or caudec is more efficient than foobar at multithreading. Both encode from and to a ramdisk though, so I'd expect them to be on par.

It's not obvious to me what could be going on, but there are a lot of moving parts here. The fact that my optimizations improve the speed on your CPU in some circumstances by over 50% will have to be enough for me right now. The bar that I set for myself before was that I didn't want situations where the new code was worse than before, and we've still met that goal.

edit: I just noticed that my msys2 compile reports itself as ver 4.70 even though the code was checked out from github and the compilation process shows stuff like
Code: [Select]
-DPACKAGE_TARNAME=\"wavpack\" -DPACKAGE_VERSION=\"4.75.0-rc\" -DPACKAGE_STRING=\"wavpack\ 4.75.0-rc\"

Are you installing the package after you build it? That can be a source of confusion because when you perform an install, not all the system pointers to the libraries get updated and you can run into this until a reboot or ldconfig.

I usually just run the programs directly in the cli directory (and never install except for a final release). The build will create a tiny script there for each command-line program that runs the executable linked to the correct libwavpack. The cleanest way is to add --disable-shared on the configure line and then when you build it will create stand-alone binaries in the cli directory that have libwavpack built in (so there's no doubt about what's what). I would not install after building like that though.

Thanks again for all your help!