I have been working on a new release of WavPack for a few months, and I am ready with a release candidate. There are no new features, and in fact the new code should generate the same binary output as 4.70, however there have been many internal changes and optimizations.
The biggest change is that I took out the MMX intrinsics and replaced them with assembly (although still using MMX instructions when they were applicable). This means the optimizations have been extended to the decoding side and to mono files (previously they were only for stereo encoding), and I can now provide 64-bit executables with the optimizations (the Visual Studio compiler dropped support for MMX intrinsics when they went to 64-bit). Basically there is a nice speed boost for most operations on most processors (although the “higher” modes and 24-bit audio benefit more).
The code has also been reorganized to be more modular, which results in smaller images when linking to projects that do not require all the functionality, and it's also easier to understand. On Linux (and related systems) the configure script chooses the appropriate assembler optimizations based on the host processor (x86, x64, or ARM) unless explicitly disabled with --disable-asm.
Any testing or feedback is greatly appreciated; I hope there are still some WavPack users around! With any luck I should be ready for a full release in a week or so.
32-bit Windows Executables (http://www.wavpack.com/wavpack-4.75.0-rc.zip)
64-bit Windows Executables (http://www.wavpack.com/wavpack-4.75.0-rc-x64.zip)
Linux Distribution (http://www.wavpack.com/wavpack-4.75.0-rc.tar.bz2)
Thanks!
--David
I hope there are still some WavPack users around!
I feel guilty if I don't post a reply after seeing this. I've been using wavpack since 2007, and love how it handles floating point and non-standard sample rate files nicely. I didn't use flac until I know about flaccl.
Just tried 64-bit wavpack.exe as foobar's encoding plugin. Couldn't see encoding speed improvements, may be my harddisk is the bottleneck. It produced identical files as the (current?) version dated in 14/12/2014 so I think I don't need to worry about compatibility.
32-bit wavpack crashes here (as well as wvunpack). 64-bit seems to be fine.
Unhandled exception at 0x00442485 in wavpack.exe: 0xC0000005: Access violation executing location 0x00442485.
[disassembly removed as useless]
Looks like the beginning of pack_decorr_stereo_pass_cont_rev_x86. Also decoding crashes at the beginning of unpack_decorr_stereo_pass_cont_x86.
Added: OS is Win7SP1 x64. No problems in VM with WinXPSP3.
Added2: the problem disappear if I compress wavpack.exe with UPX. Strange problem indeed.
Added3: found the problem, see post #9.
Not really seeing any speed improvements (win x64), I'm guessing my Phenom II does not fall under "most processors" ^^;
Also, I downloaded wavpack-4.75.0-rc.tar.bz2 and tried to build it under MSYS2+MinGW from http://xhmikosr.1f0.de/tools/msys/ (http://xhmikosr.1f0.de/tools/msys/) (I typed ./configure, then make)
make fails with the following error:
.libs/libwavpack_la-extra1.o: In function `recurse_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:228: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `sort_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:362: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `analyze_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:416: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `delta_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:293: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o: In function `analyze_mono':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:421: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra1.o:E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra1.c:318: more undefined references to `log2buffer_x86' follow
.libs/libwavpack_la-extra2.o: In function `decorr_stereo_pass':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:266: undefined reference to `pack_decorr_stereo_pass_cont_x86'
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:264: undefined reference to `pack_decorr_stereo_pass_cont_rev_x86'
.libs/libwavpack_la-extra2.o: In function `recurse_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:414: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `sort_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:549: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `analyze_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:603: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `delta_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:480: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o: In function `analyze_stereo':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:608: undefined reference to `log2buffer_x86'
.libs/libwavpack_la-extra2.o:E:\MSYS\home\user\wavpack-4.75.0-rc\src/extra2.c:505: more undefined references to `log2buffer_x86' follow
.libs/libwavpack_la-pack.o: In function `pack_samples':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/pack.c:976: undefined reference to `pack_decorr_stereo_pass_x86'
E:\MSYS\home\user\wavpack-4.75.0-rc\src/pack.c:957: undefined reference to `pack_decorr_mono_buffer_x86'
.libs/libwavpack_la-unpack.o: In function `unpack_samples':
E:\MSYS\home\user\wavpack-4.75.0-rc\src/unpack.c:122: undefined reference to `unpack_decorr_mono_pass_cont_x86'
E:\MSYS\home\user\wavpack-4.75.0-rc\src/unpack.c:180: undefined reference to `unpack_decorr_stereo_pass_cont_x86'
collect2.exe: error: ld returned 1 exit status
make[1]: *** [libwavpack.la] Error 1
make[1]: Leaving directory `/home/user/wavpack-4.75.0-rc/src'
make: *** [all-recursive] Error 1
Added: I managed to compile it when I added
leading underscores to all .globl symbols in pack_x86.s and unpack_x86.s (log2buffer_x86 -> _log2buffer_x86, etc). It seems that underscores are necessary for Win32 build.
Thanks David. This one appears 3 .. 4 % faster than 4.60.1 [default mode - win 7 - 32bit]
Here's a detailed comparison between 4.70 and 4.75rc on the test rig I use for lossless audio codec comparison (http://www.audiograaf.nl/downloads.html) I publish. Data was acquired with a AMD A4-3400 running Windows 7 with 4GB of RAM. Files were loaded and processed in a RAMdisk. CPU frequency was locked to the maximum. The encoding swiches tested are, in order, -f, [none], -h, -h -x, -hh, -hh -x.
(http://www.audiograaf.nl/misc_stuff/wavpack-4.75rc-enc.png)
(http://www.audiograaf.nl/misc_stuff/wavpack-4.75rc-dec.png)
PDF with data per track can be found here (http://www.audiograaf.nl/misc_stuff/wavpack-4.75rc-per-track.pdf).
The results indeed seem to indicate a speed boost dependent of encoding mode. The speed boost is indeed bigger for 24-bit files (see the track-01-01[1]-03-[Lf-Rf-C-LFE-Ls-Rs]-24-48000 file). There is one rather strange case in the per-track PDF, ZOOM0004, which is a extremely quiet recording of a acoustic guitar. For some reason, -h compressed less than the normal mode and the fast mode, and -hh does even worse. Maybe you want to look into that?
To whom it may concern: WavPack 4.75rc compiled with GCC 4.9.2. It should be a bit faster than MSVC.
System Requirements: at least i686 CPU. Also I'm not sure that MinGW-w64 supports Win2000 (should work on WinXP without problems).
I ran "dumpbin /headers wavpack.exe" and noticed that asmcode section in 32-bit executables has flags=0xC0000040 (Initialized Data, Read, Write) and it probably should have flags=0x60000020 (Code, Execute, Read).
(IOW wavpack crashes because it tries to execute code from data section)
So, TWIMC-2: 32-bit executables from bryant fixed with "editbin /section:asmcode,ce!i!w *.exe".
hi, I will do some testing when I get home, just wondering if the 4gb+ support is in this version (http://www.hydrogenaud.io/forums/index.php?showtopic=108891)
many thanks
Garry
...
Any testing or feedback is greatly appreciated; I hope there are still some WavPack users around!
...
Hi,
I am a happy WavPack user for several years, and thanks for updating it, but please consider reverting this change in 4.70 :
added: several Windows features to Linux (clean ^C handling, console title)
On Ubuntu, all terminals that start a wavpack process without the -z switch are left with a "WavPack completed" title even after the process has died.
Changing the terminal title on Unix is considered intrusive and should be left to a very few special programs like SSH that actually change the state of the terminal session, and need to reflect that in the title.
Thanks so much to everyone for the feedback and testing; it is greatly appreciated!!
I have made a few updates at GitHub (https://github.com/dbry/WavPack/commits/master) based on these findings and comments. Unless something else comes up, I don't think another release candidate will be needed.
The results indeed seem to indicate a speed boost dependent of encoding mode. The speed boost is indeed bigger for 24-bit files (see the track-01-01[1]-03-[Lf-Rf-C-LFE-Ls-Rs]-24-48000 file). There is one rather strange case in the per-track PDF, ZOOM0004, which is a extremely quiet recording of a acoustic guitar. For some reason, -h compressed less than the normal mode and the fast mode, and -hh does even worse. Maybe you want to look into that?
Thanks for the exhaustive testing (I have looked over your compilations with interest in the past). Yes, I understand why some very quiet tracks might compress better in faster modes. The problem is that the block headers for the higher modes get bigger and bigger. Normally the improved compression swamps these few extra bytes, but I guess in some cases it doesn't (although I think I've only seen this in experimental files). Unfortunately there's not a lot I can do that wouldn't spend extra time (I'd have to try re-encoding blocks that seemed to show this). Fortunately, I don't think this is very common...
To whom it may concern: WavPack 4.75rc compiled with GCC 4.9.2. It should be a bit faster than MSVC.
Thanks for creating these, and they do seem somewhat faster than the MSVC versions, especially the x64 version (and packing vs. unpacking). I had actually noticed this myself because my Linux versions and my Cygwin64 versions on Windows both run faster than the MSVC versions. Do you happen to know if this is something that will be better after an upgrade to Visual Studio 2013?
Thanks for the other catches too!
hi, I will do some testing when I get home, just wondering if the 4gb+ support is in this version (http://www.hydrogenaud.io/forums/index.php?showtopic=108891)
Sorry, the 4GB stuff is for the next release. I wanted to isolate these speed improvements and reorganization from the slew of new features coming...
Changing the terminal title on Unix is considered intrusive and should be left to a very few special programs like SSH that actually change the state of the terminal session, and need to reflect that in the title.
Thanks for the heads-up on this. I really like the console status because I can easily see when my WavPack operations are done by just glancing at the panel, but I never could figure out how to portably restore the previous title (and I don't think I'm alone because Vim leaves that "Thanks for flying Vim"). Anyway, I've changed the console title update to disabled by default on Linux, but it can be enabled with "-z1" (and -z still leaves it off).
Thanks for creating these, and they do seem somewhat faster than the MSVC versions, especially the x64 version (and packing vs. unpacking). I had actually noticed this myself because my Linux versions and my Cygwin64 versions on Windows both run faster than the MSVC versions. Do you happen to know if this is something that will be better after an upgrade to Visual Studio 2013?
I realize that FLAC != WavPack, but the FLAC 1.3.1 (http://www.hydrogenaud.io/forums/index.php?showtopic=107611&st=25) thread seemed to indicate that the official GCC compile beat both MSVC
and ICL compiles, at least for recent processors. It's possible GCC has just gotten that good.
Here's a detailed comparison between 4.70 and 4.75rc on the test rig I use for lossless audio codec comparison (http://www.audiograaf.nl/downloads.html) I publish. Data was acquired with a AMD A4-3400 running Windows 7 with 4GB of RAM. Files were loaded and processed in a RAMdisk. CPU frequency was locked to the maximum. The encoding swiches tested are, in order, -f, [none], -h, -h -x, -hh, -hh -x.
on my CPU (Intel Core2) encoding with -x4 becomes 1.4x faster. Maybe it makes sense to test -hh -x4? It's very slow, but usually there's a noticeable increase in compression ratio between -x3 and -x4.
PDF with data per track can be found here (http://www.audiograaf.nl/misc_stuff/wavpack-4.75rc-per-track.pdf).
I don't understand why on some tracks -h -x encodes with zero speed: (01 - Autobahn and 01 - Part One).
Thanks for creating these, and they do seem somewhat faster than the MSVC versions, especially the x64 version (and packing vs. unpacking). I had actually noticed this myself because my Linux versions and my Cygwin64 versions on Windows both run faster than the MSVC versions. Do you happen to know if this is something that will be better after an upgrade to Visual Studio 2013?
On my computer GCC 4.9.2 compiles are slightly faster than MSVC 2013.
Here's a detailed comparison between 4.70 and 4.75rc on the test rig I use for lossless audio codec comparison (http://www.audiograaf.nl/downloads.html) I publish. Data was acquired with a AMD A4-3400 running Windows 7 with 4GB of RAM. Files were loaded and processed in a RAMdisk. CPU frequency was locked to the maximum. The encoding swiches tested are, in order, -f, [none], -h, -h -x, -hh, -hh -x.
on my CPU (Intel Core2) encoding with -x4 becomes 1.4x faster. Maybe it makes sense to test -hh -x4? It's very slow, but usually there's a noticeable increase in compression ratio between -x3 and -x4.
This is actually something that has always made the WavPack results a little difficult for me to understand: the mixing of -x modes with the non-x modes on the same line. I know other codecs work this way to some extent (and none have more than one line), but WavPack is sort of two codecs with two different purposes.
For realtime encoding (like with Reaper), the -x switches makes no sense because they slow encoding down a lot with almost no gain in compression. On a compression vs. encoding speed chart they look silly. However, since the -x switches shouldn't affect decoding speed at all, they make perfect sense for archiving of material that will be decoded many times (like a new CD).
In a perfect world, I think that the only scenarios that make sense are the four basic modes without -x, and the four basic mode with -x6. Of course, if someone is ripping a lot of CDs then that might be prohibitive if it ended up taking longer than the ripping, and a test like yours might stretch to a long time (not considering your electricity bill!) but in the latest version I can encode an average CD in 5 minutes with -hx4, and that's with just one core.
I guess what I'm saying is that it might make your charts clearer to either just do the four basic modes without -x, or, if you want to include the -x switch in a comprehensible way, make a separate line containing the four basic modes with some constant -x mode, like -x4 (which lvqcl suggests is a good compromise).
That way, on the compression vs. encoding chart the two lines would be completely separated (and WavPack -xn line would not fare too well, obviously), but at least someone could look at the compression vs. decoding chart and see that extra bump that the -x4 (or whatever) gives for each mode.
I realize that these are your charts and I would not want to be accused of suggesting things to make WavPack look better than it is, but I have to admit that every time I look at them I get very confused trying to figure out the WavPack results (it makes me wish it
was two different codecs!)
PDF with data per track can be found here (http://www.audiograaf.nl/misc_stuff/wavpack-4.75rc-per-track.pdf).
I don't understand why on some tracks -h -x encodes with zero speed: (01 - Autobahn and 01 - Part One).
I missed that... there must be some bug in my scripts causing it. I haven't got a clue as to why this happens.
I realize that these are your charts and I would not want to be accused of suggesting things to make WavPack look better than it is, but I have to admit that every time I look at them I get very confused trying to figure out the WavPack results (it makes me wish it was two different codecs!)
Well, actually, I really appreciate feedback like this. I have similar problems with OptimFROG, which has way too many switches to test. (would it be better to use a higher base mode, or use one step more optimization? Difficult decisions ) Likewise, I've always wondered which x setting(s) I should test for WavPack. I'll stick with x4 then.
About having two lines, I'll keep it in mind. I agree that the charts aren't really clear this way.
Changing the terminal title on Unix is considered intrusive and should be left to a very few special programs like SSH that actually change the state of the terminal session, and need to reflect that in the title.
Thanks for the heads-up on this. I really like the console status because I can easily see when my WavPack operations are done by just glancing at the panel, but I never could figure out how to portably restore the previous title (and I don't think I'm alone because Vim leaves that "Thanks for flying Vim"). Anyway, I've changed the console title update to disabled by default on Linux, but it can be enabled with "-z1" (and -z still leaves it off).
Thank you, this would allow having a sane (not too surprising) behavior by default and keeping compatibility with the 4.70 -z switch.
Also I'm surprised because I have never seen vim change my terminal title.
I realize that FLAC != WavPack, but the FLAC 1.3.1 (http://www.hydrogenaud.io/forums/index.php?showtopic=107611&st=25) thread seemed to indicate that the official GCC compile beat both MSVC and ICL compiles, at least for recent processors. It's possible GCC has just gotten that good.
On my computer GCC 4.9.2 compiles are slightly faster than MSVC 2013.
Thanks for the info. I may consider going to this for my releases in the future. And the binaries are significantly smaller too!
Well, actually, I really appreciate feedback like this. I have similar problems with OptimFROG, which has way too many switches to test. (would it be better to use a higher base mode, or use one step more optimization? Difficult decisions ) Likewise, I've always wondered which x setting(s) I should test for WavPack. I'll stick with x4 then.
About having two lines, I'll keep it in mind. I agree that the charts aren't really clear this way.
Great, thanks!
Also I'm surprised because I have never seen vim change my terminal title.
They changed the default too, at version 4.0!
You can use ":set title" to see it now, although somehow I don't think you'll be leaving it on...
Using the 64-bit executable, I've encoded a 48kHz 24-bit album with -h -x4. Turns out there is a difference between the new and old build for me as well:
wv old: 4:52 / 919MB
wv new: 4:43 / 919MB
tak: 0:55 / 899MB
flac: 0:26 / 921MB
tak -p4m and flac -8 for comparison. This is with 4x threads using foobar2000, encoding from wav on a ramdisk.
Going -x6 slows things down considerably for almost no gain, and takes about the same time to encode.
wv old: 9:34 / 918MB
wv new: 9:27 / 918MB
Using the 64-bit executable, I've encoded a 48kHz 24-bit album with -h -x4. Turns out there is a difference between the new and old build for me as well:
Thanks for sharing your results. This is not the kind of improvement for this mode that I have been seeing, which is more like 30% - 50% faster for the new version (Intel i7 and Core2 Duo). I have an old Turion64 laptop that shows a similar [lack of] improvement to what you're seeing, but I chalked that up to something else being a bottleneck (like memory bus). Perhaps it's something I'm doing that AMD just doesn't like...
Or maybe the msvc compiler just hates AMD CPUs. The same album encoded on my Core2Duo takes 14m54.729s with the old build and 8m40.118s with the new on linux. That's a nice improvement right there.
Just to make sure, I also built it both the old and the new build with march=native on my AMD Phenom II X4 CPU on linux and it takes 4m38.738s with the old build and 3m9.274s with the new!
This is with caudec at 2 and 4 threads respectively.
wv old msvc: 4:52
wv old unix: 4:38
wv new msvc: 4:43
wv new unix: 3:09
tak: 0:55
flac: 0:26
This
is some nice improvement. Now to get that on the windows side....
edit: question about linux compile, am I getting all optimizations by simply running ./configure? Do I need any special tools for asm optimizations to work?
That would make some sense, but almost all the CPU time for an -hx4 operation is spent in 2 functions (log2buffer() and pack_decorr_stereo_pass_cont()) and I rewrote
both those in assembly language, so the compiler should not make a difference here (I am assuming that my assembly is not being optimized).
It would be interesting to try the gcc compiles that lvqcl created up in post #8. Maybe those will show the same improvement on Windows that your gcc compiles show in Linux.
edit: question about linux compile, am I getting all optimizations by simply running ./configure? Do I need any special tools for asm optimizations to work?
Yes, you get the optimizations by default (assuming the script is working). I changed the default so now --disable-asm forces the pure C code. You should see this near the bottom of the configure output:
checking if assembly optimizations are available... x64
Thanks again for trying this out!
It would be interesting to try the gcc compiles that lvqcl created up in post #8. Maybe those will show the same improvement on Windows that your gcc compiles show in Linux.
The gcc compile by lvqcl is as fast as the msvc for me =/
I made my own using msys2, march=native again, and it does not make any difference. Maybe linux is just more efficient at file operations, or caudec is more efficient than foobar at multithreading. Both encode from and to a ramdisk though, so I'd expect them to be on par.
checking if assembly optimizations are available... x64
Thanks again for trying this out!
Yes, I saw this line. I guess the arch AUR git package should be working correctly then. And you're welcome.
edit: I just noticed that my msys2 compile reports itself as ver 4.70 even though the code was checked out from github and the compilation process shows stuff like
-DPACKAGE_TARNAME=\"wavpack\" -DPACKAGE_VERSION=\"4.75.0-rc\" -DPACKAGE_STRING=\"wavpack\ 4.75.0-rc\"
I made my own using msys2, march=native again, and it does not make any difference. Maybe linux is just more efficient at file operations, or caudec is more efficient than foobar at multithreading. Both encode from and to a ramdisk though, so I'd expect them to be on par.
It's not obvious to me what could be going on, but there are a lot of moving parts here. The fact that my optimizations improve the speed on your CPU
in some circumstances by over 50% will have to be enough for me right now. The bar that I set for myself before was that I didn't want situations where the new code was
worse than before, and we've still met that goal.
edit: I just noticed that my msys2 compile reports itself as ver 4.70 even though the code was checked out from github and the compilation process shows stuff like
-DPACKAGE_TARNAME=\"wavpack\" -DPACKAGE_VERSION=\"4.75.0-rc\" -DPACKAGE_STRING=\"wavpack\ 4.75.0-rc\"
Are you installing the package after you build it? That can be a source of confusion because when you perform an install, not all the system pointers to the libraries get updated and you can run into this until a reboot or ldconfig.
I usually just run the programs directly in the cli directory (and never install except for a final release). The build will create a tiny script there for each command-line program that runs the executable linked to the correct libwavpack. The cleanest way is to add --disable-shared on the configure line and then when you build it will create stand-alone binaries in the cli directory that have libwavpack built in (so there's no doubt about what's what). I would not install after building like that though.
Thanks again for all your help!
The fact that my optimizations improve the speed on your CPU in some circumstances by over 50% will have to be enough for me right now. The bar that I set for myself before was that I didn't want situations where the new code was worse than before, and we've still met that goal.
Well, one of my CPUs anyway
But yes, it's always good to see some improvements, else all that work would have been for nothing...
I usually just run the programs directly in the cli directory (and never install except for a final release). The build will create a tiny script there for each command-line program that runs the executable linked to the correct libwavpack. The cleanest way is to add --disable-shared on the configure line and then when you build it will create stand-alone binaries in the cli directory that have libwavpack built in (so there's no doubt about what's what). I would not install after building like that though.
I did not install it after building it, but I'm running the cli\.libs\wavpack.exe, because the cli\wavpack.exe is only 36kb small and doesn't seem to do anything.
I've rebuilt wavpack with --disable-shared and now it reports as 4.75-rc correctly, thanks.
but I'm running the cli\.libs\wavpack.exe, because the cli\wavpack.exe is only 36kb small and doesn't seem to do anything.
That one is, at least on my setup, using the old libs, which would indeed explain why you were seeing 4.70 as the version number.
You can use ldd <executable name> to check which libraries are being used.
but I'm running the cli\.libs\wavpack.exe, because the cli\wavpack.exe is only 36kb small and doesn't seem to do anything.
That one is, at least on my setup, using the old libs, which would indeed explain why you were seeing 4.70 as the version number.
You can use ldd <executable name> to check which libraries are being used.
If one desires to not load a library from system paths. LD_LIBRARY_PATH/LD_PRELOAD environment variables can be used at execution time. Or -rpath can be used at link time.
The manual pages
ld.so and
ld have all the details.
Related to the recent posts above, I'm trying to get out this latest version of WavPack, but I'm running into a problem / confusion with the libraries. WavPack 4.75.0 installs and works fine on my main system (64-bit LinuxMint Qiana), but it does not install correctly on a particular test system I have (32-bit LinuxMint Qiana).
On the failing system I had no problem compiling and running the new version directly at cli/wavpack whether I built with --disable-shared or not. I could still run the old 4.70.0 version (which was installed by the package manager) simply by typing “wavpack” (/usr/bin/wavpack).
However, when I tried installing the 4.75.0 package (and doing ldconfig) I ran into trouble. When I typed “wavpack” it would pick up the new binary (/usr/local/bin/wavpack) but would use the old 4.70.0 libwavpack (/usr/lib/i386-linux-gnu/libwavpack.so.1). I could force the correct library by using “LD_LIBRARY_PATH=/usr/local/lib wavpack”, but obviously that's not an acceptable solution for a distribution.
The new library is definitely in the library cache, but it looks like ld.so is simply using the first library that matches the request, even though I remember reading somewhere that it is supposed to favor newer revisions of the same library. The search order seems to be (based on ldconfig -v output):
/usr/lib/i386-linux-gnu/
/usr/local/lib
/usr/lib/x86_64-linux-gnu/
So, when it looks for the 64-bit library it finds the one in /usr/local/lib first, but if it's looking for the 32-bit version it never gets to /usr/local/lib and uses the old library. To verify this I tried building and installing a 32-bit version of 4.75.0 on my 64-bit system and sure enough, it used the old library! This seems really broken to me.
My first idea to fix this was to bump the libtool versioning so that (I thought) my command-line programs would require the newer library (even though there was really no missing dependency in the old library). I was surprised when this did not work. I even added a new function to the library and now I get a runtime error when I try to run the programs, because it still won't use the new library. So either that feature, or my understanding of it, is broken.
I have now found two fixes that do work, but I'm not really sure either are good ideas (especially from a packager's viewpoint). The first is to put AM_DISABLE_SHARED in my configure.ac to simply force static linking. This works great, but seems like a pretty blunt solution.
The second solution I came up with was to add lines like this to the Makefile.am for the command-line programs:
wavpack_LDFLAGS = -rpath $(libdir)
This hardcodes “/usr/local/lib” into the binaries so that they'll try that directory first for the library, and this also works fine on all the systems I tried it on, but again I'm not sure how this will look from a packager's perspective.
Sorry for the longwinded post, but I've been working on this for several days and I wanted to provide as much information as I could. Hopefully there's a Linux guru in the house that can help (I'm also posting this to the WavPack devel mailing list). Any comments or insights or opinions on any of this are gladly welcome and appreciated!
Well I'm not a Linux guru, but my system (Debian, 32bit) works as you say.
My /etc/ld.so.conf is like this:
include /etc/ld.so.conf.d/*.conf
And inside of /etc/ld.so.conf.d:
fakeroot-i386-linux-gnu.conf i386-linux-gnu.conf libc.conf
So, they are loaded in this
filename order. /usr/local/lib is written in libc.conf, so it is loaded after i386-linux-gnu.conf. If x86_64-linux-gnu.conf is present, it will be loaded at the end. Hence the strange, seemingly inconsistent library search order. I can change library search order simply by renaming them (I confirmed it).
As for rpath thing, I don't think setting rpath in build process is common. However, I found this option in configure script of ffmpeg:
--enable-rpath use rpath to allow installing libraries in paths
use rpath when linking programs [USE WITH CARE]
Maybe you could offer similar option and let user decide. However, I don't think users blame you without it.
Great to see a new Wavpack release.
For the foreseeable future, I picture myself using it more and more, as Android devices abound.
/usr and
/usr/local are different prefixes.
Are you sure you are running the right executable?
My first idea to fix this was to bump the libtool versioning so that (I thought) my command-line programs would require the newer library (even though there was really no missing dependency in the old library). I was surprised when this did not work. I even added a new function to the library and now I get a runtime error when I try to run the programs, because it still won't use the new library. So either that feature, or my understanding of it, is broken.
What's relevant here is the soname.
readelf -d /usr/local/lib/libwavpack.so | grep SONAME
readelf -d `which wavpack` | grep NEEDED
If you did actually bump the soname version to 2. And you are running the right binary. Then both greps should include
libwavpack.so.2.
So, they are loaded in this filename order. /usr/local/lib is written in libc.conf, so it is loaded after i386-linux-gnu.conf. If x86_64-linux-gnu.conf is present, it will be loaded at the end. Hence the strange, seemingly inconsistent library search order. I can change library search order simply by renaming them (I confirmed it).
Thanks for taking the time to confirm this; I'm glad I'm not imagining things!
As for rpath thing, I don't think setting rpath in build process is common. However, I found this option in configure script of ffmpeg:
Maybe you could offer similar option and let user decide. However, I don't think users blame you without it.
Yeah, all you have to do is Google
rpath to see that it's not held in high esteem, but I think I'll copy ffmpeg and make it an option (see below)
/usr and /usr/local are different prefixes.
Are you sure you are running the right executable?
Yes, I'm sure I'm running the right ones because I have an option
--version that displays both the executable version and the linked library version.
What's relevant here is the soname.
readelf -d /usr/local/lib/libwavpack.so | grep SONAME
readelf -d `which wavpack` | grep NEEDED
If you did actually bump the soname version to 2. And you are running the right binary. Then both greps should include libwavpack.so.2.
Thanks for those shell lines; that saved some time, and led me to the eventual answer. It turns out that my problems seem to mostly stem from having read the
libtool manual. I had this idea that
ld.so was the
dynamic linker being referred to in there, but I have since discovered that it isn't. My first clue was when it said:
If two libraries have identical current and age numbers, then the dynamic linker chooses the library with the greater revision number.
I showed previously that this is definitely not true, or at least only true with the caveat that the library with the greatest
revision number must be the first encountered. Then I realized that the
current and
age stuff is not implemented in
ld.so either, so basically the library must simply exactly match the version requested. So, yes, I can simply bump the
soname version to 2 (and I verified this), but then nothing else will use the library. I didn't make it incompatible; I made it better!
This page (https://autotools.io/libtool/version.html) was particularly illuminating:
To set the version of the library, libtool provides the -version-info parameter, which accepts three numbers, separated by colons, that are called respectively, current, revision and age. Both their name and their behaviour, nowadays, have to be considered fully arbitrary, as the explanation provided in the official documentation is confusing to say the least, and can be, in some cases, considered completely wrong.
At this point I'm leaning toward having the executables check the libwavpack version when they start and display a warning if it's wrong and referring the user to the README where I will suggest trying
--enable-rpath, which I am also adding. I really hate that i386 users will have to go through all this trouble, but I am afraid of making rpath (or static) the default.
Great to see a new Wavpack release.
For the foreseeable future, I picture myself using it more and more, as Android devices abound.
Cool, glad to hear it!
I'm glad you found a solution.
FWIW, build-time linking and execution-time linking was always a source of confusion for some. That's why ld.so is usually referred to as a dynamic linker/loader or just a loader. Where ld is referred to as the dynamic linker.
I barely remember about these things, but, wasn't there an "alias" mechanism for this? (not sure if it was the alternatives system or something else). Basically, the true libraries are the .1, .2, etc... but then there's a non-number file that links to one of them which is what the programs are linked to.
Maybe this has been superseeded, but nevertheless, it seems that your main problem is that you are manually installing a library that is also installed by the OS package manager. make install is not the same than apt-get install, you know.
CPU: Core i7 2630QM
Norah Jones - 12. Nightingale [4:12] (
24-bit 6ch WAV) -h -x2 option:
X | run 1 | run 2 |
icc-mmx 4.70.0 64 | 29.46s | 29.32s |
4.75.0-rc 64 | 34.29s | 34.09s |
gcc 4.75.0-rc 64 | 31.01s | 30.75s |
Norah Jones - 12. Nightingale [4:12] (
16-bit 6ch WAV) -h -x2 option:
X | run 1 | run 2 |
icc-mmx 4.70.0 64 | 30.35s | 30.34s |
4.75.0-rc 64 | 34.04s | 33.94s |
gcc 4.75.0-rc 64 | 30.82s | 30.88s |
It's slightly faster than the standard 4.70.0 binary, but still not as fast as the 4.70.0 64bit ICC binary (http://www.hydrogenaud.io/forums/index.php?showtopic=103111&view=findpost&p=848375). I'm curious to see how much faster a 64bit ICC compile of 4.75.0 would be.
I'm curious to see how much faster a 64bit ICC compile of 4.75.0 would be.
How about GCC compile from post #8 (http://www.hydrogenaud.io/forums/index.php?s=&showtopic=109098&view=findpost&p=897405)?
from audition\cool_wv4.c:
// Version 2.12 - May 10, 2015 (library ver 4.75.1)
Typo?
Oops, yes, thanks!
I'm curious to see how much faster a 64bit ICC compile of 4.75.0 would be.
How about GCC compile from post #8 (http://www.hydrogenaud.io/forums/index.php?s=&showtopic=109098&view=findpost&p=897405)?
Added to the table, it's getting pretty close to ICC! Now I'm even more curious about a 4.75.0 64bit ICC build.
CPU: Core i7 2630QM
Norah Jones - 12. Nightingale [4:12] (24-bit 6ch WAV) -h -x2 option:
X | run 1 | run 2 |
icc-mmx 4.70.0 64 | 29.46s | 29.32s |
4.75.0-rc 64 | 34.29s | 34.09s |
gcc 4.75.0-rc 64 | 31.01s | 30.75s |
I have to admit that I can't really explain your results. I tried to duplicate them using a similar 24-bit file and settings on the most similar CPU I have available (Core i7-4600U) and got these results:
version encode decode
------------------------------
4.70.0 21.53s 7.63s
4.70.0 x64 25.46s 7.30s
------------------------------
4.75.0 19.19s 5.27s
4.75.0 x64 18.39s 5.12s
Of course, the old 64-bit version does not do well because MSVC won't do MMX on 64-bit (which was one of the motivations to switch to assembly). The decode performance improvement is greater because that used to be pure C, and I assume you
do see some improvement there, right?
The only other thing that comes to mind is that you call the file 6ch. I assume that this really is a 5.1 file (FL,FR,FC,LFE,BL,BR), because if it gets treated as 6 independent mono channels then the MMX code can't be used and your results make perfect sense.
A big thank you to everyone for all your testing, help, advice and support. It is greatly appreciated!!
The new released version is currently up on wavpack.com and available for download here (http://www.wavpack.com/downloads.html).
--David
Thanks a bunch David, super glad you're still working on my fav hybrid CODEC !
A big thank you to everyone for all your testing, help, advice and support. It is greatly appreciated!!
The new released version is currently up on wavpack.com and available for download here (http://www.wavpack.com/downloads.html).
--David
It's been said before I know, but it's in times like these I wish a simple 'like' button existed on HA.
Thanks a lot Bryan!
CPU: Core i7 2630QM
Norah Jones - 12. Nightingale [4:12] (24-bit 6ch WAV) -h -x2 option:
X | run 1 | run 2 |
icc-mmx 4.70.0 64 | 29.46s | 29.32s |
4.75.0-rc 64 | 34.29s | 34.09s |
gcc 4.75.0-rc 64 | 31.01s | 30.75s |
I have to admit that I can't really explain your results.
Note that the 4.70 ICC file is not just a recompile. The linked post tells it uses some replacement function.
Thanks David!
Now its time to make some noise so that the new release might show up in distros and whatnots this year....
(FreeBSD still have 4.60.1 in ports... Bah!)
BTW. Should be on HA frontpage maybe....