Skip to main content

Topic: WavPack 4.75 Release Candidate (Read 8488 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
WavPack 4.75 Release Candidate
Reply #25
The fact that my optimizations improve the speed on your CPU in some circumstances by over 50% will have to be enough for me right now. The bar that I set for myself before was that I didn't want situations where the new code was worse than before, and we've still met that goal.
Well, one of my CPUs anyway
But yes, it's always good to see some improvements, else all that work would have been for nothing...

Quote
I usually just run the programs directly in the cli directory (and never install except for a final release). The build will create a tiny script there for each command-line program that runs the executable linked to the correct libwavpack. The cleanest way is to add --disable-shared on the configure line and then when you build it will create stand-alone binaries in the cli directory that have libwavpack built in (so there's no doubt about what's what). I would not install after building like that though.
I did not install it after building it, but I'm running the cli\.libs\wavpack.exe, because the cli\wavpack.exe is only 36kb small and doesn't seem to do anything.
I've rebuilt wavpack with --disable-shared and now it reports as 4.75-rc correctly, thanks.


  • ktf
  • [*][*][*][*][*]
WavPack 4.75 Release Candidate
Reply #26
but I'm running the cli\.libs\wavpack.exe, because the cli\wavpack.exe is only 36kb small and doesn't seem to do anything.

That one is, at least on my setup, using the old libs, which would indeed explain why you were seeing 4.70 as the version number.

You can use ldd <executable name> to check which libraries are being used.
Music: sounds arranged such that they construct feelings.

  • 2012
  • [*][*][*]
WavPack 4.75 Release Candidate
Reply #27
but I'm running the cli\.libs\wavpack.exe, because the cli\wavpack.exe is only 36kb small and doesn't seem to do anything.

That one is, at least on my setup, using the old libs, which would indeed explain why you were seeing 4.70 as the version number.

You can use ldd <executable name> to check which libraries are being used.


If one desires to not load a library from system paths. LD_LIBRARY_PATH/LD_PRELOAD environment variables can be used at execution time. Or -rpath can be used at link time.

The manual pages ld.so and ld have all the details.
saldl: A lightweight well-featured CLI downloader optimized for speed and early preview.
https://saldl.github.io

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.75 Release Candidate
Reply #28
Related to the recent posts above, I'm trying to get out this latest version of WavPack, but I'm running into a problem / confusion with the libraries. WavPack 4.75.0 installs and works fine on my main system (64-bit LinuxMint Qiana), but it does not install correctly on a particular test system I have (32-bit LinuxMint Qiana).

On the failing system I had no problem compiling and running the new version directly at cli/wavpack whether I built with --disable-shared or not. I could still run the old 4.70.0 version (which was installed by the package manager) simply by typing “wavpack” (/usr/bin/wavpack).

However, when I tried installing the 4.75.0 package (and doing ldconfig) I ran into trouble. When I typed “wavpack” it would pick up the new binary (/usr/local/bin/wavpack) but would use the old 4.70.0  libwavpack (/usr/lib/i386-linux-gnu/libwavpack.so.1). I could force the correct library by using “LD_LIBRARY_PATH=/usr/local/lib wavpack”, but obviously that's not an acceptable solution for a distribution.

The new library is definitely in the library cache, but it looks like ld.so is simply using the first library that matches the request, even though I remember reading somewhere that it is supposed to favor newer revisions of the same library. The search order seems to be (based on ldconfig -v output):

/usr/lib/i386-linux-gnu/
/usr/local/lib
/usr/lib/x86_64-linux-gnu/

So, when it looks for the 64-bit library it finds the one in /usr/local/lib first, but if it's looking for the 32-bit version it never gets to /usr/local/lib and uses the old library. To verify this I tried building and installing a 32-bit version of 4.75.0 on my 64-bit system and sure enough, it used the old library! This seems really broken to me.

My first idea to fix this was to bump the libtool versioning so that (I thought) my command-line programs would require the newer library (even though there was really no missing dependency in the old library). I was surprised when this did not work. I even added a new function to the library and now I get a runtime error when I try to run the programs, because it still won't use the new library. So either that feature, or my understanding of it, is broken.

I have now found two fixes that do work, but I'm not really sure either are good ideas (especially from a packager's viewpoint). The first is to put AM_DISABLE_SHARED in my configure.ac to simply force static linking. This works great, but seems like a pretty blunt solution.

The second solution I came up with was to add lines like this to the Makefile.am for the command-line programs:

wavpack_LDFLAGS = -rpath $(libdir)

This hardcodes “/usr/local/lib” into the binaries so that they'll try that directory first for the library, and this also works fine on all the systems I tried it on, but again I'm not sure how this will look from a packager's perspective.

Sorry for the longwinded post, but I've been working on this for several days and I wanted to provide as much information as I could. Hopefully there's a Linux guru in the house that can help (I'm also posting this to the WavPack devel mailing list). Any comments or insights or opinions on any of this are gladly welcome and appreciated!



  • nu774
  • [*][*][*][*][*]
  • Developer
WavPack 4.75 Release Candidate
Reply #29
Well I'm not a Linux guru, but my system (Debian, 32bit) works as you say.
My /etc/ld.so.conf is like this:
Code: [Select]
include /etc/ld.so.conf.d/*.conf

And inside of /etc/ld.so.conf.d:
Code: [Select]
fakeroot-i386-linux-gnu.conf i386-linux-gnu.conf libc.conf

So, they are loaded in this filename order. /usr/local/lib is written in libc.conf, so it is loaded after i386-linux-gnu.conf. If x86_64-linux-gnu.conf is present, it will be loaded at the end. Hence the strange, seemingly inconsistent library search order. I can change library search order simply by renaming them (I confirmed it).

As for rpath thing, I don't think setting rpath in build process is common. However, I found this option in configure script of ffmpeg:
Code: [Select]
  --enable-rpath           use rpath to allow installing libraries in paths
                           use rpath when linking programs [USE WITH CARE]

Maybe you could offer similar option and let user decide. However, I don't think users blame you without it.

WavPack 4.75 Release Candidate
Reply #30
Great to see a new Wavpack release.

For the foreseeable future, I picture myself using it more and more, as Android devices abound.
  • Last Edit: 15 May, 2015, 10:28:00 AM by includemeout
Listen to the music, not the media.

  • 2012
  • [*][*][*]
WavPack 4.75 Release Candidate
Reply #31
/usr and /usr/local are different prefixes.
Are you sure you are running the right executable?

My first idea to fix this was to bump the libtool versioning so that (I thought) my command-line programs would require the newer library (even though there was really no missing dependency in the old library). I was surprised when this did not work. I even added a new function to the library and now I get a runtime error when I try to run the programs, because it still won't use the new library. So either that feature, or my understanding of it, is broken.


What's relevant here is the soname.

Code: [Select]
readelf -d /usr/local/lib/libwavpack.so | grep SONAME
readelf -d `which wavpack` | grep NEEDED


If you did actually bump the soname version to 2. And you are running the right binary. Then both greps should include libwavpack.so.2.
saldl: A lightweight well-featured CLI downloader optimized for speed and early preview.
https://saldl.github.io

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.75 Release Candidate
Reply #32
So, they are loaded in this filename order. /usr/local/lib is written in libc.conf, so it is loaded after i386-linux-gnu.conf. If x86_64-linux-gnu.conf is present, it will be loaded at the end. Hence the strange, seemingly inconsistent library search order. I can change library search order simply by renaming them (I confirmed it).
Thanks for taking the time to confirm this; I'm glad I'm not imagining things!

As for rpath thing, I don't think setting rpath in build process is common. However, I found this option in configure script of ffmpeg:
Maybe you could offer similar option and let user decide. However, I don't think users blame you without it.
Yeah, all you have to do is Google rpath to see that it's not held in high esteem, but I think I'll copy ffmpeg and make it an option (see below)

/usr and /usr/local are different prefixes.
Are you sure you are running the right executable?
Yes, I'm sure I'm running the right ones because I have an option --version that displays both the executable version and the linked library version.

What's relevant here is the soname.

Code: [Select]
readelf -d /usr/local/lib/libwavpack.so | grep SONAME
readelf -d `which wavpack` | grep NEEDED

If you did actually bump the soname version to 2. And you are running the right binary. Then both greps should include libwavpack.so.2.
Thanks for those shell lines; that saved some time, and led me to the eventual answer. It turns out that my problems seem to mostly stem from having read the libtool manual. I had this idea that ld.so was the dynamic linker being referred to in there, but I have since discovered that it isn't. My first clue was when it said:

Quote
If two libraries have identical current and age numbers, then the dynamic linker chooses the library with the greater revision number.

I showed previously that this is definitely not true, or at least only true with the caveat that the library with the greatest revision number must be the first encountered.  Then I realized that the current and age stuff is not implemented in ld.so either, so basically the library must simply exactly match the version requested. So, yes, I can simply bump the soname version to 2 (and I verified this), but then nothing else will use the library. I didn't make it incompatible; I made it better! 

This page was particularly illuminating:

Quote
To set the version of the library, libtool provides the -version-info parameter, which accepts three numbers, separated by colons, that are called respectively, current, revision and age. Both their name and their behaviour, nowadays, have to be considered fully arbitrary, as the explanation provided in the official documentation is confusing to say the least, and can be, in some cases, considered completely wrong.

At this point I'm leaning toward having the executables check the libwavpack version when they start and display a warning if it's wrong and referring the user to the README where I will suggest trying --enable-rpath, which I am also adding. I really hate that i386 users will have to go through all this trouble, but I am afraid of making rpath (or static) the default.

Great to see a new Wavpack release.
For the foreseeable future, I picture myself using it more and more, as Android devices abound.
Cool, glad to hear it!

  • 2012
  • [*][*][*]
WavPack 4.75 Release Candidate
Reply #33
I'm glad you found a solution.

FWIW, build-time linking and execution-time linking was always a source of confusion for some. That's why ld.so is usually referred to as a dynamic linker/loader or just a loader. Where ld is referred to as the dynamic linker.
saldl: A lightweight well-featured CLI downloader optimized for speed and early preview.
https://saldl.github.io

  • [JAZ]
  • [*][*][*][*][*]
WavPack 4.75 Release Candidate
Reply #34
I barely remember about these things, but, wasn't there an "alias" mechanism for this? (not sure if it was the alternatives system or something else). Basically, the true libraries are the .1, .2, etc... but then there's a non-number file that links to one of them which is what the programs are linked to.

Maybe this has been superseeded, but nevertheless, it seems that your main problem is that you are manually installing a library that is also installed by the OS package manager.  make install is not the same than apt-get install, you know.

  • bb10
  • [*][*][*]
WavPack 4.75 Release Candidate
Reply #35
CPU: Core i7 2630QM
Norah Jones - 12. Nightingale [4:12] (24-bit 6ch WAV) -h -x2 option:

Code: [Select]
X                  |     run 1     |     run 2      |
icc-mmx 4.70.0  64 |     29.46s    |     29.32s     |
4.75.0-rc 64       |     34.29s    |     34.09s     |
gcc 4.75.0-rc 64   |     31.01s    |     30.75s     |


Norah Jones - 12. Nightingale [4:12] (16-bit 6ch WAV) -h -x2 option:

Code: [Select]
X                  |     run 1     |     run 2      |
icc-mmx 4.70.0  64 |     30.35s    |     30.34s     |
4.75.0-rc 64       |     34.04s    |     33.94s     |
gcc 4.75.0-rc 64   |     30.82s    |     30.88s     |


It's slightly faster than the standard 4.70.0 binary, but still not as fast as the 4.70.0 64bit ICC binary. I'm curious to see how much faster a 64bit ICC compile of 4.75.0 would be.
  • Last Edit: 21 May, 2015, 11:15:33 PM by kode54

  • lvqcl
  • [*][*][*][*][*]
  • Developer
WavPack 4.75 Release Candidate
Reply #36
I'm curious to see how much faster a 64bit ICC compile of 4.75.0 would be.

How about GCC compile from post #8?

  • lvqcl
  • [*][*][*][*][*]
  • Developer
WavPack 4.75 Release Candidate
Reply #37
from audition\cool_wv4.c:

Code: [Select]
// Version 2.12 - May 10, 2015 (library ver 4.75.1)


Typo?

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.75 Release Candidate
Reply #38
Oops, yes, thanks! 

  • bb10
  • [*][*][*]
WavPack 4.75 Release Candidate
Reply #39
I'm curious to see how much faster a 64bit ICC compile of 4.75.0 would be.

How about GCC compile from post #8?

Added to the table, it's getting pretty close to ICC! Now I'm even more curious about a 4.75.0 64bit ICC build.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.75 Release Candidate
Reply #40
CPU: Core i7 2630QM
Norah Jones - 12. Nightingale [4:12] (24-bit 6ch WAV) -h -x2 option:

Code: [Select]
X                  |     run 1     |     run 2      |
icc-mmx 4.70.0  64 |     29.46s    |     29.32s     |
4.75.0-rc 64       |     34.29s    |     34.09s     |
gcc 4.75.0-rc 64   |     31.01s    |     30.75s     |

I have to admit that I can't really explain your results. I tried to duplicate them using a similar 24-bit file and settings on the most similar CPU I have available (Core i7-4600U) and got these results:

Code: [Select]
version      encode     decode
------------------------------
4.70.0       21.53s     7.63s
4.70.0 x64   25.46s     7.30s
------------------------------
4.75.0       19.19s     5.27s
4.75.0 x64   18.39s     5.12s


Of course, the old 64-bit version does not do well because MSVC won't do MMX on 64-bit (which was one of the motivations to switch to assembly). The decode performance improvement is greater because that used to be pure C, and I assume you do see some improvement there, right?

The only other thing that comes to mind is that you call the file 6ch. I assume that this really is a 5.1 file (FL,FR,FC,LFE,BL,BR), because if it gets treated as 6 independent mono channels then the MMX code can't be used and your results make perfect sense.





  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.75 Release Candidate
Reply #41
A big thank you to everyone for all your testing, help, advice and support. It is greatly appreciated!!

The new released version is currently up on wavpack.com and available for download here.

--David

  • DARcode
  • [*][*][*][*][*]
  • Members (Donating)
WavPack 4.75 Release Candidate
Reply #42
Thanks a bunch David, super glad you're still working on my fav hybrid CODEC  !
WavPack 5.1.0 -b384hx6cmv / qaac 2.64 -V 100

WavPack 4.75 Release Candidate
Reply #43
A big thank you to everyone for all your testing, help, advice and support. It is greatly appreciated!!

The new released version is currently up on wavpack.com and available for download here.

--David

It's been said before I know, but it's in times like these I wish a simple 'like' button existed on HA.

Thanks a lot Bryan!
Listen to the music, not the media.

  • Case
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.75 Release Candidate
Reply #44
CPU: Core i7 2630QM
Norah Jones - 12. Nightingale [4:12] (24-bit 6ch WAV) -h -x2 option:

Code: [Select]
X                  |     run 1     |     run 2      |
icc-mmx 4.70.0  64 |     29.46s    |     29.32s     |
4.75.0-rc 64       |     34.29s    |     34.09s     |
gcc 4.75.0-rc 64   |     31.01s    |     30.75s     |

I have to admit that I can't really explain your results.

Note that the 4.70 ICC file is not just a recompile. The linked post tells it uses some replacement function.

WavPack 4.75 Release Candidate
Reply #45
Thanks David!
Now its time to make some noise so that the new release might show up in distros and whatnots this year....
(FreeBSD still have 4.60.1 in ports... Bah!)

BTW. Should be on HA frontpage maybe....

"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
       - Oceania Association of Autonomous Astronauts