Skip to main content

Topic: Optimized vorbis encoder for Linux (SSE instructions) (Read 24834 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
A quick look around the forum seemed to indicate that no-one has ever attempted to build Vorbis with Lancer optimisations for Linux ?!

So I spent a few hours fiddling and produced an executable:
- Linux
- aoTuV-beta5
- Lancer optimisations for SSE
- 32 bit (x86)
- statically linked
- around 3x faster compared to the standard aoTuV-beta5 build

I did not attempt to port Lancer's SSE2/SSE3 instructions to Linux because the difference between SSE/SSE2/SSE3 is minimal on the PCs I use.
(Intel Celeron M and AMD Turion 64x2)

Any interest? If so, I will post the binary.

  • skamp
  • [*][*][*][*][*]
  • Developer
Optimized vorbis encoder for Linux (SSE instructions)
Reply #1
Better yet, post your (I assume) cleaned-up sources, so that we can build it ourselves.
See my profile for measurements, tools and recommendations.

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #2
Better yet, post your (I assume) cleaned-up sources, so that we can build it ourselves.


Excellent idea! I don't know how things work around here or in the Linux world, source-wise, as I'm very much a Windows person.

Usually I would commit my changes into a source code repository, perhaps with optional build switches, and then add a few more test cases to the regression tests to ensure that my changes won't break anything.

What I did to arrive at the current source code:
1. Downloaded aoTuV-beta5 source.
2. Applied latest Lancer patches.
3. Made mods so these patches would build on Linux.

I guess I would be able to produce a patch against the Lancer patches - but would this be at all useable to anyone?

Is it just me, or is distributing a set of patches just an absolutely horrid way to do things?

  • Soap
  • [*][*][*][*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #3
Is it just me, or is distributing a set of patches just an absolutely horrid way to do things?


Only if your changes are the only changes someone wants.
Creature of habit.

  • HotshotGG
  • [*][*][*][*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #4
Quote
Any interest? If so, I will post the binary.


Yes and possible have rarewares distribute the binary or host it externally. I will place a link to it in the wiki page when I get the opportunity! Thanks.
budding I.T professional

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #5
Yes and possible have rarewares distribute the binary or host it externally. I will place a link to it in the wiki page when I get the opportunity! Thanks.


Do you have rarewares' contact info ?

  • skamp
  • [*][*][*][*][*]
  • Developer
Optimized vorbis encoder for Linux (SSE instructions)
Reply #6
I guess I would be able to produce a patch against the Lancer patches - but would this be at all useable to anyone?

Rather, against either vanilla libvorbis-1.2.3 or aotuv's latest beta source tarball (libvorbis-aotuv_b5.7).
Code: [Select]
diff -Naur aotuv-b5.7_20090301/ aotuv-b5.7_20090301-lancer/ > aotuv-b5.7_20090301-lancer.diff

That would absolutely work for me.
See my profile for measurements, tools and recommendations.

  • HotshotGG
  • [*][*][*][*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #7
Quote
Do you have rarewares' contact info ?


I don't I am sure they wouldn't' mind hosting it though! Contact Rjamorim on here. I am not sure who else has access to the website?
  • Last Edit: 05 November, 2009, 11:59:32 AM by HotshotGG
budding I.T professional

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #8
I put the required patches and modifications under source control, so everyone can:
- build
- look at the diffs
- generate patches

Use the repository's web interface to browse it. You can also download the accelerated SSE Linux executable there.
http://repo.or.cz/w/vorbis-lancer-gcc.git/tree

  • imre_herceg
  • [*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #9
Thank you for the binary. It is twice as fast as mainstream oggenc.
Unfortunately it accepts only wav files as input. Could you please add flac support?
You used aotuv-b5 version of 2006-10-24.  Do you plan to make a binary of the latest aotuv-b5.7 (2009-03-01)?

  • skamp
  • [*][*][*][*][*]
  • Developer
Optimized vorbis encoder for Linux (SSE instructions)
Reply #10
It is twice as fast as mainstream oggenc.

Funny, it's exactly twice as fast as aotuv b5.7 on my netbook (Arch Linux, GCC 4.4.2 on an Intel Atom N270 1.6GHz): 14.88x vs. 7.44x. Very nice!
Ideally Aoyumi would incorporate the optimizations…
See my profile for measurements, tools and recommendations.

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #11
Very nice!

The credit goes to the author of the Lancer Vorbis acceleration project:
http://homepage3.nifty.com/blacksword/

I just made a few patches to make it work on Linux.

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #12
Unfortunately it accepts only wav files as input. Could you please add flac support?

FLAC support now added. Get the updated binary direct from the repository:
http://repo.or.cz/w/vorbis-lancer-gcc.git/blob/HEAD:/oggenc

I will try to encorporate Aoyoumi's latest version. It will be a good test for Git ;-)
  • Last Edit: 14 November, 2009, 06:42:24 PM by The_Sven

  • skamp
  • [*][*][*][*][*]
  • Developer
Optimized vorbis encoder for Linux (SSE instructions)
Reply #13
I will try to encorporate Aoyoumi's latest version. It will be a good test for Git ;-)

Gotta love Free Software. One guy invents a cool codec, another guy improves it, yet another guy makes it faster, and a fourth party makes the latter improvements work under linux. Granted, it took some time, but better late than never. Closed source software developers, take notice!

Thanks for your work.
See my profile for measurements, tools and recommendations.

  • imre_herceg
  • [*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #14
FLAC support now added. I will try to encorporate Aoyoumi's latest version. It will be a good test for Git ;-)


Thanks for both. Flac support works OK.

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #15
You used aotuv-b5 version of 2006-10-24.  Do you plan to make a binary of the latest aotuv-b5.7 (2009-03-01)?

Unfortunately I can't. Aoyumi has updated the very same algorithms which the Lancer author optimised. Put another way, a plain old conflict.

So unless the Lancer author 'wakes up' again (he hasn't worked on it since 2006 ?), then beta 5 is the latest accelerated version we're ever going to get. Not bad, as it is the current peer-reviewed Hydrogen-audio recommended version. (See http://wiki.hydrogenaudio.org/index.php?ti...ded_Ogg_Vorbis)

Optimized vorbis encoder for Linux (SSE instructions)
Reply #16
I compiled your snapshot vorbis-lancer-gcc-master.tar.gz locally by running ./build-script. The result here are not a static build and seems to have compiled my systems libogg in dynamically:

Code: [Select]
 
~/src/lancer/vorbis-lancer-gcc$ file oggenc
oggenc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.8, dynamically linked (uses shared libs), not stripped
~/src/lancer/vorbis-lancer-gcc$ ldd oggenc
    linux-gate.so.1 =>  (0xb7fae000)
    libogg.so.0 => /usr/lib/libogg.so.0 (0xb7f63000)
    libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f3d000)
    libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ddf000)
    /lib/ld-linux.so.2 (0xb7f94000)
~/src/lancer/vorbis-lancer-gcc$

"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
       - Oceania Association of Autonomous Astronauts

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #17
I compiled your snapshot vorbis-lancer-gcc-master.tar.gz locally by running ./build-script. The result here are not a static build and seems to have compiled my systems libogg in dynamically:

You're absolutely right: my build script doesn't generate a completely static binary. To do that, I would have to modify the autoconf build script and I don't know how, being a Windows person. I've posted a question about it on LinuxQuestions:
http://www.linuxquestions.org/questions/li...utoconf-769088/

If you want to get a statically linked binary, do the following from the ogg-aoTuVb5-lancer directory (i.e. the root):
1. $ ./build-script (if you haven't done this already)
2. $ cd vorbis-tools-1.2.0/oggenc/
3. $ make clean; make
4. Copy the last gcc line on your console, paste it on your command line and add the -static switch. Execute it.
5. $ cd ..
6. $ make install
7. The binary is now in ogg-aoTuVb5-lancer/out/bin

Any help with fixing this is very welcome!!
  • Last Edit: 15 November, 2009, 08:44:26 AM by The_Sven

Optimized vorbis encoder for Linux (SSE instructions)
Reply #18
If you want to get a statically linked binary, do the following from the ogg-aoTuVb5-lancer directory (i.e. the root):
1. $ ./build-script (if you haven't done this already)
2. $ cd vorbis-tools-1.2.0/oggenc/
3. $ make clean; make
4. Copy the last gcc line on your console, paste it on your command line and add the -static switch. Execute it.
5. $ cd ..
6. $ make install
7. The binary is now in ogg-aoTuVb5-lancer/out/bin


That last gcc line also had a pointer to /usr/lib/libogg.so instead of to ${WORKDIR}/out/lib/libbogg.a which made it fail with:

Code: [Select]
/usr/bin/ld: attempted static link of dynamic object `/usr/lib/libogg.so'
collect2: ld returned 1 exit status

Correcting that bit I succeeded in making the build static. 
Neither I know enough to fix this...

EDIT:
My build fails anyway, somethings wrong with it, trying to encode a file:
Quote
$ ./oggenc -q5 Luckie.wav -o Luckie.ogg
Open with wav-module: WAV file reader
Encoding "Luckie.wav" to
        "Luckie.ogg"
at quality 5,00
Mode initialisation failed: invalid parameters for quality


Well, well... 

  • Last Edit: 15 November, 2009, 10:16:57 AM by Mr_Rabid_Teddybear
"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
       - Oceania Association of Autonomous Astronauts

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #19
Well, well... 

Does your build machine have libogg, liborbis and libflac installed? Mine doesn't. I suspect that vorbis-tools's makefile prefers installed libraries, and simply ignores the --with-ogg, --with-vorbis switches. I already had to do a work-around for the --with-flac switch, which didn't work.

As an aside: I'm a Windows person, and know very little of the Linux dev world. It's completely baffling to me why *anyone* would use a tool like autoconf/automake. For starters, I haven't yet found a single software developer who understands these tools. Is this really the state-of-the-art in Linux? With Visual Studio, build maintenance consists of a mouse click here and there. Time saved there is time gained for important things, such as code development/debugging/test etc.

The --with-flac switch bug was discussed by its developers on a Xiph.org forum as early as 2007(!!). Are we the only ones struggling with the build system? I doubt it...

Optimized vorbis encoder for Linux (SSE instructions)
Reply #20
Does your build machine have libogg, liborbis and libflac installed? Mine doesn't.

Yes. I'm pretty sure problems stems from it compiling against system-installed libraries instead of the provided ones.

"ONLY THOSE WHO ATTEMPT THE IMPOSSIBLE WILL ACHIEVE THE ABSURD"
       - Oceania Association of Autonomous Astronauts

  • [JAZ]
  • [*][*][*][*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #21
autoconf/automake works in linux x86, linux ARM, UNIX x86, UNIX Sparc, .... list goes on.
Not only that, but autoconf is old and tested.

Visual studio needs visual studio, and everything that visual studio provides (runtimes, etc..).

The truth is that nowadays, autoconf is still used (because it exists everywhere), but more and more projects use other tools, like scons or other python based ones.


Now, about your comment of not being able to update b5 to b5.7... Is that because you think you cannot do as good as lancer is, or because you don't know about intrinsics?
I haven't looked at the code, but did some work on intrinsics a few years ago (for a windows/linux project, dsp related, but mostly mixing)

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #22

autoconf/automake works in linux x86, linux ARM, UNIX x86, UNIX Sparc, .... list goes on.
Not only that, but autoconf is old and tested.

vi and ed are also old and tested, work in Linux, Sparc, HP-UX, Irix, etc. Why not use those? :-) Same with bash, you could write shell scripts to calculate your tax returns...
Anyways, let's not fight over this. Rather, let's work together to find an alternative that works!


Now, about your comment of not being able to update b5 to b5.7... Is that because you think you cannot do as good as lancer is, or because you don't know about intrinsics?

I haven't looked at intrinsics yet. All I did, this far, was to 'port' Lancer to Linux. Would you like to look for accelerations? If so, it'd be best to get cooperation from Aoyumi.

First thing is to profile OggEnc - applications use 5% of the code 95% of the time. We should find out where the 'expensive' functions are, and concentrate on those.

What do you think?

  • [JAZ]
  • [*][*][*][*][*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #23
vi and ed are also old and tested, work in Linux, Sparc, HP-UX, Irix, etc. Why not use those? :-)

I do use vi from time to time... well, vim.

First thing is to profile OggEnc - applications use 5% of the code 95% of the time. We should find out where the 'expensive' functions are, and concentrate on those.


Yes, that would be adequate. Anyway, the answer most probably is the mdct.

I am comparing sources right now to get a clear idea of what does lancer modify. I am also downloading the snapshot from your git sources to see what you've changed.

I am PM'ing you my e-mail.

  • The_Sven
  • [*]
Optimized vorbis encoder for Linux (SSE instructions)
Reply #24
My build fails anyway, somethings wrong with it

I'm sorry to hear it doesn't work on your build machine. If you really want to build it, you can try setting up an Ubuntu 9.10 installation from scratch. That's what I am using.