Skip to main content

Topic: How about multi-threaded wvunpack? (Read 1198 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Funk
  • [*]
How about multi-threaded wvunpack?
Hi, is there a reason not to use multiple threads in wvunpack to speed up the unpacking of lossless WavPack files to WAV? If not, could you make a multi-threaded unpacker?
  • Last Edit: 22 June, 2017, 07:21:58 PM by Funk

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: How about multi-threaded wvunpack?
Reply #1
Too few people use Wavpack to encode 10MHz data, I suppose...  ;)

  • atomnuker
  • [*]
Re: How about multi-threaded wvunpack?
Reply #2
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.

  • saratoga
  • [*][*][*][*][*]
Re: How about multi-threaded wvunpack?
Reply #3
Hi, is there a reason not to use multiple threads in wvunpack to speed up the unpacking of lossless WavPack files to WAV? If not, could you make a multi-threaded unpacker?

I think most common programs already do this.  Foobar, dbpoweramp, etc. 

  • lvqcl
  • [*][*][*][*][*]
  • Developer
Re: How about multi-threaded wvunpack?
Reply #4
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.

I just tested wavpack 5.1.0 and ffmpeg 3.3.1 and ffmpeg was noticeably slower in single-thread mode.

  • Funk
  • [*]
Re: How about multi-threaded wvunpack?
Reply #5
Too few people use Wavpack to encode 10MHz data, I suppose...  ;)
LOL, yes. There has been extremely good sporadic E radio reception (https://en.wikipedia.org/wiki/Sporadic_E_propagation) this summer and I have saved files of several hundreds of gigabytes compressed to 56-85% reduction of their original sizes thanks to WavPack. Yes some of the files are 15% of their original size. I have begun to make Autohotkey scripts to decode different parts of the files in different threads and then combine them with sox. It is a lot faster than to decode in one thread but it needs a lot of disk space for the temporary wav files.
  • Last Edit: 23 June, 2017, 05:08:06 PM by Funk

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How about multi-threaded wvunpack?
Reply #6
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.

I just tested wavpack 5.1.0 and ffmpeg 3.3.1 and ffmpeg was noticeably slower in single-thread mode.

Thanks lvqcl, this makes sense. Since ffmpeg does not have the asm optimizations that libwavpack has, it should be slower. Of course, it's always possible that on a particular architecture, having the right -mtune could make ffmpeg faster on that machine.

As for the OP, I have thought about how this could be done with pthreads, and it's not that difficult because every WavPack block is independently decodable. The only thing I would worry about would be whether disk I/O would start to become the bottleneck, but based on what you've been doing, that's not the case. This is on my list of things to try one day...  :)

  • dutch109
  • [*][*][*]
Re: How about multi-threaded wvunpack?
Reply #7
Thanks lvqcl, this makes sense. Since ffmpeg does not have the asm optimizations that libwavpack has, it should be slower. Of course, it's always possible that on a particular architecture, having the right -mtune could make ffmpeg faster on that machine.

On my system (Arch Linux, core i7 950), FFmpeg is compiled with -march=native -mtune=native.
For what it's worth, a quick test (tested WavPack file was encoded with -hhx6):
Code: [Select]
$ wavpack --version
wavpack 5.1.0
libwavpack 5.1.0

$ ffmpeg -version
ffmpeg version 3.3.2 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 7.1.1 (GCC) 20170516
configuration: --prefix=/usr --extra-cflags=-I/opt/cuda/include --extra-ldflags='-L/opt/cuda/lib64 -Wl,-rpath -Wl,/opt/intel/mediasdk/lib64' --toolchain=hardened --enable-rpath --enable-gpl --enable-version3 --enable-nonfree --disable-static --enable-shared --enable-avresample --enable-cuda --enable-cuvid --enable-libnpp --enable-libmfx --enable-nvenc --enable-omx --enable-omx-rpi --enable-avisynth --enable-chromaprint --enable-decoder=atrac3 --enable-decoder=atrac3p --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gpl --enable-gray --enable-iconv --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcelt --enable-libdc1394 --enable-libfdk-aac --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libilbc --enable-libkvazaar --enable-libmodplug --enable-libmp3lame --enable-libnut --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopencv --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lzma --enable-netcdf --enable-openal --enable-opencl --enable-opengl --enable-openssl --enable-sdl2 --enable-vaapi --enable-vdpau --enable-videotoolbox --enable-xlib --enable-zlib
libavutil      55. 58.100 / 55. 58.100
libavcodec     57. 89.100 / 57. 89.100
libavformat    57. 71.100 / 57. 71.100
libavdevice    57.  6.100 / 57.  6.100
libavfilter     6. 82.100 /  6. 82.100
libavresample   3.  5.  0 /  3.  5.  0
libswscale      4.  6.100 /  4.  6.100
libswresample   2.  7.100 /  2.  7.100
libpostproc    54.  5.100 / 54.  5.100

$ cat Vulgar\ Display\ of\ Power.wv > /dev/null

$ time wvunpack -q Vulgar\ Display\ of\ Power.wv -o - > /dev/null

real 0m21,754s
user 0m21,710s
sys 0m0,040s

$ time ffmpeg -loglevel quiet -threads 1 -i Vulgar\ Display\ of\ Power.wv -c:a pcm_s16le -f null /dev/null

real 0m27,633s
user 0m27,530s
sys 0m0,100s

$ time ffmpeg -loglevel quiet -i Vulgar\ Display\ of\ Power.wv -c:a pcm_s16le -f null /dev/null

real 0m7,354s
user 0m47,360s
sys 0m0,190s


So wvunpack is faster than FFmpeg for single threaded decoding, but with 8 threads FFmpeg is a lot faster (as expected).
  • Last Edit: 25 June, 2017, 05:17:12 PM by dutch109
Opus 96 kb/s (Android) / Vorbis -q5 (PC) / WavPack -hhx6m (Archive)
Developer of r128gain, SACAD and https://playnoise.com/

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
Re: How about multi-threaded wvunpack?
Reply #8
So wvunpack is faster than FFmpeg for single threaded decoding, but with 8 threads FFmpeg is a lot faster (as expected).
Wow, I am really surprised by that level of improvement...thanks for testing and posting this!