Hi, is there a reason not to use multiple threads in wvunpack to speed up the unpacking of lossless WavPack files to WAV? If not, could you make a multi-threaded unpacker?
Too few people use Wavpack to encode 10MHz data, I suppose... ;)
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.
Hi, is there a reason not to use multiple threads in wvunpack to speed up the unpacking of lossless WavPack files to WAV? If not, could you make a multi-threaded unpacker?
I think most common programs already do this. Foobar, dbpoweramp, etc.
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.
I just tested wavpack 5.1.0 and ffmpeg 3.3.1 and ffmpeg was noticeably slower in single-thread mode.
Too few people use Wavpack to encode 10MHz data, I suppose... ;)
LOL, yes. There has been extremely good sporadic E radio reception (https://en.wikipedia.org/wiki/Sporadic_E_propagation) this summer and I have saved files of several hundreds of gigabytes compressed to 56-85% reduction of their original sizes thanks to WavPack. Yes some of the files are 15% of their original size. I have begun to make Autohotkey scripts to decode different parts of the files in different threads and then combine them with sox. It is a lot faster than to decode in one thread but it needs a lot of disk space for the temporary wav files.
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.
I just tested wavpack 5.1.0 and ffmpeg 3.3.1 and ffmpeg was noticeably slower in single-thread mode.
Thanks lvqcl, this makes sense. Since ffmpeg does not have the asm optimizations that libwavpack has, it
should be slower. Of course, it's always possible that on a particular architecture, having the right -mtune could make ffmpeg faster on that machine.
As for the OP, I have thought about how this could be done with pthreads, and it's not that difficult because every WavPack block is independently decodable. The only thing I would worry about would be whether disk I/O would start to become the bottleneck, but based on what you've been doing, that's not the case. This is on my list of things to try one day... :)
Thanks lvqcl, this makes sense. Since ffmpeg does not have the asm optimizations that libwavpack has, it should be slower. Of course, it's always possible that on a particular architecture, having the right -mtune could make ffmpeg faster on that machine.
On my system (Arch Linux, core i7 950), FFmpeg is compiled with
-march=native -mtune=native.
For what it's worth, a quick test (tested WavPack file was encoded with -hhx6):
$ wavpack --version
wavpack 5.1.0
libwavpack 5.1.0
$ ffmpeg -version
ffmpeg version 3.3.2 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 7.1.1 (GCC) 20170516
configuration: --prefix=/usr --extra-cflags=-I/opt/cuda/include --extra-ldflags='-L/opt/cuda/lib64 -Wl,-rpath -Wl,/opt/intel/mediasdk/lib64' --toolchain=hardened --enable-rpath --enable-gpl --enable-version3 --enable-nonfree --disable-static --enable-shared --enable-avresample --enable-cuda --enable-cuvid --enable-libnpp --enable-libmfx --enable-nvenc --enable-omx --enable-omx-rpi --enable-avisynth --enable-chromaprint --enable-decoder=atrac3 --enable-decoder=atrac3p --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gpl --enable-gray --enable-iconv --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcelt --enable-libdc1394 --enable-libfdk-aac --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libilbc --enable-libkvazaar --enable-libmodplug --enable-libmp3lame --enable-libnut --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopencv --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lzma --enable-netcdf --enable-openal --enable-opencl --enable-opengl --enable-openssl --enable-sdl2 --enable-vaapi --enable-vdpau --enable-videotoolbox --enable-xlib --enable-zlib
libavutil 55. 58.100 / 55. 58.100
libavcodec 57. 89.100 / 57. 89.100
libavformat 57. 71.100 / 57. 71.100
libavdevice 57. 6.100 / 57. 6.100
libavfilter 6. 82.100 / 6. 82.100
libavresample 3. 5. 0 / 3. 5. 0
libswscale 4. 6.100 / 4. 6.100
libswresample 2. 7.100 / 2. 7.100
libpostproc 54. 5.100 / 54. 5.100
$ cat Vulgar\ Display\ of\ Power.wv > /dev/null
$ time wvunpack -q Vulgar\ Display\ of\ Power.wv -o - > /dev/null
real 0m21,754s
user 0m21,710s
sys 0m0,040s
$ time ffmpeg -loglevel quiet -threads 1 -i Vulgar\ Display\ of\ Power.wv -c:a pcm_s16le -f null /dev/null
real 0m27,633s
user 0m27,530s
sys 0m0,100s
$ time ffmpeg -loglevel quiet -i Vulgar\ Display\ of\ Power.wv -c:a pcm_s16le -f null /dev/null
real 0m7,354s
user 0m47,360s
sys 0m0,190s
So wvunpack is faster than FFmpeg for single threaded decoding, but with 8 threads FFmpeg is a lot faster (as expected).
So wvunpack is faster than FFmpeg for single threaded decoding, but with 8 threads FFmpeg is a lot faster (as expected).
Wow, I am really surprised by that level of improvement...thanks for testing and posting this!