Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How about multi-threaded wvunpack? (Read 7224 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How about multi-threaded wvunpack?

Hi, is there a reason not to use multiple threads in wvunpack to speed up the unpacking of lossless WavPack files to WAV? If not, could you make a multi-threaded unpacker?

Re: How about multi-threaded wvunpack?

Reply #1
Too few people use Wavpack to encode 10MHz data, I suppose...  ;)

Re: How about multi-threaded wvunpack?

Reply #2
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.

Re: How about multi-threaded wvunpack?

Reply #3
Hi, is there a reason not to use multiple threads in wvunpack to speed up the unpacking of lossless WavPack files to WAV? If not, could you make a multi-threaded unpacker?

I think most common programs already do this.  Foobar, dbpoweramp, etc. 

Re: How about multi-threaded wvunpack?

Reply #4
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.

I just tested wavpack 5.1.0 and ffmpeg 3.3.1 and ffmpeg was noticeably slower in single-thread mode.

 

Re: How about multi-threaded wvunpack?

Reply #5
Too few people use Wavpack to encode 10MHz data, I suppose...  ;)
LOL, yes. There has been extremely good sporadic E radio reception (https://en.wikipedia.org/wiki/Sporadic_E_propagation) this summer and I have saved files of several hundreds of gigabytes compressed to 56-85% reduction of their original sizes thanks to WavPack. Yes some of the files are 15% of their original size. I have begun to make Autohotkey scripts to decode different parts of the files in different threads and then combine them with sox. It is a lot faster than to decode in one thread but it needs a lot of disk space for the temporary wav files.

Re: How about multi-threaded wvunpack?

Reply #6
FFmpeg's wavpack decoder is multithreaded and is much faster than wvunpack even with 1 thread.

I just tested wavpack 5.1.0 and ffmpeg 3.3.1 and ffmpeg was noticeably slower in single-thread mode.

Thanks lvqcl, this makes sense. Since ffmpeg does not have the asm optimizations that libwavpack has, it should be slower. Of course, it's always possible that on a particular architecture, having the right -mtune could make ffmpeg faster on that machine.

As for the OP, I have thought about how this could be done with pthreads, and it's not that difficult because every WavPack block is independently decodable. The only thing I would worry about would be whether disk I/O would start to become the bottleneck, but based on what you've been doing, that's not the case. This is on my list of things to try one day...  :)

Re: How about multi-threaded wvunpack?

Reply #7
Thanks lvqcl, this makes sense. Since ffmpeg does not have the asm optimizations that libwavpack has, it should be slower. Of course, it's always possible that on a particular architecture, having the right -mtune could make ffmpeg faster on that machine.

On my system (Arch Linux, core i7 950), FFmpeg is compiled with -march=native -mtune=native.
For what it's worth, a quick test (tested WavPack file was encoded with -hhx6):
Code: [Select]
$ wavpack --version
wavpack 5.1.0
libwavpack 5.1.0

$ ffmpeg -version
ffmpeg version 3.3.2 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 7.1.1 (GCC) 20170516
configuration: --prefix=/usr --extra-cflags=-I/opt/cuda/include --extra-ldflags='-L/opt/cuda/lib64 -Wl,-rpath -Wl,/opt/intel/mediasdk/lib64' --toolchain=hardened --enable-rpath --enable-gpl --enable-version3 --enable-nonfree --disable-static --enable-shared --enable-avresample --enable-cuda --enable-cuvid --enable-libnpp --enable-libmfx --enable-nvenc --enable-omx --enable-omx-rpi --enable-avisynth --enable-chromaprint --enable-decoder=atrac3 --enable-decoder=atrac3p --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gpl --enable-gray --enable-iconv --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcelt --enable-libdc1394 --enable-libfdk-aac --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libilbc --enable-libkvazaar --enable-libmodplug --enable-libmp3lame --enable-libnut --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopencv --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lzma --enable-netcdf --enable-openal --enable-opencl --enable-opengl --enable-openssl --enable-sdl2 --enable-vaapi --enable-vdpau --enable-videotoolbox --enable-xlib --enable-zlib
libavutil      55. 58.100 / 55. 58.100
libavcodec     57. 89.100 / 57. 89.100
libavformat    57. 71.100 / 57. 71.100
libavdevice    57.  6.100 / 57.  6.100
libavfilter     6. 82.100 /  6. 82.100
libavresample   3.  5.  0 /  3.  5.  0
libswscale      4.  6.100 /  4.  6.100
libswresample   2.  7.100 /  2.  7.100
libpostproc    54.  5.100 / 54.  5.100

$ cat Vulgar\ Display\ of\ Power.wv > /dev/null

$ time wvunpack -q Vulgar\ Display\ of\ Power.wv -o - > /dev/null

real 0m21,754s
user 0m21,710s
sys 0m0,040s

$ time ffmpeg -loglevel quiet -threads 1 -i Vulgar\ Display\ of\ Power.wv -c:a pcm_s16le -f null /dev/null

real 0m27,633s
user 0m27,530s
sys 0m0,100s

$ time ffmpeg -loglevel quiet -i Vulgar\ Display\ of\ Power.wv -c:a pcm_s16le -f null /dev/null

real 0m7,354s
user 0m47,360s
sys 0m0,190s


So wvunpack is faster than FFmpeg for single threaded decoding, but with 8 threads FFmpeg is a lot faster (as expected).
Opus 96 kb/s (Android) / Vorbis -q5 (PC) / WavPack -hhx6m (Archive)

Re: How about multi-threaded wvunpack?

Reply #8
So wvunpack is faster than FFmpeg for single threaded decoding, but with 8 threads FFmpeg is a lot faster (as expected).
Wow, I am really surprised by that level of improvement...thanks for testing and posting this!