Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: WavPack 5.6.4 Has Multithreading Option (Read 1193 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

WavPack 5.6.4 Has Multithreading Option

Some of you might be aware that I have been working with the Allen Institute for Neural Dynamics team on the applicability of WavPack for compressing electrophysiology data collected from probes inserted into the brain. These probes generally have many channels (e.g., 384) and while they were looking at FLAC they were referred to WavPack (thanks @ktf ) because it natively handles up to 4096 channels. You can read about this on GitHub here.

One of the issues that came up in this project was that while the encoding was parallelized in the Python front end, the decoding was not, and therefore was slow. I realized that it would not be that complicated to parallelize multichannel decoding right inside libwavpack. All of the encoded data for every channel must already be resident before decoding starts and there’s no interaction between the various streams. I put together a proof-of-concept for this and it greatly sped up their decoding. While it wasn’t needed initially, I also extended this to encoding because the organization was similar. I refer to this as spatial multithreading.

Once I had written all the code to have multiple worker threads in libwavpack I could not help but begin thinking about extending this to mono and stereo streams, which would have to be done temporally instead of spatially. This is more complex because it would only work with operations that span multiple WavPack blocks (which are already fairly long) so the command-line programs, and any other application that wanted to use this, would also have to change. And with encoding there’s an additional complication that WavPack, being continuously adaptive, uses state information from the end of the previous block to start encoding the next block.

But long story made short, I found ways around most of these issues and created a set of command-line programs that include a --threads option to allow multithreaded encoding and decoding, and the results are pretty dramatic. The speed improvement is generally between 2x and 4x, depending on the modes used, and can be as high as about 6x with the maximum thread count (--threads=12). The only serious limitation that I wasn’t able to get around is that the hybrid mode (lossy or lossless) does not encode multithreaded with stereo or mono files (although most multichannel encoding is fine with it). Multithreaded decoding is available for all files.

To be clear, this is not any more efficient than older WavPack versions, and in fact is slightly less so because of the context switching overhead. The speedup comes from using multiple cores or threads, so unless a single core can't keep up (which seems an unlikely situation) then using this for realtime playback or recording does not make sense. This does make sense for some offline operations though, like verify or transcode, and on my system CD encoding with --hhx4 goes from 11x RT to 59x RT!

Other notes:

  • This includes and lot of new and changed code and so I except bugs to appear, so it’s more important than ever to use the -v option when encoding to make sure everything worked.
  • If enabled, the default number of worker threads is 4 and that seems to be a reasonable compromise, even on computers that have less than 4 cores (see below). The maximum parameter is 12 workers, and this can give an additional speedup in some cases, however not all. For example, when on battery power only my new HP Dev One seems to throttle down the CPU speed when more than 4 cores are running, causing no improvement beyond that. On AC power it doesn't.
  • I have tested this on an old Core 2 Duo system that served as my main PC until this happened and even there I get a nice speedup. This old system somewhat explains why I wasn’t too interested in this earlier (I was a little late to the multi-core party).
  • For multichannel encoding, the output files should be identical whether threading is used or not. For now, dynamic noise shaping (dns) in the hybrid lossless mode and the --merge-blocks (for lossyWAV) option prevent multithreaded operation (although I might be able to fix those eventually).
  • For mono or stereo encoding, the output files probably will not be identical for different numbers of worker threads, however the compression ratios will be roughly similar. I recommend always using at least -x1 for all mono or stereo encodes because with multithreading it's basically free. Again, multithreaded encoding does not work for mono or stereo files in any hybrid mode.

Thanks in advance for any testing and/or feedback!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #1 –
Cool!

Probably a good idea to instruct oss-fuzz to use the thread sanitizer (TSan) for fuzzing to catch some bugs. Perhaps you've already run tests with TSan locally?
Music: sounds arranged such that they construct feelings.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #2 –
Quote from: ktf on
Cool!

Probably a good idea to instruct oss-fuzz to use the thread sanitizer (TSan) for fuzzing to catch some bugs. Perhaps you've already run tests with TSan locally?
Thanks!

I definitely run all the sanitizers (including tsan) locally quite often.

As for the oss-fuzz, that's probably a good thing to add too. Unfortunately an issue I can see is that it requires big files and big reads to invoke the multithreading (at least the temporal variety) and the fuzzers get slow and inefficient with large files (I spent a lot of time distilling the seeds down to the absolute smallest size I could). But there may be some way to spoof that (like using really small frames).

So, why? Did you see something?   :D

Re: WavPack 5.6.4 Has Multithreading Option

Reply #3 –
Quote from: bryant on
So, why? Did you see something?   :D
No, I just wanted to help  :)) I have been busy the last few months improving code coverage of oss-fuzz for flac and fixing the found bugs, so it really was the first thing that popped in my mind reading this.

Quote from: bryant on
(I spent a lot of time distilling the seeds down to the absolute smallest size I could).
Ah, yes. I guess I can consider myself lucky, I really only need to worry about coverage and fixing bugs with flac, I haven't really bothered seeding anything.

Quote from: bryant on
I wasn’t too interested in this earlier (I was a little late to the multi-core party).[/li][/list]
I think WavPack is the first codec employing multithreading "within" a single input (at least with 'reference' software), so in that sense, you're the first to the party really!

Music: sounds arranged such that they construct feelings.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #4 –
Quickly testing one file (tested for ID3 import capability as you can see, but I might as well run --threads)

tl;dr:
Confirming that -x1 runs "virtually for free".


Some indicative times, recorded from WavPack's own timer - for any sort of rigor I would have to run this computer up to a stable temperature, and well ... not tonight.

-fx<N>, running the following loop with the x64 build, on an SSD:
for %x IN (0,1,4,6) DO (C:\bin\wavpack-5.6.4-win64\wavpack.exe --import-id3 -yfx%x ".\12277922_Galaxia_(Extended_Mix).aiff" & C:\bin\wavpack-5.6.4-win64\wavpack.exe --threads --import-id3 -yfx%x ".\12277922_Galaxia_(Extended_Mix).aiff")
0.9 vs 0.7 seconds for x0
1.6 vs 0.6 seconds for x1
6.5 vs 2.5 seconds for x4
9.3 vs 3.4 seconds for x6

Adding "mv", so --import-id3 -yfmvx%x,
1.8 vs 1.1 seconds for x0
2.3 vs 1.1 seconds for x1
7.3 vs 2.9 seconds for x4
9.9 vs 3.9 seconds for x6

Keep mv but use -g (normal mode):
2.9 vs 1.7 seconds for x0
4.0 vs 1.8 seconds for x1
30 vs 10 seconds for x4
49 vs 17 seconds for x6

h:
2.3 vs 1.3 seconds for x0
3.0 vs 1.6 seconds for x1
19.4 vs 6.8 seconds for x4
55 vs 19 seconds for x6

hh:
3.5 vs 2.0 seconds for x0
5.1 vs 2.2 seconds for x1
45 vs 17 seconds for x4
98 vs 32 seconds for x6
 

Re: WavPack 5.6.4 Has Multithreading Option

Reply #5 –
Quote from: ktf on
Quote from: bryant on
I wasn’t too interested in this earlier (I was a little late to the multi-core party).
I think WavPack is the first codec employing multithreading "within" a single input (at least with 'reference' software), so in that sense, you're the first to the party really!
Good point! I'm still not sure how much sense multithreading makes there, but after I had spent so much time getting the infrastructure in, it seemed silly not to go all the way. Fortunately I did it in a way that it can easily be disabled in the build if it turns out to be not ready for prime time.

Quote from: Porcus on
Quickly testing one file (tested for ID3 import capability as you can see, but I might as well run --threads)

tl;dr:
Confirming that -x1 runs "virtually for free".


Some indicative times, recorded from WavPack's own timer - for any sort of rigor I would have to run this computer up to a stable temperature, and well ... not tonight.
Thanks Porcus! Yeah, at this point I'm happy to see it not crash, so that's good. I've done a lot of multithreading programming on embedded systems, but not on Windows.

Am considering making the base "extra" mode come on by default when "--threads" is specified. And of course, if this ends up universally stable, might be worth making threading the default too (and add "--no-threads" perhaps).

Re: WavPack 5.6.4 Has Multithreading Option

Reply #6 –
Quote from: bryant on
Quote from: ktf on
Quote from: bryant on
I wasn’t too interested in this earlier (I was a little late to the multi-core party).
I think WavPack is the first codec employing multithreading "within" a single input (at least with 'reference' software), so in that sense, you're the first to the party really!
Good point!

Except, TAK does it already. The -tn switch makes an impact even when processing a single file:
Code: [Select]
C:\tmp>\bin\TAK\Applications\Takc.exe -e -p4 -tn1 -overwrite NNN.wav
NNN.wav                             ..........  36.18%  256*

Compression:     36.18 %
Duration:         2.45 sec
Speed:          256.16 * real time

C:\tmp>\bin\TAK\Applications\Takc.exe -e -p4 -tn4 -overwrite NNN.wav
NNN.wav                             ..........  36.18%  692*

Compression:     36.18 %
Duration:         0.91 sec
Speed:          691.99 * real time

Re: WavPack 5.6.4 Has Multithreading Option

Reply #7 –
Tested the following, on one source file only (an AIFF available from https://soundcloud.com/kavakon/kava-kon-sakau-bar-heavy-mix )
Quote from: bryant on
For mono or stereo encoding, the output files probably will not be identical for different numbers of worker threads, however the compression ratios will be roughly similar.

I tested "all 364 possibilities" f, g, h, hh, -x0 to -x6, --threads=0 to --threads=12. With the reservation that follows from testing only one file, there is an unfortunate effect here:
The recommended "-x" is where the multi-threading does have adverse size impact.
Worst for -hx, any --threads (apart from 0) gave a size penalty of >2%
For -hhx, >1.34%
For -x, 0.66% to 1.34%
For -fx, 0.57% to 0.62%.
So yeah, well, -x is free but not free.
Also, for -x0, then any --threads=4 or above was within 11 ppm of minimum size, and better than --threads=0 (except two cherry-pickings: -hh beat --hh --threads=11 and --hh --threads=5 by eighteen resp. two bytes)

First, --threads=0 makes no difference (always gives bitwise-identical .wv files) to not using "--threads" at all. That is probably as you would expect.
But --threads=1 makes a difference.  Not sure if that was intended - it is still only one thread, right? And, typically it gave bigger files than --threads=0.  Not always though.

--threads=0 made for
* smallest files in the following settings: fx, fx2, x, x2, x4, x6, hx1, hx2, hx4, hx5, hx6, hhx1, hhx2
* BIGGEST files in the following settings: fx0 (+0.03% over smallest), x0 (+0.02%), x3, h (+0.02%), hx3, hhx3, hhx4 (+0.09%).  hhx4 is a setting people might want to use.

Settings where the "worst" makes a bigger impact over the "best", than the 0.09% of hhx4, are: the fx, x, hx, hhx as mentioned above, and anything with x2 (0.10% for fx2 to 0.90% for hx2) and [g|h]x{4|5|6], ranging +0.14% to +0.24%.

The CPU has 8 threads, so --threads=9 to --threads=12 are oddballs (I guess it is too much work to probe the hardware).  Anyway, one of those four settings -- indicated by 9, A, B and C in the table - make for smallest files in the following settings: fx3, x, x3, h, hh, hhx3, hhx4. Again, hhx4 is something that could make a difference.


File sizes follow, sorted. The command included -v and --import-id3, in case anyone gets nearly the same.
Code: [Select]
17966398	 -hh -x6 --threads4
17966450	 -hh -x6 --threads5
17967270	 -hh -x6 --threads6
17967560	 -hh -x6 --threads7
17967600	 -hh -x6 --threads8
17967668	 -hh -x6 --threads0
17967756	 -hh -x6 --threads2
17967958	 -hh -x6 --threadsC
17968006	 -hh -x6 --threads1
17968026	 -hh -x6 --threads9
17968488	 -hh -x6 --threads3
17968490	 -hh -x6 --threadsB
17968652	 -hh -x6 --threadsA
17979998	 -hh -x5 --threadsC
17980986	 -hh -x5 --threadsB
17982024	 -hh -x5 --threadsA
17982170	 -hh -x5 --threads9
17983078	 -hh -x5 --threads8
17983766	 -hh -x5 --threads7
17984294	 -hh -x5 --threads6
17985008	 -hh -x5 --threads4
17985234	 -hh -x5 --threads5
17987356	 -hh -x5 --threads0
17987624	 -hh -x5 --threads2
17987740	 -hh -x5 --threads1
17988302	 -hh -x5 --threads3
17995618	 -h -x6 --threads0
17996732	 -h -x6 --threads2
17997394	 -h -x6 --threads1
17998212	 -h -x6 --threads3
18002710	 -h -x5 --threads0
18003940	 -h -x5 --threads2
18004596	 -h -x5 --threads1
18005428	 -h -x5 --threads3
18006674	 -h -x6 --threads5
18011782	 -h -x5 --threads4
18012066	 -h -x5 --threads7
18012914	 -h -x6 --threads7
18015318	 -h -x5 --threads5
18015976	 -h -x5 --threads6
18016120	 -h -x6 --threads8
18017020	 -h -x6 --threads4
18017184	 -h -x5 --threads8
18019110	 -h -x5 --threads9
18023404	 -h -x5 --threadsA
18026424	 -h -x5 --threadsB
18028274	 -h -x6 --threads9
18028480	 -hh -x4 --threadsC
18029350	 -h -x5 --threadsC
18030972	 -hh -x4 --threadsB
18031734	 -h -x6 --threads6
18033154	 -hh -x4 --threadsA
18033974	 -h -x6 --threadsC
18035038	 -hh -x4 --threads9
18036618	 -hh -x4 --threads8
18037902	 -hh -x4 --threads7
18038380	 -h -x6 --threadsA
18039316	 -h -x6 --threadsB
18039978	 -hh -x4 --threads6
18040884	 -h -x4 --threads0
18041266	 -hh -x4 --threads5
18041314	 -h -x4 --threads1
18041348	 -h -x4 --threads2
18041436	 -h -x4 --threads3
18043078	 -hh -x4 --threads4
18044320	 -hh -x4 --threads2
18044324	 -hh -x4 --threads3
18044466	 -hh -x4 --threads1
18044554	 -h -x4 --threads4
18044598	 -hh -x4 --threads0
18046308	 -h -x4 --threads5
18047586	 -h -x4 --threads6
18048766	 -h -x4 --threads7
18050652	 -h -x4 --threads8
18057090	 -h -x4 --threads9
18058732	 -h -x4 --threadsA
18063230	 -h -x4 --threadsB
18068098	 -h -x4 --threadsC
18171702	 -g -x6 --threads0
18171988	 -g -x6 --threads1
18171990	 -g -x6 --threads3
18172008	 -g -x6 --threads2
18175502	 -g -x6 --threads4
18176508	 -g -x6 --threads5
18178420	 -g -x6 --threads6
18180804	 -g -x6 --threads7
18184430	 -g -x6 --threads8
18187504	 -g -x6 --threads9
18190894	 -g -x6 --threadsB
18191862	 -g -x6 --threadsC
18196432	 -g -x6 --threadsA
18206414	 -hh -x2 --threads0
18210310	 -hh -x3 --threadsB
18210328	 -hh -x3 --threads7
18210338	 -hh -x3 --threads5
18210342	 -hh -x3 --threads8
18210344	 -hh -x3 --threadsA
18210344	 -hh -x3 --threads9
18210368	 -hh -x3 --threadsC
18210380	 -hh -x3 --threads4
18210394	 -hh -x3 --threads6
18210586	 -hh -x3 --threads3
18210616	 -hh -x3 --threads2
18210766	 -hh -x3 --threads1
18211166	 -hh -x3 --threads0
18226536	 -hh -x1 --threads0
18260694	 -g -x5 --threads2
18260780	 -g -x5 --threads1
18261060	 -g -x5 --threads0
18261076	 -g -x5 --threads3
18268110	 -hh -x2 --threads1
18273464	 -hh -x2 --threads2
18277048	 -hh -x2 --threads3
18280678	 -g -x5 --threads6
18281794	 -g -x4 --threads0
18281950	 -g -x4 --threads1
18282254	 -g -x4 --threads2
18282620	 -g -x4 --threads3
18286700	 -g -x5 --threads5
18287026	 -hh -x2 --threads4
18287250	 -g -x5 --threads4
18289230	 -hh -x2 --threads9
18289744	 -hh -x2 --threads7
18289816	 -hh -x2 --threads5
18289874	 -hh -x2 --threads8
18290282	 -hh -x2 --threads6
18290642	 -g -x5 --threads7
18290838	 -hh -x2 --threadsC
18291320	 -hh -x2 --threadsB
18291568	 -hh -x2 --threadsA
18294614	 -g -x5 --threadsC
18294914	 -g -x5 --threads8
18295936	 -g -x5 --threads9
18299050	 -g -x4 --threads6
18299988	 -g -x5 --threadsB
18300490	 -g -x5 --threadsA
18301486	 -g -x4 --threads4
18304786	 -g -x4 --threads5
18305224	 -g -x4 --threads7
18310760	 -g -x4 --threadsC
18311384	 -g -x4 --threads8
18314142	 -g -x4 --threads9
18317336	 -g -x4 --threadsA
18319496	 -g -x4 --threadsB
18345958	 -h -x3 --threads7
18345960	 -h -x3 --threadsC
18345962	 -h -x3 --threadsB
18345968	 -h -x3 --threads9
18345992	 -h -x3 --threads6
18345994	 -h -x3 --threadsA
18345996	 -h -x3 --threads8
18346000	 -h -x3 --threads4
18346008	 -h -x3 --threads5
18346210	 -h -x3 --threads3
18346320	 -h -x3 --threads2
18346382	 -h -x3 --threads1
18346894	 -h -x3 --threads0
18365224	 -h -x2 --threads0
18371064	 -h -x1 --threads0
18452494	 -h -x2 --threads1
18471850	 -h -x2 --threads2
18480962	 -hh -x1 --threads1
18483476	 -hh -x1 --threads2
18483642	 -hh -x1 --threads3
18483940	 -h -x2 --threads3
18502496	 -hh -x1 --threads4
18502850	 -hh -x1 --threads7
18502892	 -hh -x1 --threads5
18502922	 -hh -x1 --threads6
18502994	 -hh -x1 --threads8
18503142	 -hh -x1 --threadsC
18503178	 -hh -x1 --threads9
18503206	 -hh -x1 --threadsA
18503392	 -hh -x1 --threadsB
18503564	 -hh -x0 --threads9
18503582	 -hh -x0 --threads4
18503646	 -hh -x0 --threads6
18503704	 -hh -x0 --threadsA
18503742	 -hh -x0 --threads3
18503744	 -hh -x0 --threadsC
18503750	 -hh -x0 --threads1
18503752	 -hh -x0 --threads8
18503766	 -hh -x0 --threads7
18503792	 -hh -x0 --threads0
18503794	 -hh -x0 --threads5
18503816	 -hh -x0 --threadsB
18503998	 -hh -x0 --threads2
18522798	 -h -x2 --threads5
18524750	 -h -x2 --threads6
18525096	 -h -x2 --threadsA
18525114	 -h -x2 --threads8
18525270	 -h -x2 --threads4
18525632	 -h -x2 --threads7
18528106	 -h -x2 --threadsB
18529356	 -h -x2 --threadsC
18529942	 -h -x2 --threads9
18621858	 -g -x3 --threadsC
18621870	 -g -x3 --threadsB
18621874	 -g -x3 --threads8
18621890	 -g -x3 --threads7
18621892	 -g -x3 --threads5
18621894	 -g -x3 --threads9
18621894	 -g -x3 --threads6
18621904	 -g -x3 --threadsA
18621914	 -g -x3 --threads4
18622202	 -g -x3 --threads3
18622430	 -g -x3 --threads2
18622590	 -g -x3 --threads1
18623474	 -g -x3 --threads0
18629112	 -g -x2 --threads0
18639844	 -g -x2 --threads1
18642336	 -g -x2 --threads2
18644196	 -g -x2 --threads3
18647808	 -g -x1 --threads0
18648506	 -g -x2 --threads7
18648642	 -g -x2 --threads8
18648766	 -g -x2 --threads4
18648806	 -g -x2 --threads5
18648864	 -g -x2 --threadsB
18648934	 -g -x2 --threads9
18648940	 -g -x2 --threads6
18649362	 -g -x2 --threadsC
18649390	 -g -x2 --threadsA
18767168	 -h -x1 --threads1
18767934	 -h -x1 --threads2
18768062	 -h -x1 --threads3
18770870	 -g -x1 --threads1
18788350	 -h -x1 --threads6
18788606	 -h -x1 --threads7
18788866	 -h -x1 --threads8
18788920	 -h -x1 --threadsC
18789020	 -h -x1 --threads5
18789166	 -h -x1 --threads4
18789500	 -h -x1 --threads9
18789600	 -h -x1 --threadsB
18789686	 -h -x1 --threadsA
18795458	 -h -x0 --threads9
18795510	 -h -x0 --threadsB
18795530	 -h -x0 --threads7
18795530	 -h -x0 --threadsA
18795552	 -h -x0 --threads8
18795580	 -h -x0 --threadsC
18795606	 -h -x0 --threads6
18795632	 -h -x0 --threads5
18795684	 -h -x0 --threads4
18796362	 -h -x0 --threads3
18797028	 -h -x0 --threads2
18797802	 -h -x0 --threads1
18799708	 -h -x0 --threads0
18809416	 -g -x1 --threads2
18831798	 -g -x1 --threads3
18892180	 -g -x1 --threads4
18893526	 -g -x1 --threads5
18896490	 -g -x1 --threads7
18896502	 -g -x1 --threads6
18897408	 -g -x1 --threads8
18897970	 -g -x1 --threadsA
18898090	 -g -x1 --threadsC
18898372	 -g -x1 --threads9
18898520	 -g -x1 --threadsB
18903674	 -g -x0 --threadsC
18903724	 -g -x0 --threadsB
18903732	 -g -x0 --threads8
18903778	 -g -x0 --threads9
18903792	 -g -x0 --threads7
18903794	 -g -x0 --threads5
18903830	 -g -x0 --threads6
18903858	 -g -x0 --threads4
18903880	 -g -x0 --threadsA
18904900	 -g -x0 --threads3
18905554	 -g -x0 --threads2
18905764	 -g -x0 --threads1
18907852	 -g -x0 --threads0
19217028	 -f -x6 --threads3
19217420	 -f -x6 --threads2
19217608	 -f -x6 --threads1
19218416	 -f -x6 --threads0
19226180	 -f -x6 --threads5
19227064	 -f -x6 --threads4
19229064	 -f -x6 --threads6
19229588	 -f -x6 --threads7
19231542	 -f -x6 --threads8
19231618	 -f -x6 --threads9
19231780	 -f -x6 --threadsA
19231802	 -f -x6 --threadsB
19231858	 -f -x6 --threadsC
19236974	 -f -x5 --threads1
19237248	 -f -x5 --threads3
19237432	 -f -x5 --threads0
19238492	 -f -x5 --threads2
19241172	 -f -x5 --threads6
19242336	 -f -x5 --threads5
19242442	 -f -x5 --threads4
19243552	 -f -x5 --threads7
19243578	 -f -x5 --threads9
19243616	 -f -x5 --threadsB
19243810	 -f -x5 --threads8
19244176	 -f -x5 --threadsA
19244806	 -f -x5 --threadsC
19245262	 -f -x4 --threads3
19246692	 -f -x4 --threads1
19246970	 -f -x4 --threads2
19248010	 -f -x4 --threads0
19249508	 -f -x4 --threads6
19250700	 -f -x4 --threads5
19251072	 -f -x4 --threadsB
19251670	 -f -x4 --threads4
19251788	 -f -x4 --threads7
19252452	 -f -x4 --threadsA
19252488	 -f -x4 --threads9
19252728	 -f -x4 --threads8
19252898	 -f -x4 --threadsC
19339416	 -f -x3 --threadsB
19339424	 -f -x3 --threads8
19339430	 -f -x3 --threadsC
19339434	 -f -x3 --threads5
19339440	 -f -x3 --threads6
19339448	 -f -x3 --threads7
19339450	 -f -x3 --threadsA
19339464	 -f -x3 --threads9
19339480	 -f -x3 --threads4
19339842	 -f -x3 --threads3
19340032	 -f -x3 --threads2
19340068	 -f -x3 --threads1
19340950	 -f -x3 --threads0
19356152	 -f -x2 --threads0
19368958	 -f -x2 --threads1
19370280	 -f -x2 --threads3
19371322	 -f -x2 --threads2
19374952	 -f -x2 --threads4
19375570	 -f -x2 --threads9
19375590	 -f -x2 --threads7
19375782	 -f -x2 --threadsC
19376148	 -f -x2 --threads5
19376278	 -f -x2 --threadsB
19376292	 -f -x2 --threads8
19376314	 -f -x1 --threads0
19376324	 -f -x2 --threads6
19376372	 -f -x2 --threadsA
19487358	 -f -x1 --threads1
19488134	 -f -x1 --threads2
19493332	 -f -x1 --threads3
19496996	 -f -x1 --threads6
19497002	 -f -x1 --threads8
19497006	 -f -x1 --threadsA
19497020	 -f -x1 --threads7
19497022	 -f -x1 --threadsB
19497024	 -f -x1 --threadsC
19497032	 -f -x1 --threads5
19497038	 -f -x1 --threads9
19497058	 -f -x1 --threads4
21372156	 -f -x0 --threadsC
21372156	 -f -x0 --threads8
21372218	 -f -x0 --threadsB
21372298	 -f -x0 --threadsA
21372326	 -f -x0 --threads6
21372328	 -f -x0 --threads7
21372342	 -f -x0 --threads9
21372458	 -f -x0 --threads4
21372488	 -f -x0 --threads5
21374204	 -f -x0 --threads3
21374966	 -f -x0 --threads2
21375374	 -f -x0 --threads1
21378356	 -f -x0 --threads0
46772270	 aiff