Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: WavPack 5.6.4 Has Multithreading Option (Read 7044 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #25 –
Quote from: bryant on
Also, there's a preprint available of the "brainwave" compression paper.
The paper mentions the use of two different ADCs and the NP1 is "high-pass". Do you know the specs? For example, does the ADC use bandpass sampling or just typical sampling from DC to Nyquist but with a highpass filter applied?

[edit]OK, the high-pass is not very high so unlikely to use bandpass sampling.
Quote
NP1 devices include an analog 300-Hz high-pass filter [...] which is sampled at 30 kHz.

Also, the paper emphasizes "accurate reproduction of fast transients (i.e., spikes)" and due to the non-audio nature I guess the differences between x4 and x3 should be very big. Do you have some of their data to try out, or did you give some advice to them about how to encode the data efficiently in terms of size vs speed?

Re: WavPack 5.6.4 Has Multithreading Option

Reply #26 –
Quote from: bennetng on
Also, the paper emphasizes "accurate reproduction of fast transients (i.e., spikes)" and due to the non-audio nature I guess the differences between x4 and x3 should be very big. Do you have some of their data to try out, or did you give some advice to them about how to encode the data efficiently in terms of size vs speed?
They did give me data to experiment with, although not until just recently. What I was surprised about was how noisy the signal is and how sparse the spikes are. In fact, some of the 384 channels seemed to be, just looking in an editor and listening, completely free of spikes and consisted of just white noise, with a few constant low-level sines added in probably from power supplies (e.g., 60 Hz, 10,070 Hz, 14,570 Hz).

Needless to say, the higher modes and the extra modes made very little difference, like 1.5% from -f to -hhx6, and all of that improvement would have been from picking up those sines. So unless some other data samples look significantly different, the extra modes are not going to achieve much, especially considering their cost. In fact, I think they ended up deciding that even the "high" mode was unjustified.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #27 –
Quote from: bryant on
Needless to say, the higher modes and the extra modes made very little difference, like 1.5% from -f to -hhx6, and all of that improvement would have been from picking up those sines. So unless some other data samples look significantly different, the extra modes are not going to achieve much, especially considering their cost. In fact, I think they ended up deciding that even the "high" mode was unjustified.

Hm, ... at the risk of making a fool out of myself by miscounting figures after the decimal point:
I look at lines 550 to 555 of the PDF version: blosc-zstd would be cheaper if decompressed 528 times blah blah. How they got to that:
If I understand the numbers right, 5 cents per CPU hour and a speed of 12.82 x RT, encoding this hour of data costs only 0.4 cent (and so does every decoding)? A one-time -x makes for ... how much, fifty percent more time? 0.2 cents? That is like 1/30th of a percent of a year's 615 cents storage cost, so if -x reduces the file by 0.1 percent it is regained after four months in the cloud?

The benefits of -h would depend on the number of decompressions of course.

Mildly surprising at first glance that they measure WavPack to decoding faster than FLAC - but then on the other hand, how did they even make FLAC decode 384 channels? Too lazy to dig into the details, and so should they be, when a format that supports it is put up against one that doesn't ...

Re: WavPack 5.6.4 Has Multithreading Option

Reply #28 –
The cloud cost calculations are a mystery to me, but I assume the authors know what they’re talking about (I certainly don’t). Specifically, it seems to me that CPU cost would depend on the number of threads consumed (i.e., the “total” CPU power cost), not just the user time that a process took, but again I really have no clue how that’s billed (and it probably varies from service to service).

As for the comparison of WavPack and FLAC, there is some other code in there that flattens the 384 channels into stereo pairs for FLAC, which of course is essentially what WavPack does. And, like I recently verified, “high” mode in WavPack makes essentially no difference (0.16% on the file they provided).

WavPack decodes faster than FLAC because of the multithreading I recently added (which works really well for multichannel). It originally was slower by the amount one would expect, but I noticed that some of the general-purpose compressors that were getting amazing decoding speeds were natively multithreaded, and I thought that a little unfair.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #29 –
Quote from: bryant on
And, like I recently verified, “high” mode in WavPack makes essentially no difference (0.16% on the file they provided).
Essentially no difference is ... relative. 0.16 percent - not percentage points - saves 1 cent annual storage cost for that amount of data. Quite a lot compared to differences in encoding CPU costs (in their figures). If a normal-mode encode/decode costs 0.4 and -h is a factor of 1.3 (I think that is sane for CDDA, but ...) then the 1 cent breaks even at seven decodes a year. Not saying it is worth it, but my point is that if storage costs so much more than processing, the "essentially no difference" will be weighted up accordingly.

Anyway, if I were referee on that part of the paper, I would be very fine with not trying to tweak one of the compressors. Fine-tuning for whatever optimum would be subject to pricing plan (or even does encoding locally), blah blah you know all this of course.
(Still to a reader who don't know audio codecs at all, there is a point that there are settings available that incur only one-time costs upon encoding and do not cost decoding effort.)

Re: WavPack 5.6.4 Has Multithreading Option

Reply #30 –
Quote from: bryant on
Windows only? I compiled a fresh binary (git master) on Ubuntu 22.04 using the:
Code: [Select]
-DWAVPACK_ENABLE_THREADS:BOOL=ON
and when I try to encode a wav file I get a warning:
Code: [Select]
--threads not enabled, ignoring option!
What's my mistake?

Re: WavPack 5.6.4 Has Multithreading Option

Reply #31 –
You made no mistake; it was me...   :(

I don't usually use cmake, and the problem is there. With the enable-threads option enabled (which is the default, no need to specify) libwavpack is built with thread support, but not the command-line programs. Even though I'm completely cmake-incompetent, I was able to fix this and will check in a fix shortly.

In the meantime, using Autotools works fine (i.e., ./autogen.sh, make, etc.)

Thanks for bringing this to my attention!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #32 –
Thank you! I'll test it.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #34 –
I recompiled and confirmed: everything works. The increase in encoding speed is obvious even when using
Code: [Select]
-hhx6
option. Great!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #35 –
Quote from: greenvorbisfan on
I recompiled and confirmed: everything works. The increase in encoding speed is obvious even when using
Code: [Select]
-hhx6
option. Great!
Yeah, a speedup of --hx6 was one of the goals (along with decoding, of course). Thanks for letting me know my fix worked!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #36 –
A related question arose: is it possible to introduce a threads option into the wvgain binary? This would greatly speed up scanning of heavy WavPack-DSD files… Thanks!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #37 –
Quote from: greenvorbisfan on
A related question arose: is it possible to introduce a threads option into the wvgain binary? This would greatly speed up scanning of heavy WavPack-DSD files… Thanks!
It's already there: -t

I don't know about greatly sped up, but I seem to remember it doubling the speed.

Of course, the ReplayGain processing portion is not multithreaded (just the decoding and conversion to PCM). Doing the whole thing multithreaded is more complicated.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #38 –
Wow! I guess I wasn't careful when reading the information in --help  :D  Sorry  :)

Re: WavPack 5.6.4 Has Multithreading Option

Reply #39 –
Quote from: greenvorbisfan on
Wow! I guess I wasn't careful when reading the information in --help  :D  Sorry  :)
Haha, don't be sorry!  "It already works" is my favorite answer!   :D