Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: WavPack 5.6.4 Has Multithreading Option (Read 11393 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #25 –
Quote from: bryant on
Also, there's a preprint available of the "brainwave" compression paper.
The paper mentions the use of two different ADCs and the NP1 is "high-pass". Do you know the specs? For example, does the ADC use bandpass sampling or just typical sampling from DC to Nyquist but with a highpass filter applied?

[edit]OK, the high-pass is not very high so unlikely to use bandpass sampling.
Quote
NP1 devices include an analog 300-Hz high-pass filter [...] which is sampled at 30 kHz.

Also, the paper emphasizes "accurate reproduction of fast transients (i.e., spikes)" and due to the non-audio nature I guess the differences between x4 and x3 should be very big. Do you have some of their data to try out, or did you give some advice to them about how to encode the data efficiently in terms of size vs speed?

Re: WavPack 5.6.4 Has Multithreading Option

Reply #26 –
Quote from: bennetng on
Also, the paper emphasizes "accurate reproduction of fast transients (i.e., spikes)" and due to the non-audio nature I guess the differences between x4 and x3 should be very big. Do you have some of their data to try out, or did you give some advice to them about how to encode the data efficiently in terms of size vs speed?
They did give me data to experiment with, although not until just recently. What I was surprised about was how noisy the signal is and how sparse the spikes are. In fact, some of the 384 channels seemed to be, just looking in an editor and listening, completely free of spikes and consisted of just white noise, with a few constant low-level sines added in probably from power supplies (e.g., 60 Hz, 10,070 Hz, 14,570 Hz).

Needless to say, the higher modes and the extra modes made very little difference, like 1.5% from -f to -hhx6, and all of that improvement would have been from picking up those sines. So unless some other data samples look significantly different, the extra modes are not going to achieve much, especially considering their cost. In fact, I think they ended up deciding that even the "high" mode was unjustified.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #27 –
Quote from: bryant on
Needless to say, the higher modes and the extra modes made very little difference, like 1.5% from -f to -hhx6, and all of that improvement would have been from picking up those sines. So unless some other data samples look significantly different, the extra modes are not going to achieve much, especially considering their cost. In fact, I think they ended up deciding that even the "high" mode was unjustified.

Hm, ... at the risk of making a fool out of myself by miscounting figures after the decimal point:
I look at lines 550 to 555 of the PDF version: blosc-zstd would be cheaper if decompressed 528 times blah blah. How they got to that:
If I understand the numbers right, 5 cents per CPU hour and a speed of 12.82 x RT, encoding this hour of data costs only 0.4 cent (and so does every decoding)? A one-time -x makes for ... how much, fifty percent more time? 0.2 cents? That is like 1/30th of a percent of a year's 615 cents storage cost, so if -x reduces the file by 0.1 percent it is regained after four months in the cloud?

The benefits of -h would depend on the number of decompressions of course.

Mildly surprising at first glance that they measure WavPack to decoding faster than FLAC - but then on the other hand, how did they even make FLAC decode 384 channels? Too lazy to dig into the details, and so should they be, when a format that supports it is put up against one that doesn't ...

Re: WavPack 5.6.4 Has Multithreading Option

Reply #28 –
The cloud cost calculations are a mystery to me, but I assume the authors know what they’re talking about (I certainly don’t). Specifically, it seems to me that CPU cost would depend on the number of threads consumed (i.e., the “total” CPU power cost), not just the user time that a process took, but again I really have no clue how that’s billed (and it probably varies from service to service).

As for the comparison of WavPack and FLAC, there is some other code in there that flattens the 384 channels into stereo pairs for FLAC, which of course is essentially what WavPack does. And, like I recently verified, “high” mode in WavPack makes essentially no difference (0.16% on the file they provided).

WavPack decodes faster than FLAC because of the multithreading I recently added (which works really well for multichannel). It originally was slower by the amount one would expect, but I noticed that some of the general-purpose compressors that were getting amazing decoding speeds were natively multithreaded, and I thought that a little unfair.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #29 –
Quote from: bryant on
And, like I recently verified, “high” mode in WavPack makes essentially no difference (0.16% on the file they provided).
Essentially no difference is ... relative. 0.16 percent - not percentage points - saves 1 cent annual storage cost for that amount of data. Quite a lot compared to differences in encoding CPU costs (in their figures). If a normal-mode encode/decode costs 0.4 and -h is a factor of 1.3 (I think that is sane for CDDA, but ...) then the 1 cent breaks even at seven decodes a year. Not saying it is worth it, but my point is that if storage costs so much more than processing, the "essentially no difference" will be weighted up accordingly.

Anyway, if I were referee on that part of the paper, I would be very fine with not trying to tweak one of the compressors. Fine-tuning for whatever optimum would be subject to pricing plan (or even does encoding locally), blah blah you know all this of course.
(Still to a reader who don't know audio codecs at all, there is a point that there are settings available that incur only one-time costs upon encoding and do not cost decoding effort.)

Re: WavPack 5.6.4 Has Multithreading Option

Reply #30 –
Quote from: bryant on
Windows only? I compiled a fresh binary (git master) on Ubuntu 22.04 using the:
Code: [Select]
-DWAVPACK_ENABLE_THREADS:BOOL=ON
and when I try to encode a wav file I get a warning:
Code: [Select]
--threads not enabled, ignoring option!
What's my mistake?

Re: WavPack 5.6.4 Has Multithreading Option

Reply #31 –
You made no mistake; it was me...   :(

I don't usually use cmake, and the problem is there. With the enable-threads option enabled (which is the default, no need to specify) libwavpack is built with thread support, but not the command-line programs. Even though I'm completely cmake-incompetent, I was able to fix this and will check in a fix shortly.

In the meantime, using Autotools works fine (i.e., ./autogen.sh, make, etc.)

Thanks for bringing this to my attention!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #32 –
Thank you! I'll test it.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #34 –
I recompiled and confirmed: everything works. The increase in encoding speed is obvious even when using
Code: [Select]
-hhx6
option. Great!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #35 –
Quote from: greenvorbisfan on
I recompiled and confirmed: everything works. The increase in encoding speed is obvious even when using
Code: [Select]
-hhx6
option. Great!
Yeah, a speedup of --hx6 was one of the goals (along with decoding, of course). Thanks for letting me know my fix worked!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #36 –
A related question arose: is it possible to introduce a threads option into the wvgain binary? This would greatly speed up scanning of heavy WavPack-DSD files… Thanks!

Re: WavPack 5.6.4 Has Multithreading Option

Reply #37 –
Quote from: greenvorbisfan on
A related question arose: is it possible to introduce a threads option into the wvgain binary? This would greatly speed up scanning of heavy WavPack-DSD files… Thanks!
It's already there: -t

I don't know about greatly sped up, but I seem to remember it doubling the speed.

Of course, the ReplayGain processing portion is not multithreaded (just the decoding and conversion to PCM). Doing the whole thing multithreaded is more complicated.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #38 –
Wow! I guess I wasn't careful when reading the information in --help  :D  Sorry  :)

Re: WavPack 5.6.4 Has Multithreading Option

Reply #39 –
Quote from: greenvorbisfan on
Wow! I guess I wasn't careful when reading the information in --help  :D  Sorry  :)
Haha, don't be sorry!  "It already works" is my favorite answer!   :D

Re: WavPack 5.6.4 Has Multithreading Option

Reply #40 –
Hi everyone, sorry for a question, I'm using BatchEncoder on Windows 11 to transform dsf files into wv, if I wanted to do the reverse operation, i.e. bring the wv back into dsf how should I do it?
Thank you

Re: WavPack 5.6.4 Has Multithreading Option

Reply #41 –
I don't know BatchEncoder, but try search them up, and drag and drop them onto the wvunpack.exe file icon?

Re: WavPack 5.6.4 Has Multithreading Option

Reply #42 –
It seem I have found a bug or something related to --threads. LossyWAV compressed in lossless mode by WavPack becomes very small and sounds glitchy. To get the expected fine sounding result I have either to remove --threads or to remove --blocksize=512 --merge-blocks, but the latter goes against the official instruction.

Code: [Select]
$ sox vangelis.wav cut.wav trim 18 14 fade t 1 -0 2

$ lossywav cut.wav -s h -A
lossyWAV 1.4.2, Copyright (C) 2007-2016 Nick Currie. Copyleft.
Settings  : --quality standard --shaping hybrid --altspread
File Info : 44.10kHz; 2 channel; 16 bit; 00:14.00, 2.355MiB
Results   : 1.7040 bits; 15.36x; 00:00.91; [F]
Feedback  : None.

$ wavpack --blocksize=512 --merge-blocks --threads cut.lossy.wav
WAVPACK  Hybrid Lossless Audio Compressor  Win64 Version 5.6.6
Copyright (c) 1998 - 2023 David Bryant.  All Rights Reserved.
created cut.lossy.wv in 0.09 secs (lossless, 86.00%)

• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: WavPack 5.6.4 Has Multithreading Option

Reply #43 –
Dear @bryant, you've probably read the comparison of Wavpack 3.97 Hybrid with Optimfrog Dualstream provided by the author of the latter. Please tell us how relevant those bullet points today. I believe your tireless work on improving the codec has taken some of that into account.

Code: [Select]
Advantages of OptimFROG DualStream compared to WavPack hybrid:
- distortions up to 15 times lower during sharp transients
- slightly lower distortions during quasi-stationary areas
- true separate quantization levels for each channel
- quality mode maintains constant quality thorough the whole file
- no overshots or postecho
- advanced noise shaping option, improving transparency
- up to 3% better lossless compression for OFS+OFC over WV+WVC
- fast seek, supported by the compressed file format
- error resilience, supported by the compressed file format
- exact bitrate control when using average bitrate
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: WavPack 5.6.4 Has Multithreading Option

Reply #44 –
Quote from: Kraeved on
It seem I have found a bug or something related to --threads. LossyWAV compressed in lossless mode by WavPack becomes very small and sounds glitchy. To get the expected fine sounding result I have either to remove --threads or to remove --blocksize=512 --merge-blocks, but the latter goes against the official instruction.
Yes, I discovered this too and fixed it in November. Unfortunately that was after the last unofficial release, however the fix is to simply not allow threading with --blocksize, so just omitting --threads should do the trick. Thanks for the head-up though!

GitHub commit

Re: WavPack 5.6.4 Has Multithreading Option

Reply #45 –
Quote from: Kraeved on
Dear @bryant, you've probably read the comparison of Wavpack 3.97 Hybrid with Optimfrog Dualstream provided by the author of the latter. Please tell us how relevant those bullet points today. I believe your tireless work on improving the codec has taken some of that into account.

Code: [Select]
Advantages of OptimFROG DualStream compared to WavPack hybrid:
- distortions up to 15 times lower during sharp transients
- slightly lower distortions during quasi-stationary areas
- true separate quantization levels for each channel
- quality mode maintains constant quality thorough the whole file
- no overshots or postecho
- advanced noise shaping option, improving transparency
- up to 3% better lossless compression for OFS+OFC over WV+WVC
- fast seek, supported by the compressed file format
- error resilience, supported by the compressed file format
- exact bitrate control when using average bitrate
WavPack was essentially rewritten from scratch for version 4.0 and specifically the hybrid mode was greatly improved (even though in a very general way it's similar to the old version). And of course, things like seeking and error resilience were notably added to WavPack in the rewrite.

Since OptimFROG is a highly-optimized codec my guess would be that its DualStream would still slightly out-perform WavPack in absolute quality (S/N ratio vs. bitrate) just like its lossless mode does, but of course at a correspondingly slower speed.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #46 –
Dear @bryant, our community has found a techno sample that puts Wavpack Hybrid and many other lossy codecs to shame. It seemed to me that I finally found an acceptable, easily memorable setting for a vintage computer, namely -b4x or even -b4x3, which scales well with CD audio and Hi-Res, so you do not need to change the bitrate yourself when input switches from 44 kHz 16 bit to, for example, 48 kHz 24 bit, but now I feel lost. One of the prominent users, @shadowking, suggests using -s 0.5 to make all that noisy sand at the beginning of the file less audible, but perhaps you can suggest something else that will bring back the joy of audio encoding?
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data

Re: WavPack 5.6.4 Has Multithreading Option

Reply #47 –
So I spent some time analyzing that sample. I isolated a period from 0.674 – 0.837 seconds because I think that’s where the noise is most audible. The spectrum of this section shows a burst of noise centered at around 15 kHz (which the ear is not very sensitive to) which tapers off quickly to a dip around 5 kHz (which is just above the frequency we are most sensitive to). The peak to trough is around 38 dB, making this an extremely difficult sample for any simple time-domain codec!

Encoding this with WavPack lossy at 4 bits per sample (no other options) shows the following spectrum with the signal in cyan and the noise in magenta. The area where the noise is louder than the signal is where it’s most audible. In this time period the dynamic noise shaping function (DNS) should be tilting the noise upward, but doesn’t (it’s a simple algorithm and in this case the high level at lower frequencies confuses it). There’s a quiet section just before this where the DNS does work correctly, but not here:



Adding the -s1 option forces the noise shaping to 1st order positive during this section (because it turns off DNS completely) and we see this spectrum. Note that the noise appearing above the signal is much less (and I can no longer hear it here):



Here’s a comparison showing the DNS-selected noise in cyan compared to the fixed -s1 noise in magenta to show how the shaping moves the noise:



What would be better is 2nd-order noise shaping or even better would be a noise-shaper peaking at 15 kHz (neither of which WavPack supports). Another thing that would work would be simply detecting that the noise level was above the signal at audible frequencies and boosting the bitrate at that point (I think lossyWAV does something like that with this sample). These are all possible improvements to WavPack lossy that I wish I had time for. It is what it is.  :)

The problem with overriding the DNS with the -s option is that there are samples that benefit greatly from dynamic shaping and the negative sloping noise. Material with relatively little high-frequencies (like most “acoustic music”, one of my favorite retronyms) will end up with audible HF hiss when using -s1, assuming the listener does not have age-related high-frequency hearing loss (like me).

In conclusion, I don’t think it makes a lot of sense to choose a codec or codec settings based on this sample, or any single sample. It was almost certainly not crafted intentionally to throw off lossy codecs, but looking at it one could suspect that. And it’s certainly possible for an even more difficult “naturally occurring” sample to be found one day, but that doesn’t change the vast majority of encodings that are perfectly transparent.

Re: WavPack 5.6.4 Has Multithreading Option

Reply #48 –
Thanks for the explanation David.  Fortunately this sample responds well to -hx6 or  better -hhx6 .  Except DNS
 and even -s0 doesn't work too good .  I agree -s1 is something that always stood out for me . Not really bad but a bit
too correlated to the signal . -s.5 is more natural and 'spread out' not as much as -s0 but enough to go undetected and even then still sounds good.
 

Re: WavPack 5.6.4 Has Multithreading Option

Reply #49 –
Quote from: bryant on
What would be better is 2nd-order noise shaping or even better would be a noise-shaper peaking at 15 kHz (neither of which WavPack supports). Another thing that would work would be simply detecting that the noise level was above the signal at audible frequencies and boosting the bitrate at that point (I think lossyWAV does something like that with this sample). These are all possible improvements to WavPack lossy that I wish I had time for.

Dear @bryant, thank you for taking the user concerns related to the use of WavPack seriously. Based on your detailed explanation, I see that the hybrid mode needs to be updated in the noise department. Somehow this reminded me of your ART resampler, which you limited to TPDF dithering, so I had to try another one with noise-shaping profiles. Let's pray that one day, perhaps in v6.0, we'll see an improvement that will make the hybrid mode more shrewd. Godspeed.
• Join our efforts to make Helix MP3 encoder great again
• Opus complexity & qAAC dependence on Apple is an aberration from Vorbis & Musepack breakthroughs
• Let's pray that D. Bryant improve WavPack hybrid, C. Helmrich update FSLAC, M. van Beurden teach FLAC to handle non-audio data