Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multi-threading Encoding on WavPack (Read 5198 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multi-threading Encoding on WavPack

Hello David,
 I wonder is it able for wavpack to use multi-threading encoding?
Since covert single file it looks like only use 1 thread . And I saw some other formats use multi-threading.
(maybe add a -mt option to enable multi-thread if possible )

Although you said that you don't plan to improve performance, I'm still curious whether there are ways to increase performance. (SSE2 AVX2  or somewhat) But I think the multi-thread is the most critical.


In my PC (i7-8700K oc 4.8ghz) Convert a single file to wavpack is only using 11.5% of the cpu resource and the speed is around 4.5x.
As the Comparison,in the same situation, the speed of flac is 317x! (only use one core though)

Re: Multi-threading Encoding

Reply #1
Why don't you encode more than one file?

A host like foobar is able to provide as much threads as the CPU offers for anything you give it to convert. Meaning: if you have 8 threads you can encode 8 files at the same time.
marlene-d.blogspot.com

Re: Multi-threading Encoding on WavPack

Reply #2
Why don't you encode more than one file?

A host like foobar is able to provide as much threads as the CPU offers for anything you give it to convert.
Yes I always do it. But sometimes we need to convert file that is less than the core number (I have a 6c12t cpu). Especially converting single file is slow on WavPack. I saw the TAK encoder released today has the feature multi-thread encoding so I wonder is it too hard for wavpack to implement. This feature can let us always use more cores.

Re: Multi-threading Encoding on WavPack

Reply #3
Hello David,
 I wonder is it able for wavpack to use multi-threading encoding?
Since covert single file it looks like only use 1 thread . And I saw some other formats use multi-threading.

It's usually possible to add this to encoders, but usually a bad idea to use since running multiple encoders in parallel is faster than trying to parallelize one instance.

Re: Multi-threading Encoding on WavPack

Reply #4
In my PC (i7-8700K oc 4.8ghz) Convert a single file to
Hello David,
 I wonder is it able for wavpack to use multi-threading encoding?
Since covert single file it looks like only use 1 thread . And I saw some other formats use multi-threading.

It's usually possible to add this to encoders, but usually a bad idea to use since running multiple encoders in parallel is faster than trying to parallelize one instance.

Yes but I usually convert not too many songs at once. For example a single cd is 4 songs. Lots of albums is 12 songs.
If use multi-instance for example 12 songs can boost the speed at first but after the song is less than 12, my PC has some idle resource.

In my PC (i7-8700K oc 4.8ghz) Convert a single file to wavpack is only using 11.5% of the cpu resource and the speed is around 4.5x. The speed of flac is 317x!
Since the hexa-core and the octa-core would be the mainstream I think it is good to add this feature for the option.

Re: Multi-threading Encoding on WavPack

Reply #5
Hello David,
 I wonder is it able for wavpack to use multi-threading encoding?
Since covert single file it looks like only use 1 thread . And I saw some other formats use multi-threading.
(maybe add a -mt option to enable multi-thread if possible )

Although you said that you don't plan to improve performance, I'm still curious whether there are ways to increase performance. (SSE2 AVX2  or somewhat) But I think the multi-thread is the most critical.


In my PC (i7-8700K oc 4.8ghz) Convert a single file to wavpack is only using 11.5% of the cpu resource and the speed is around 4.5x.
As the Comparison,in the same situation, the speed of flac is 317x! (only use one core though) 70 times faster.


Re: Multi-threading Encoding on WavPack

Reply #6
In my PC (i7-8700K oc 4.8ghz) Convert a single file to wavpack is only using 11.5% of the cpu resource and the speed is around 4.5x. The speed of flac is 317x!
Since the hexa-core and the octa-core would be the mainstream I think it is good to add this feature for the option.

4.5 encoding speed? You should have mentioned that you use WavPack Hybrid (lossy). In lossy mode, WavPack works completely different compared to FLAC. You're comparing Apples to Oranges. In any case, as Saratoga already said, it is wiser to use multiple encoders in parallel.
marlene-d.blogspot.com

Re: Multi-threading Encoding on WavPack

Reply #7
In my PC (i7-8700K oc 4.8ghz) Convert a single file to wavpack is only using 11.5% of the cpu resource and the speed is around 4.5x.
It doesn't make sense to use the slowest possible mode of Wavpack if you care about encoding speed.

Re: Multi-threading Encoding on WavPack

Reply #8
 I use the latest 64bit binary with the lossless mode with the best compression. And the flac testing result aboved also use the level 8 (slowest mode).

Re: Multi-threading Encoding on WavPack

Reply #9
Hello David,
 I wonder is it able for wavpack to use multi-threading encoding?
Since covert single file it looks like only use 1 thread . And I saw some other formats use multi-threading.
(maybe add a -mt option to enable multi-thread if possible )

Although you said that you don't plan to improve performance, I'm still curious whether there are ways to increase performance. (SSE2 AVX2  or somewhat) But I think the multi-thread is the most critical.


In my PC (i7-8700K oc 4.8ghz) Convert a single file to wavpack is only using 11.5% of the cpu resource and the speed is around 4.5x.
As the Comparison,in the same situation, the speed of flac is 317x! (only use one core though)


Why would you need multithreading if you can use Foobar?
When you start foorbar you can even select specific CPU's in your taskmanager and the job gets balanced over these cores. You can also balance this out via a command line option. Too many simultaneously running encodes can introduce lack on the harddrive and fragmentation of files.
Only when encoding single huge files speed could be a 'problem'.

Even in normal compression programs like 7zip multithreading hurts compression ratio, because per solid block you cannot split the process so separate blocks are created.

PS. If wavpack is that slow, you have some extreme compression settings set. This is not comparable. Even at high speed Wavpack creates smaller files most of the time.

Re: Multi-threading Encoding on WavPack

Reply #10
As others have said, the best way to parallelize WavPack encoding is by file, but I have thought about how it might be handled in the command-line program. The faster modes won't work well because the output of each block provides clues for the next block, and these would be lost with multiple threads. However, the -X4 to -X6 pure lossless modes are performed independently, and so those would be good candidates (and also already quite slow).

And of course the decoding could be in parallel threads because those are always independent, but except for weird situations the decoding speed is never an issue.

Unfortunately all of this is academic because I have no free time, but this is on my list of possible projects when time permits.

Re: Multi-threading Encoding on WavPack

Reply #11
I see. Thanks for the reply !