HydrogenAudio

Lossless Audio Compression => WavPack => Topic started by: gl.tter on 2008-08-08 17:56:25

Title: CheckWavpackFiles 2.0 BETA released
Post by: gl.tter on 2008-08-08 17:56:25
NEW v2.0 BETA2 Version - now with multi-processor/core support![/i]

'CheckWavpackFiles' is a free standalone and easy-to-use Wavpack file scanner for Windows.

    * check single Wavpack files, or all .wv files in a file tree or drive for errors.
    * easy two-click operation via drag & drop or Explorer shell extension.
    * bad files are written to a text file for reference/further processing.
    * no command-line skills required (although you can use that too).

http://wavpack.gl.tter.org (http://wavpack.gl.tter.org)

This new beta version adds multi-core/processor support for faster scans & updates to Wavpack 4.50 code.  Let me know how it works for you.

gl.tter
Title: CheckWavpackFiles 2.0 BETA released
Post by: gl.tter on 2008-08-09 11:43:05
Updated to 2.0 Beta2.

Fixes several issues (display corruption, recycle bin folder enumeration & more).

gl.tter
Title: CheckWavpackFiles 2.0 BETA released
Post by: bryant on 2008-08-12 05:38:00
Thanks, gl.tter, this is very cool!

Guess I'll have to get a multicore system to test it thoroughly! 
Title: CheckWavpackFiles 2.0 BETA released
Post by: esa372 on 2008-08-12 16:50:36
Excellent!

Thank you, gl.tter!

Title: CheckWavpackFiles 2.0 BETA released
Post by: gl.tter on 2008-08-12 20:11:49
Thanks guys.  It's kinda fun to watch 4 files being checked at once.

Unfortunately there's a limiting factor, and that's the disk access time.  With multiple files (and even worse if the disk is also fragmented), the disk heads are jumping all over the place - it can be slower than the previous single-file version.  On the fastest of my SATA disks I get a good 50-80% CPU utilisation (Q6600 quad-core @ 3.4GHz), but on the other there's almost none at all due to the trashing.

One idea I'm looking into for the next version is to cache the file accesses more intelligently.  A better solution though would be to multi-processor enable the decoding parts of the library, so that consecutive blocks of a single file are transparently processed in parallel & cached for when they're needed (I've mentioned this before).  David, have you ever looked into doing that?
Title: CheckWavpackFiles 2.0 BETA released
Post by: bryant on 2008-08-14 13:33:15
One idea I'm looking into for the next version is to cache the file accesses more intelligently.  A better solution though would be to multi-processor enable the decoding parts of the library, so that consecutive blocks of a single file are transparently processed in parallel & cached for when they're needed (I've mentioned this before).  David, have you ever looked into doing that?

There's no reason why it wouldn't work, although for decoding I wonder if the disk I/O would ever be able to keep up (or justify it), although I guess it would certainly eliminate the thrashing problem.

Also, for encoding (especially the -x modes) doing the same thing would certainly be a gain.

It would probably be a better use of time than going to hand-coded assembler for the single threads (which is another thing I think about), but unfortunately I don't have time at this point to look into either. I am happy to kibitz, however... 
Title: CheckWavpackFiles 2.0 BETA released
Post by: gl.tter on 2008-08-14 13:59:00
There's no reason why it wouldn't work, although for decoding I wonder if the disk I/O would ever be able to keep up (or justify it)


It will, as I said, I can get 50-80% CPU utilisation scanning 4 files simultaneously on a quad-core @ 3.4GHz (that figure is a across all 4 cores) - the only reason it's not 100% is that I'm disk starved from the trashing.  Frequent disk seeks drastically reduce throughput, as we all know when we get pagefile trashing & everything grinds to a halt.  With linear single-file streaming I could easily max my CPU and then some.

Quote
Also, for encoding (especially the -x modes) doing the same thing would certainly be a gain.


Sure, but probably a little more complex (due to not knowing how much compresses into each block).

Quote
It would probably be a better use of time than going to hand-coded assembler for the single threads (which is another thing I think about), but unfortunately I don't have time at this point to look into either. I am happy to kibitz, however... 


kibitz? : )  I can provide all the threading and CPU code - in fact if you could split the decoding context into independent block-specific ones (ie. ones that have all the decoding state needed to decode an entire block in isolation), I can do the rest.  With multi-cores so widespread now, this not only gets you a massive bang-for-buck, but better yet all apps benefit transparently.
Title: CheckWavpackFiles 2.0 BETA released
Post by: bryant on 2008-08-15 06:23:10
kibitz? : )  I can provide all the threading and CPU code - in fact if you could split the decoding context into independent block-specific ones (ie. ones that have all the decoding state needed to decode an entire block in isolation), I can do the rest.  With multi-cores so widespread now, this not only gets you a massive bang-for-buck, but better yet all apps benefit transparently.

I think the functionality you're looking for already exists in the current library, although it's a little convoluted to get to it. You just need to start up a WavPack decoder context for each thread you want to have, and set the OPEN_STREAMING flag for each one.

The main code would read the WavPack file and parse it into blocks, then pass the blocks (using the WavpackStreamReader callbacks) to the decoding contexts to convert a single block each. Using the OPEN_STREAMING flag makes the contexts not care that the data they suck in is from discontiguous blocks (although your parser must make sure they do get complete blocks). The WavPack DirectShow filter uses this same method (not to use multiple cores but to implement the parsing/seeking separate from the actual decoding).
Title: CheckWavpackFiles 2.0 BETA released
Post by: gl.tter on 2008-08-15 11:41:45
I think the functionality you're looking for already exists in the current library, although it's a little convoluted to get to it. You just need to start up a WavPack decoder context for each thread you want to have, and set the OPEN_STREAMING flag for each one.

The main code would read the WavPack file and parse it into blocks, then pass the blocks (using the WavpackStreamReader callbacks) to the decoding contexts to convert a single block each. Using the OPEN_STREAMING flag makes the contexts not care that the data they suck in is from discontiguous blocks (although your parser must make sure they do get complete blocks). The WavPack DirectShow filter uses this same method (not to use multiple cores but to implement the parsing/seeking separate from the actual decoding).


Cool, I'll look into it.