Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: CheckWavpackFiles 2.0 BETA released (Read 9531 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

CheckWavpackFiles 2.0 BETA released

NEW v2.0 BETA2 Version - now with multi-processor/core support![/i]

'CheckWavpackFiles' is a free standalone and easy-to-use Wavpack file scanner for Windows.

    * check single Wavpack files, or all .wv files in a file tree or drive for errors.
    * easy two-click operation via drag & drop or Explorer shell extension.
    * bad files are written to a text file for reference/further processing.
    * no command-line skills required (although you can use that too).

http://wavpack.gl.tter.org

This new beta version adds multi-core/processor support for faster scans & updates to Wavpack 4.50 code.  Let me know how it works for you.

gl.tter
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-

CheckWavpackFiles 2.0 BETA released

Reply #1
Updated to 2.0 Beta2.

Fixes several issues (display corruption, recycle bin folder enumeration & more).

gl.tter
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-

CheckWavpackFiles 2.0 BETA released

Reply #2
Thanks, gl.tter, this is very cool!

Guess I'll have to get a multicore system to test it thoroughly! 

CheckWavpackFiles 2.0 BETA released

Reply #3
Excellent!

Thank you, gl.tter!


 

CheckWavpackFiles 2.0 BETA released

Reply #4
Thanks guys.  It's kinda fun to watch 4 files being checked at once.

Unfortunately there's a limiting factor, and that's the disk access time.  With multiple files (and even worse if the disk is also fragmented), the disk heads are jumping all over the place - it can be slower than the previous single-file version.  On the fastest of my SATA disks I get a good 50-80% CPU utilisation (Q6600 quad-core @ 3.4GHz), but on the other there's almost none at all due to the trashing.

One idea I'm looking into for the next version is to cache the file accesses more intelligently.  A better solution though would be to multi-processor enable the decoding parts of the library, so that consecutive blocks of a single file are transparently processed in parallel & cached for when they're needed (I've mentioned this before).  David, have you ever looked into doing that?
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-

CheckWavpackFiles 2.0 BETA released

Reply #5
One idea I'm looking into for the next version is to cache the file accesses more intelligently.  A better solution though would be to multi-processor enable the decoding parts of the library, so that consecutive blocks of a single file are transparently processed in parallel & cached for when they're needed (I've mentioned this before).  David, have you ever looked into doing that?

There's no reason why it wouldn't work, although for decoding I wonder if the disk I/O would ever be able to keep up (or justify it), although I guess it would certainly eliminate the thrashing problem.

Also, for encoding (especially the -x modes) doing the same thing would certainly be a gain.

It would probably be a better use of time than going to hand-coded assembler for the single threads (which is another thing I think about), but unfortunately I don't have time at this point to look into either. I am happy to kibitz, however... 

CheckWavpackFiles 2.0 BETA released

Reply #6
There's no reason why it wouldn't work, although for decoding I wonder if the disk I/O would ever be able to keep up (or justify it)


It will, as I said, I can get 50-80% CPU utilisation scanning 4 files simultaneously on a quad-core @ 3.4GHz (that figure is a across all 4 cores) - the only reason it's not 100% is that I'm disk starved from the trashing.  Frequent disk seeks drastically reduce throughput, as we all know when we get pagefile trashing & everything grinds to a halt.  With linear single-file streaming I could easily max my CPU and then some.

Quote
Also, for encoding (especially the -x modes) doing the same thing would certainly be a gain.


Sure, but probably a little more complex (due to not knowing how much compresses into each block).

Quote
It would probably be a better use of time than going to hand-coded assembler for the single threads (which is another thing I think about), but unfortunately I don't have time at this point to look into either. I am happy to kibitz, however... 


kibitz? : )  I can provide all the threading and CPU code - in fact if you could split the decoding context into independent block-specific ones (ie. ones that have all the decoding state needed to decode an entire block in isolation), I can do the rest.  With multi-cores so widespread now, this not only gets you a massive bang-for-buck, but better yet all apps benefit transparently.
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-

CheckWavpackFiles 2.0 BETA released

Reply #7
kibitz? : )  I can provide all the threading and CPU code - in fact if you could split the decoding context into independent block-specific ones (ie. ones that have all the decoding state needed to decode an entire block in isolation), I can do the rest.  With multi-cores so widespread now, this not only gets you a massive bang-for-buck, but better yet all apps benefit transparently.

I think the functionality you're looking for already exists in the current library, although it's a little convoluted to get to it. You just need to start up a WavPack decoder context for each thread you want to have, and set the OPEN_STREAMING flag for each one.

The main code would read the WavPack file and parse it into blocks, then pass the blocks (using the WavpackStreamReader callbacks) to the decoding contexts to convert a single block each. Using the OPEN_STREAMING flag makes the contexts not care that the data they suck in is from discontiguous blocks (although your parser must make sure they do get complete blocks). The WavPack DirectShow filter uses this same method (not to use multiple cores but to implement the parsing/seeking separate from the actual decoding).

CheckWavpackFiles 2.0 BETA released

Reply #8
I think the functionality you're looking for already exists in the current library, although it's a little convoluted to get to it. You just need to start up a WavPack decoder context for each thread you want to have, and set the OPEN_STREAMING flag for each one.

The main code would read the WavPack file and parse it into blocks, then pass the blocks (using the WavpackStreamReader callbacks) to the decoding contexts to convert a single block each. Using the OPEN_STREAMING flag makes the contexts not care that the data they suck in is from discontiguous blocks (although your parser must make sure they do get complete blocks). The WavPack DirectShow filter uses this same method (not to use multiple cores but to implement the parsing/seeking separate from the actual decoding).


Cool, I'll look into it.
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-