Skip to main content

Topic: How to deal with wavpack errors? (Read 7050 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Hi,

A few days after encoding my albums to wavpack,  I used Speek's frontend to verify my files. I don't know what caused it, but at least one of my albums have corrupted songs (CRC errors found in 9 blocks). I only noticed errors in this specific album, so maybe something went wrong during encoding or tagging, I'm not sure.
Anyway, i'd like to know if there's a better solution for error reporting than just wvunpack -v. You see, it is possible to batch test several files using speek's frontend, but since an error report isn't generated, is very difficult to properly identify all the files that contain errors. I could use the wvunpack -m option to display md5 checksum, but it'd be even harder to compare the original md5 against the decoded md5 (I didn't store them in the encoding process).
Is it possible to make wvunpack -v generate a log file after the analysis is done? I'm also considering creating some PAR2 files for each album. What do you think the best settings are? Probably using the same block size wavpack uses?

Thanks in advance.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
How to deal with wavpack errors?
Reply #1
Quote
Anyway, i'd like to know if there's a better solution for error reporting than just wvunpack -v. You see, it is possible to batch test several files using speek's frontend, but since an error report isn't generated, is very difficult to properly identify all the files that contain errors. I could use the wvunpack -m option to display md5 checksum, but it'd be even harder to compare the original md5 against the decoded md5 (I didn't store them in the encoding process).
[a href="index.php?act=findpost&pid=240074"][{POST_SNAPBACK}][/a]

Sorry to hear that you got corrupted files! Please let me know if you suspect WavPack... 

Anyway, there is an easy way to make a log file with the frontend. Simply add this string to the "extra options" box:

-q 2> status.txt

This will create a file with any errors detected (the -q prevents all the progress percentages from going into the file). Also, in the next revision I am going to display at the end of a batch run whether or not any errors occured so you won't have to scroll up through the output to check for errors.

BTW, the MD5 checksums are really not needed to verify WavPack files; their internal checks on every block are virtually as robust in a real-world sense.

  • sven_Bent
  • [*][*][*][*][*]
How to deal with wavpack errors?
Reply #2
md5 verification algo. got errors as well

i can make give you two different files off same size that has the same md5.

just name the size
Sven Bent - Denmark

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
How to deal with wavpack errors?
Reply #3
Are you talking about an implementation bug in Wavpack or the crypthographic attacks found against MD5 recently?

The latters are completely irrelevant for Wavpack.

  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Reply #4
Quote
Sorry to hear that you got corrupted files! Please let me know if you suspect WavPack...  [a href="index.php?act=findpost&pid=240081"][{POST_SNAPBACK}][/a]


Well, I just finished verifying my files with the string you suggested, bryant. Basically, I used 3 programs to manipulate wavpack files:
  • wavpack itself
  • The GodFather for APEv2 tagging
  • Winamp for playback (Wavpack player 2.0 [in_wv.dll])
This might be a coincidence, but 3 of the 4 total corrupted files belong to the same album, the only one I've ever played in foobar2000. Unfortunatelly, I can't remember if I opened the other file in foobar. If that's the case, foobar is probably corrupting the files (using foobar2000 v0.8.3 - special).
I can play around with some more files later tonight, and try to find out if it's foobar's fault or not.

  • Ariakis
  • [*][*][*]
  • Members (Donating)
How to deal with wavpack errors?
Reply #5
I dunno if this is directly related, but when I was in the process of ripping a whole album to a single WavPack + Cue tag, foobar2000 always chokes on the cue for an album which contains the é character.  Whenever I add the tag in fb2k, updating the file after a RG scan generates an error... Were there any extended ASCII/Unicode characters in those 3 or 4 file tags?

  • sven_Bent
  • [*][*][*][*][*]
How to deal with wavpack errors?
Reply #6
Quote
Are you talking about an implementation bug in Wavpack or the crypthographic attacks found against MD5 recently?

The latters are completely irrelevant for Wavpack.
[a href="index.php?act=findpost&pid=240097"][{POST_SNAPBACK}][/a]


it was the bug in md5
nothing to do with wavepack

But since ther is found collision in MD5 shoudl wavepack try to add some newer/better digest algo. ?
Sven Bent - Denmark

  • Garf
  • [*][*][*][*][*]
  • Developer (Donating)
How to deal with wavpack errors?
Reply #7
Quote
Quote
Are you talking about an implementation bug in Wavpack or the crypthographic attacks found against MD5 recently?

The latters are completely irrelevant for Wavpack.
[a href="index.php?act=findpost&pid=240097"][{POST_SNAPBACK}][/a]


it was the bug in md5
nothing to do with wavepack

But since ther is found collision in MD5 shoudl wavepack try to add some newer/better digest algo. ?
[a href="index.php?act=findpost&pid=240297"][{POST_SNAPBACK}][/a]


(BTW. "Of couse" MD5 has collisions, all hashes have them! it's just that a more efficient method to find them was found recently.)

The fact that an attack was found against MD5 does not mean that there's a 'bug" in MD5. In fact, some attacks against MD5 were already known which is why it has been phased out in cryptography earlier. They were just improved. And since the method behind the attack wasn't published yet, just that there is one, it's unknown what else (besides what they mentioned) is vulnerable to it.

And what would be the point of replacing that in Wavpack to begin with? Are you concerned someone is using farm of supercomputers to do the needed calculations so they can replace your Wavpack files with some similar-sounding ones that have sublimal messages but still the same MD5?

If so, I would recommend other more stringent security measures than replacing the MD5 in Wavpack ;-)

Again, this is *completely irrelevant* for the usage in Wavpack, which is to verify the file is not damaged, AFAIK.
  • Last Edit: 08 September, 2004, 12:16:38 PM by Garf

  • seanyseansean
  • [*][*][*][*]
  • Members (Donating)
How to deal with wavpack errors?
Reply #8
Quote
md5 verification algo. got errors as well

i can make give you two different files off same size that has the same md5.

just name the size
[a href="index.php?act=findpost&pid=240085"][{POST_SNAPBACK}][/a]


You don't seem to understand the point of using md5. If you're thinking that getting hash collisions is bad, then you can only prevent it mathematically by having a hash the same size as the original. See the flaw? The md5 only proves *beyond all reasonable certainty* whether something has been tampered with.

Apologies for going off-topic.

  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Reply #9
It was a coincidence. It is not foobar's fault. It's a bug in wavpack 4, i'm pretty sure. I just encoded an album, and immediatelly after (whitout even tagging, or whatever), I run the verification test.
Here's the result:

Code: [Select]
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\01-the_temple-millionaire.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\02-the_temple-baby_hate.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\03-the_temple-22_belzebu.wv:
crc errors detected in 9 block(s)!
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\04-the_temple-dds.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\05-the_temple-fightbull.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\06-the_temple-bankrupt.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\07-the_temple-1_min_and_life.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\08-the_temple-paper_chains.wv:
crc errors detected in 7 block(s)!
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\09-the_temple-drum.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\10-the_temple-devils_lover.wv:
crc errors detected in 1 block(s)!
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\11-the_temple-the_snake.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\12-the_temple-falling.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\13-the_temple-ticket_please.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\14-the_temple-soul_tattoo.wv:
E:\Media\Music\Ripped\The_Temple-Diesel_Dog_Sound-2004\15-the_temple-burn.wv:


And I didn't even play the songs. When I play them through winamp there are huge distortions and pauses, aswell as erroneous indication of the kbps.
I'm going to try and cut the samples to see if the problem still occurs, so I can upload it somewhere.

  • Peter
  • [*][*][*][*][*]
  • Administrator
How to deal with wavpack errors?
Reply #10
If you encode the same album twice, does the problem occur in same locations ?

  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Reply #11
Quote
If you encode the same album twice, does the problem occur in same locations ?
[a href="index.php?act=findpost&pid=240361"][{POST_SNAPBACK}][/a]

Yes, I just tried it (for the third time) to be sure. The same number of blocks at the exact same locations.
I'm using the following command line, in Speek's Multi Frontend (not Wavpack Frontend, since it's a bit outdated and there's no easy way to add your custom commandline):

-hb192x -cc -d

Be careful with the -d setting, it deletes the input file.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
How to deal with wavpack errors?
Reply #12
Well, this is obviously not good. If this were Monkey's Audio I would just say "check your hardware!"... 

I am running some test files through that command-line to see if I can see anything.

I had a problem many months ago where the decorrelation filter would become unstable with certain samples when using the "extra" mode and very low bitrates. The problem was that the "extra" mode was calculating the optimum filter based on the original sample, but during the hybrid (or lossy) compression the filter works on the signal with the added quantization noise. Normally, these would be pretty similar (because the noise is low), but with samples that had gaps in the frequency spectrum there were sometimes cases where the filter would amplify the noise in a feedback loop and create bursts of loud clipping and, in the worst cases, numeric overflow resulting in crc errors.

This problem was fixed by adding "simulated" noise to the original signal passed to the "extra" mode so that the computed filter would have a more representative signal to work with. After implementing that I was never able to duplicate the behavior again.

The reason I suspect that this might be the problem is that your command line is sort of a "worst case" for generating this behavior. The "high" mode, the very low bitrate, the -cc option which adds an additional path for feedback and, of course, the "extra" mode. And the fact that this only happens with a single CD makes me suspect that there's some weird frequency distribution at those spots that's triggering this behavior.

It would really be nice if I could get some samples that cause this. If what I suspect is true, then you should be able to just isolate 10-15 seconds before the worst spot and run the same command-line on it and get it to happen, at least sometimes. If not, then if you could find one near the beginning of the track and isolate up to that spot.

I'm sorry that you ran into this problem and I appreciate you pointing it out (and helping me pin it down). Thanks! 

  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Reply #13
Quote
I dunno if this is directly related, but when I was in the process of ripping a whole album to a single WavPack + Cue tag, foobar2000 always chokes on the cue for an album which contains the é character.  Whenever I add the tag in fb2k, updating the file after a RG scan generates an error... Were there any extended ASCII/Unicode characters in those 3 or 4 file tags?
[a href="index.php?act=findpost&pid=240111"][{POST_SNAPBACK}][/a]
Sorry, Ariakis, I missed your question... But no, as you can see in the log, there are no unicode chars.
Quote
It would really be nice if I could get some samples that cause this. If what I suspect is true, then you should be able to just isolate 10-15 seconds before the worst spot and run the same command-line on it and get it to happen, at least sometimes. If not, then if you could find one near the beginning of the track and isolate up to that spot.[a href="index.php?act=findpost&pid=240417"][{POST_SNAPBACK}][/a]
I managed to have 30 sec. samples ready, and the problem is easy to reproduce. I just need an upload location. Should I create another thread at the "Uploads" forum? Maybe it was easier if a mod move this one, and leave the "Moved: " link at the original forum.

Quote
I'm sorry that you ran into this problem and I appreciate you pointing it out (and helping me pin it down). Thanks!  [a href="index.php?act=findpost&pid=240417"][{POST_SNAPBACK}][/a]
It's my pleasure to help. I agree this is not good for wavpack final, though 
  • Last Edit: 09 September, 2004, 07:26:21 AM by emtee

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
How to deal with wavpack errors?
Reply #14
Quote
I managed to have 30 sec. samples ready, and the problem is easy to reproduce. I just need an upload location. Should I create another thread at the "Uploads" forum? Maybe it was easier if a mod move this one, and leave the "Moved: " link at the original forum.

I would say that starting a new thread over there would be okay; the mods can merge them if they want. I assume it works fine with -hx, right? Thanks, again.

Quote
It's my pleasure to help. I agree this is not good for wavpack final, though 

Yes, you're right. Although if it happens with the betas, and nobody saw it with the release in well over a month, I suspect it's not going to be common (especially after it's fixed).

  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Reply #15
Quote
Quote
I managed to have 30 sec. samples ready, and the problem is easy to reproduce. I just need an upload location. Should I create another thread at the "Uploads" forum? Maybe it was easier if a mod move this one, and leave the "Moved: " link at the original forum.

I would say that starting a new thread over there would be okay; the mods can merge them if they want. I assume it works fine with -hx, right? Thanks, again.


OK, thread started. Let's see if the mods merge them later.
About the problem samples... Yeah, -h works fine. I tried it with the problem samples and it sounded perfect. I don't know about -hx, though.

Unfortunatelly, I can only upload one sample at the moment. Although the Hydrogenaudio's global upload limit is 9Mb, there appears to be a 2Mb limit per sample. One of the other samples needs at least 30 seconds for the problem to became noticeable, the other one needs more than 1 minute, which exceeds the imposed 2MB limit.
There's also a 4th sample, from Metallica's "Kill 'Em All" album. Is the track 05, called "(Anesthesia) Pulling Teeth". Unfortunatelly I can't get it at the moment 

Decode from FLAC to wav, and encode it with the following wavpack settings:

-hb192x -cc

Hope this helps.
  • Last Edit: 09 September, 2004, 01:39:52 PM by emtee

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
How to deal with wavpack errors?
Reply #16
Quote
About the problem samples... Yeah, -h works fine. I tried it with the problem samples and it sounded perfect. I don't know about -hx, though.
[a href="index.php?act=findpost&pid=240550"][{POST_SNAPBACK}][/a]

Actually, what I meant was would the sample compress okay in pure lossless mode for uploading (I would think that the FLAC rule probably wouldn't be enforced here). Anyway, WavPack -hx does compress it without errors and about 10% more than FLAC. 

Anyway, my diagnosis seems to be correct. Raising the bitrate to 256 or eliminating either the "extra" mode or the "-cc" mode completely eliminates the problem. Also, the -cc mode makes about a 6% improvement in compression for this sample, whereas it's usually less then 1%. So, there's something funny about this sample (although it may be more common in this type of music).

I would recommend that until I figure this out the -x and -cc modes not be used together. I don't think I need any more samples, but maybe when I have something you could run it on your other files that failed?

Thanks again.

  • emtee
  • [*][*][*]
How to deal with wavpack errors?
Reply #17
Quote
I would recommend that until I figure this out the -x and -cc modes not be used together. I don't think I need any more samples, but maybe when I have something you could run it on your other files that failed?

Thanks again.
[a href="index.php?act=findpost&pid=240569"][{POST_SNAPBACK}][/a]

Of course. Just ask here, or PM me. Thanks for the support.

  • sven_Bent
  • [*][*][*][*][*]
How to deal with wavpack errors?
Reply #18
just a quick note about my md5 post.

You are complet right i don't know where i had my head that day.
Offcause there will always be collisiosn on a digest as there are only finit numbers of digest values and an (almost) infinet numbers of files possibilities.

btw can somebody correct my spelling of finit and infinet ? :-)
  • Last Edit: 09 September, 2004, 04:25:04 PM by sven_Bent
Sven Bent - Denmark

How to deal with wavpack errors?
Reply #19
Quote
btw can somebody correct my spelling of finit and infinet ? :-)

It's heartening to see such a positive, learning attitude  Here we go:

finit => finite
infinet => infinite

As a bonus:

complet => completely
Offcause => Of course
collisiosn => collision

Cheers,
CD