Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Maximum Filesize vor Wavpack? (Read 15289 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Maximum Filesize vor Wavpack?

Hi!

At the moment I'm encountering a strange problem with Wavpack: ~7GB WAV-Files packed with wavpack can't be restored; writing of the restored file stops at exactly 4.194.048KB, then memory usage in Taskmanager slowly rises to around 1,4GByte until "wvunpack.exe encountered a problem and needs to close".
The files are Audition 2.0 32bit-files (mentioned in another thread).

I've tried to restore with both 4.40 and 4.41b.

When encoding these big files wavpack tells me the compression ratio not as "% compressed%, but as 100-(% compressed); furthermore the encoder reaches "100% done" relatively quickly (when reaching the 4GB-"barrier"?) but it takes a healthy amount of time from then on until it really finishes.

Now I'm going to look into my backups and hope that I do have a copy of the original .wav-files left.

Maximum Filesize vor Wavpack?

Reply #1 –
This is probably a silly question, but:  you aren't writing to a FAT32 filesystem, are you?

-brendan

Maximum Filesize vor Wavpack?

Reply #2 –
Quote from: bhoar on
This is probably a silly question, but:  you aren't writing to a FAT32 filesystem, are you?

-brendan

It's not a silly question , but: of course I use NTFS. FAT32 imposes a 2GB-file-limit IIRC.

Maximum Filesize vor Wavpack?

Reply #3 –
If there's a limitation from the filesystem, then yes, using NTFS instead of FAT32 might help. But somehow, I think Paul Brain is already using NTFS; how could he have got the 7 GB+ audio files in the first place?

However, the WAV format has a 4 GB limit. If you had .wav files that were larger than 4 GB, then technically they weren't allowed.

http://www.hydrogenaudio.org/forums/index....showtopic=32422

Maybe you could decode to raw PCM instead?

Quote from: Paul Brain on
Quote from: bhoar on

This is probably a silly question, but:  you aren't writing to a FAT32 filesystem, are you?

-brendan

It's not a silly question , but: of course I use NTFS. FAT32 imposes a 2GB-file-limit IIRC.

OK so you do use NTFS:), but come to think of it, bhoar might have been thinking of multiple partitions, one NTFS and one FAT32...

Maximum Filesize vor Wavpack?

Reply #4 –
I should have looked at that thread more carefully because I would have noticed that you said your WAV file was 7 Gig. I have no idea how Audition is formatting a WAV file that big, but as Kjoonlee says, that's not a valid WAV file.

What is probably happening is that Audition is writing two chunks of audio and WavPack is compressing the first one and storing the second one as RIFF data. This would explain why the percentage reaches 100% on encoding. If you do a -ss with wvunpack you should see the size of the RIFF trailer as being huge (and the audio length being shorter than the real file).

On decode, wvunpack attempts to read the RIFF data into memory, but it's too big for that (I never anticipated this situation).

If you try -i during encode you might be able to encode these files correctly, but you'd have to try it. But, of course, they're still not valid WAV files and I need to find some way to warn about this beforehand.

If you don't actually have those files backed up anywhere else, I could probably create up a special version of wvunpack that would restore them correctly (I suspect that the files are fine). I would also really like to get one of those files for examination, if possible. Will the compressed versions fit on a DVD-R?

Thanks (and sorry for the trouble)...

David

Maximum Filesize vor Wavpack?

Reply #5 –
Quote from: bryant on
If you don't actually have those files backed up anywhere else, I could probably create up a special version of wvunpack that would restore them correctly (I suspect that the files are fine). I would also really like to get one of those files for examination, if possible. Will the compressed versions fit on a DVD-R?

This exact same issue was also “discovered” during the alpha testing of TAK, and also initially assumed to be a limitation of TAK. I don't recall how Thomas has solved this, but the posts can be found very easily I guess since they're in the beta or alpha testing threads...

Thomas wanted to test the very large WAV files, too, of course. And my suggestion was to create a silent WAV file in a compressed folder, but he was still using Windows 98SE and therefore had no NTFS partition or something like that.

So if you want to test such large files, I think a compressed folder and a silent WAV file will do fine. Actually that compressed file will only consume a few hundred kBytes for real.

Maximum Filesize vor Wavpack?

Reply #6 –
Mmm, sparse files.

-brendan

Maximum Filesize vor Wavpack?

Reply #7 –
Not really, because at least 4GiB will be written to disc, when the second chunk starts. Besides there could be some markers bytes in the audio chunk even when it is digitally silent, that will probably trigger the file to be written to disc write. Besides you still cannot allocate a sparse file that is bigger than the free space left on the partition.

Using compressed files/folders only make sense, btw, if a) Brian has not enough disc space and/or b) he doesn't have Audition to create such an incompliant WAV file.

In case of b) using archive based compression is much better anyway. Someone could create the silent gigantic WAV file compress it with gzip or rar and simply upload it somewhere. I guess a silent audio will do for testing.

a) was the case with Thomas. Actually using compressed folder doesn't really help here I guess, since I rather suspect that Bryan wants to get such a file in the first place because he can't create them himself. Audition is a commercial software after all, even tho it is widely available in P2P networks...

Nevertheless using compressed folders is a "quick and cheap" way to handle large testing files when you simply don't want to waste disc space for real.

Maximum Filesize vor Wavpack?

Reply #8 –
Quote from: Fandango on
Not really, because at least 4GiB will be written to disc, when the second chunk starts. Besides there could be some markers bytes in the audio chunk even when it is digitally silent, that will probably trigger the file to be written to disc write. Besides you still cannot allocate a sparse file that is bigger than the free space left on the partition.

Using compressed files/folders only make sense, btw, if a) Brian has not enough disc space and/or b) he doesn't have Audition to create such an incompliant WAV file.


I made my four word post more as a silly aside commentary that using a compressed folder would be similar to using a sparse file if the WAV data is primarily 00s, since they way sparse files are implemented in MS's NTFS driver is tied strongly with the file compressor.  I really had nothing important to add, and if my post causes confusion, then I apologize and please ignore my post.

-brendan

Maximum Filesize vor Wavpack?

Reply #9 –
Please apologize my delay in response, but my PC is out of order at the moment ;-) (RMA'd graphics card) and I need some time to verify these things.


Quote from: bryant on
What is probably happening is that Audition is writing two chunks of audio and WavPack is compressing the first one and storing the second one as RIFF data. This would explain why the percentage reaches 100% on encoding. If you do a -ss with wvunpack you should see the size of the RIFF trailer as being huge (and the audio length being shorter than the real file). On decode, wvunpack attempts to read the RIFF data into memory, but it's too big for that (I never anticipated this situation).
A wvunpack -ss leads to the same memory/termination-error that's happening when doing a "normal" unpack; as for the RIFF-header: how do I see it's length?

Quote from: bryant on
If you try -i during encode you might be able to encode these files correctly, but you'd have to try it. But, of course, they're still not valid WAV files and I need to find some way to warn about this beforehand.
I'll try this when my PC is functioning again.

Quote from: bryant on
If you don't actually have those files backed up anywhere else, I could probably create up a special version of wvunpack that would restore them correctly (I suspect that the files are fine). I would also really like to get one of those files for examination, if possible. Will the compressed versions fit on a DVD-R?

Thanks (and sorry for the trouble)...
Thank you very much for your answer! You absolutely do not need to apologize as Wavpack is a great program, and as others stated, my WAV-files aren't standard-compliant. I'm pretty sure that I have those files backed up, so you do not need to waste your spare time on programming my very special wavpack version, limited edition

At the moment I do not have a DVD-writer, but I'd need someone with a DL-writer as ~5GBytes is more than what will fit on a standard DVD-R.
Perhaps I could send you such a special, compressed "silent" WAV-file? If this would fit your needs...

Thanks again for your efforts in this great piece of software!

Maximum Filesize vor Wavpack?

Reply #10 –
Since I posted last on this thread I was informed of another case of this problem that wasn't exactly the same but was crashing wvunpack because audio data was mistaken for a huge RIFF chunk. So, now that I'm sure about what's going on I will be fixing the crashing issue so you'll be able to [hopefully] restore your files. And, yes, the -ss option doesn't work either!

I'm also going to put in some checks to not allow any WAV file over 4 GB and not allow large (>16 MB) RIFF sections (whether they're valid or not).

I still believe that the -i option will allow your file to compress correctly, but keep in mind that if the resulting WavPack file is over 4 GB then there will be trouble seeking. Since the part after 4 GB on your file was not being compressed at all, you should get a much smaller file than before (which probably explains the difference in compression that you were seeing between the short and long versions).

It would still be nice to get a sample 7GB WAV file from Audition. If you made it all silence, then it should compress with something like ZIP to a very small size. I actually have Audition 1.0 and I'll try the same thing with that, except that I have to backup some files before I have enough free space! 

Thanks again,

David

Maximum Filesize vor Wavpack?

Reply #11 –
Quote from: bryant on
I actually have Audition 1.0 and I'll try the same thing with that, except that I have to backup some files before I have enough free space! 

Use a NTFS compressed folder like I said, it works great. But be sure to also compress the Temp folders. Just in case. And it's wise to disable undo, too, so hopefully the Temp folders won't get used in the first place.

I have Audition 2.0 and I can upload some Giga-WAVs...

Btw, Audition warns the user before saving a WAV that exceeds the 4GB limit:
I had to find a compressor that gives similar results on "empty" files as NTFS' compression.

I guess bzip2 is pretty good and I'll use that, you can find binaries at http://www.bzip.org/ and some optimised versions can be found via the Wikipedia article http://en.wikipedia.org/wiki/Bzip2#Implementations

Ok, here are the two non-conform WAVs: http://www.hydrogenaudio.org/forums/index....showtopic=53611

Maximum Filesize vor Wavpack?

Reply #12 –
Quote from: Fandango on
Quote from: bryant on
I actually have Audition 1.0 and I'll try the same thing with that, except that I have to backup some files before I have enough free space! 

Use a NTFS compressed folder like I said, it works great. But be sure to also compress the Temp folders. Just in case. And it's wise to disable undo, too, so hopefully the Temp folders won't get used in the first place.

I have Audition 2.0 and I can upload some Giga-WAVs...

Thanks for the files, Fandango, I got them. I wonder though why you didn't just upload them here when they were only a few K...? 

Anyway, I'll be interested to see if my Audition 1.0 handles the "extended" wavs or whether this was something added later. It's nice though that they are finally warning the user that the wav files they are creating are non-standard. Back in the Cooledit days they had no hesitation in just making their own "standard" (i.e. with float data stored as ints).

Maximum Filesize vor Wavpack?

Reply #13 –
There is the RIFF64 standard to allow such large files, I am guessing they are just setting the riff chunk size to -1 and have two data chunks.

Maximum Filesize vor Wavpack?

Reply #14 –
Quote from: Paul Brain on
FAT32 imposes a 2GB-file-limit IIRC.

FAT32 has 4 GB limit for file length. Buggy software might interpret the value as signed integer and function properly only up to half of that size.

Sorry for offtopic.

Maximum Filesize vor Wavpack?

Reply #16 –
Thanks Guys for your efforts!

At the moment I can't use my PC (defective graphics card), nevertheless I'm watching this thread closely

Maximum Filesize vor Wavpack?

Reply #17 –
As sort of a late follow-up on this topic, I have incorporated some changes into the latest beta (see this thread). Basically I no longer accept files over 4 GB (at least until I have full W64 support) and I don't allow over 16 MB of RIFF data (because this is certainly some sort of non-conforming or corrupt file).

I also corrected a bug in WvUnpack that will allow any WavPack files created with huge RIFF chunks to be correctly unpacked.

Sorry for the trouble and the delay... 

Maximum Filesize vor Wavpack?

Reply #18 –
Hi!

So if wvunpack is now able to process these files, why not let them be encoded? 
I know, non-standard-conformant files aren't preferable of course, but sometimes they simply exist. In my opinion, a message like "You are about to encode a non standard compliant wav-file; there could be problems with decoding/playing this file. Do you still want to continue? Y/N" would be better, if wavpack/wvunpack is able to process these files correctly in almost any case.

But, of course, thank you very much for the improvements!

Maximum Filesize vor Wavpack?

Reply #19 –
Quote from: Paul Brain on
So if wvunpack is now able to process these files, why not let them be encoded? 

Well, the audio data is being processed as RIFF info instead of audio, which means it's not even being compressed. While these files can be restored to their original WAV format with wvunpack, they cannot be played or read by another application. I think it's best to simply refuse rather than deal with the other issues that will come up.

Maximum Filesize vor Wavpack?

Reply #20 –
But files >4GB are compressed up to 4GB, only the rest of the file is a bit-per-bit-copy right?

I understand your decision of not wanting to deal with all the possible issues. If I feel adventuresome, I could still use anything up to 4.41b2... 
 

Maximum Filesize vor Wavpack?

Reply #21 –
Quote from: Paul Brain on
But files >4GB are compressed up to 4GB, only the rest of the file is a bit-per-bit-copy right?

I understand your decision of not wanting to deal with all the possible issues. If I feel adventuresome, I could still use anything up to 4.41b2... 

I think it would be better to use the latest version and specify -i instead to force compression of the whole file as audio. Then you could at least play the whole thing and could always extract it as a "raw" file. Watch out though that the resulting WavPack is not > 4 GB because then it won't be seekable.

Enjoy your adventure!