HydrogenAudio

Hydrogenaudio Forum => General Audio => Topic started by: Quantos on 2005-03-16 23:51:36

Title: The 2GB Wav-limit!?
Post by: Quantos on 2005-03-16 23:51:36
Hello everybody,

I just wanted to know if there is by any chance a program which can
record wav-files larger than 2GB. The Wavelab-lite version I'm using
right know stops any recording  which is exceeding that limit.

Thanks for your help!
Title: The 2GB Wav-limit!?
Post by: CiTay on 2005-03-17 01:32:44
There are a number of programs that have problems with files larger than 2 GB. The WAV format itself is officially limited to 2 GB.

You could use a program that records to a lossy format on-the-fly, like Messer (http://www.dago.pmp.com.pl/messer/), that can record to MP3. Or you cut the recording in between... i mean, 2 GB, that's over 3 hours in one piece...
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-17 02:40:56
The common wav header can only allocate 4 bytes to state the total wav length in bytes which will be why there's a two Gig limit. That detail could be depreciated (make -1 = read till the end of file) -would be nice, might crash old players hmmm...

Since
Linco.exe (http://sourceforge.net/project/showfiles.php?group_id=99332&package_id=107331) can write a raw PCM file (wav file without any headers) it may not have its own restriction on length. That would allow a continuous recording, then have to reencode or chop up and format the raw.
Title: The 2GB Wav-limit!?
Post by: jaybeee on 2005-03-17 08:15:41
Quote
Hello everybody,

I just wanted to know if there is by any chance a program which can
record wav-files larger than 2GB. The Wavelab-lite version I'm using
right know stops any recording  which is exceeding that limit.

Thanks for your help!
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=282816")


[a href="http://audacity.sourceforge.net/]Audacity[/url] can.  And it's free... and very good.
Title: The 2GB Wav-limit!?
Post by: cliveb on 2005-03-17 09:35:34
Quote
The common wav header can only allocate 4 bytes to state the total wav length in bytes which will be why there's a two Gig limit.
[a href="index.php?act=findpost&pid=282850"][{POST_SNAPBACK}][/a]

This isn't quite correct. It is true that the file length in the WAV header is a 4 byte integer, but it's an *unsigned* number, which means that the maximum size of a WAV file is actually 4GB, not 2GB.

Some audio programs get this wrong, and interpret the file length as a signed number, thus imposing a 2GB limit. I can confirm that Wave Repair supports WAV files up to 4GB, and if used just for recording it's free. And I see someone else has pointed out that Audacity (also free) will also go above 2GB.
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-17 15:50:33
Quote
Quote
The common wav header can only allocate 4 bytes to state the total wav length in bytes which will be why there's a two Gig limit.
[a href="index.php?act=findpost&pid=282850"][{POST_SNAPBACK}][/a]

This isn't quite correct. It is true that the file length in the WAV header is a 4 byte integer, but it's an *unsigned* number, which means that the maximum size of a WAV file is actually 4GB, not 2GB.


Theres no need to correct to clarify 
Wether the number is read as signed or not only becomes an an issue beyond 2g.
The 'actual' maximum size neednt have a limit if handled ideally.
Title: The 2GB Wav-limit!?
Post by: kjoonlee on 2005-03-17 16:11:04
... which begs the question, how do you overcome the 4GB limit?

Using -1?
Title: The 2GB Wav-limit!?
Post by: Sebastian Mares on 2005-03-17 16:21:58
Quote
... which begs the question, how do you overcome the 4GB limit?

Using -1?
[a href="index.php?act=findpost&pid=283014"][{POST_SNAPBACK}][/a]


No, because -1 is signed and the size descriptor in the Wave header is unsigned. -1 would be 0xff 0xff 0xff 0xff which already stands for the unsigned 4294967295.

I guess there is no way, since it is a limitation of the format. You could use raw PCM and wrap it into a Matroska container (if supported). If I am not mistaken, the WAV container was invented by Microsoft and at that time, FAT16 was the only HDD file system supported by Windows - which by design has a file size limitation of 2 or 4 GB (not exactly sure).
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-17 16:55:39
Quote
Quote
... which begs the question, how do you overcome the 4GB limit?

Using -1?
[a href="index.php?act=findpost&pid=283014"][{POST_SNAPBACK}][/a]


No, because -1 is signed and the size descriptor in the Wave header is unsigned. -1 would be 0xff 0xff 0xff 0xff which already stands for the unsigned 4294967295.


If a program wont read over 2gigs, the size descriptor never gets to have a sign because the msb is never used. Data has no sign or specific meaning - theres what maybe it should mean, what it can, and what it usualy does... 
By -1 i meant 0xffffffff to be used as an escape value.

Quote
I guess there is no way, since it is a limitation of the format.

Its a limit of how the format is usualy read. An ideal wav reading routine can read to the end of file or use filesize-headersize to compute the PCM length.

Quote
You could use raw PCM and wrap it into a Matroska container (if supported). If I am not mistaken, the WAV container was invented by Microsoft and at that time, FAT16 was the only HDD file system supported by Windows - which by design has a file size limitation of 2 or 4 GB (not exactly sure).
[a href="index.php?act=findpost&pid=283017"][{POST_SNAPBACK}][/a]


The wav format was invented - heh like monkey poke stick in log pickup ants 
Title: The 2GB Wav-limit!?
Post by: cliveb on 2005-03-17 17:10:46
Quote
Quote
I guess there is no way, since it is a limitation of the format.

Its a limit of how the format is usualy read. An ideal wav reading routine can read to the end of file or use filesize-headersize to compute the PCM length.

Actually, this won't work. A WAV file is free to have arbitrary non-PCM data at the end, and some audio processing programs do in fact place proprietory information there. As well as the file length near the front of the RIFF header, there's also a "chunk length" (another unsigned 4 byte number) at the front of the data chunk. Nothing beyond the defined length of the data chunk should be interpreted as PCM data.

Quote
If I am not mistaken, the WAV container was invented by Microsoft and at that time, FAT16 was the only HDD file system supported by Windows - which by design has a file size limitation of 2 or 4 GB (not exactly sure).
[a href="index.php?act=findpost&pid=283017"][{POST_SNAPBACK}][/a]

FAT32 also has a file size limit of 4GB. Even if the WAV standard was changed to support longer files, they would only be possible on NTFS partitions. Lots of people still use FAT32, so it's simplest to just accept that 4GB is the limit.
Title: The 2GB Wav-limit!?
Post by: xmixahlx on 2005-03-17 17:38:51
http://www.uwsg.iu.edu/hypermail/linux/ker...912.3/0009.html (http://www.uwsg.iu.edu/hypermail/linux/kernel/9912.3/0009.html)

anyways, audacity supports >2/4GB thanx to filesystems in linux > kernel 2.2 (or with LFS patch).

see http://linuxreviews.org/sysadmin/filesystems/ (http://linuxreviews.org/sysadmin/filesystems/)


later
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-17 17:39:41
Quote
Quote
Its a limit of how the format is usualy read. An ideal wav reading routine can read to the end of file or use filesize-headersize to compute the PCM length.

Actually, this won't work. A WAV file is free to have arbitrary non-PCM data at the end, and some audio processing programs do in fact place proprietory information there.

Thats a point - but it is also free not too eh? To expand the wav format, make some very unlikely filesize like max-3 equal an escape value which implies a PCM size 'overflow'.
-a proprietory deviation from the not so common compliancy ~?
Quote
As well as the file length near the front of the RIFF header, there's also a "chunk length" (another unsigned 4 byte number) at the front of the data chunk. Nothing beyond the defined length of the data chunk should be interpreted as PCM data.
FAT32 also has a file size limit of 4GB. Even if the WAV standard was changed to support longer files, they would only be possible on NTFS partitions. Lots of people still use FAT32, so it's simplest to just accept that 4GB is the limit.
[a href="index.php?act=findpost&pid=283029"][{POST_SNAPBACK}][/a]

Nice and comfy for programmers to keep the format capped.
Title: The 2GB Wav-limit!?
Post by: NumLOCK on 2005-03-17 19:00:45
Quote
Nice and comfy for programmers to keep the format capped.
[a href="index.php?act=findpost&pid=283047"][{POST_SNAPBACK}][/a]


Well, that's how it goes by Microsoft standards. They're designed to sell well, not to be future-proof.

Most modern formats use variable-length encoding for integers:
- you actually gain bits when dealing with small files
- you have virtually no limit on file size.

The wav format is so simple, that it would be better to just use something else when dealing with big files, instead of adding ugly special-cases that will work in 10% of players, and will make people believe their files are corrupt.

What will happen when a user creates a long WAV file for a family video, and it won't play in Windows media player ?

Even worse, if you use this kind of hack with a legacy audio editor, it will truncate your audio without warning !

Good standards deserve to last, but IMHO, this ain't one of them.
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-17 19:56:13
Quote
.....
What will happen when a user creates a long WAV file for a family video, and it won't play in Windows media player ?

Even worse, if you use this kind of hack with a legacy audio editor, it will truncate your audio without warning !

The max-3 escape 'implication' i mention could only possibly misinterprate PCM files of exact size max-3 (highly improbable length and impossible with 16bit or stereo) -which have also postfixed metadata.*
Quote
Good standards deserve to last, but IMHO, this ain't one of them.
[a href="index.php?act=findpost&pid=283063"][{POST_SNAPBACK}][/a]


Its so common, for a container of PCM I think it should evolve rather than be ditched and replaced. Its a choice between k.i.s.s for the programmers or the users 

[span style='font-size:8pt;line-height:100%']edit* -or just fail to read because of the filesize i admit - this happens already...[/span]
Title: The 2GB Wav-limit!?
Post by: NumLOCK on 2005-03-17 23:06:52
I completely agree: more things should evolve instead of getting replaced every few years, that would prevent much useless re-development...

But wave ? I mean, if you do anything than switching between sample formats, you need to rewrite  all applications that read or write wave files...

The problem is that, in the first place, there was not much design to evolve from.

This page sums it up pretty well:

http://www.borg.com/~jglatt/tech/wave.htm (http://www.borg.com/~jglatt/tech/wave.htm)

The format cannot be improved much, without breaking compatibility with most existing programs. So when new features are needed, why not use the better alternatives ?
Title: The 2GB Wav-limit!?
Post by: Mr_Rabid_Teddybear on 2005-03-17 23:56:40
I think I read somewhere that aiff does not have this restriction. Is this correct?
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-18 00:32:50
Quote
I completely agree: more things should evolve instead of getting replaced every few years, that would prevent much useless re-development...

But wave ? I mean, if you do anything than switching between sample formats, you need to rewrite  all applications that read or write wave files...

Not if the 'simple' form PCM size specifier is escaped for file sizes over 4 gig. Old players wouldnt play files that needed to use the escape, old editors couldnt work with the files or convert them to their own format, but they couldnt anyway cause the file is just too big.
Its backward compatibilty for all the files which dont use the extended feature.

Maybe its too much fuss, but its not a hard limitation of the format that it cant be extended past 4gig. In practice i guess theres enough work to do if many wav openers misinterpret the 4byte addresser as signed.

Quote
This page sums it up pretty well:

http://www.borg.com/~jglatt/tech/wave.htm (http://www.borg.com/~jglatt/tech/wave.htm)
Quote
A Bastardized Standard
The WAVE format is sort of a bastardized standard that was concocted by too many "cooks" who didn't properly coordinate the addition of "ingredients" to the "soup". Unlike with the AIFF standard which was mostly designed by a small, coordinated group, the WAVE format has had all manner of much-too-independent, uncoordinated aberrations inflicted upon it.


I could have my stance skewed by my embryonic experiments in wav file handling but the Articles comments:
'much-too-independent'- is irrelevant, probably owing to an unrelated interest of the Authors; p (fine page though thankyou)
- If one wishes to inflict aberrations on a format, that is impossible to do independantly.
If a format can be worked within to include prefered capabilities without breaking its compatibilty (less the capabilities) -thats a programmers perogitive.
Good stuff should eventually get coordinated into the format

The common standard for packing chunks of pcm data could just accomodate simple wav formats with escape values to quirky ones. I dont see a point just now in sticking to origional limitations... hmm..
Title: The 2GB Wav-limit!?
Post by: NumLOCK on 2005-03-18 08:35:11
Code: [Select]
typedef struct {
 ID             chunkID;
 long           chunkSize;

 short          wFormatTag;
 unsigned short wChannels;
 unsigned long  dwSamplesPerSec;
 unsigned long  dwAvgBytesPerSec;
 unsigned short wBlockAlign;
 unsigned short wBitsPerSample;

/* Note: there may be additional fields here, depending upon wFormatTag. */

} FormatChunk;


Code: [Select]
typedef struct {
 ID             chunkID;
 long           chunkSize;

 unsigned char  waveformData[];
} DataChunk;


You could always define wFormatTag=1, as if audio data were compressed, set chunkSize to 0xFFFFFFFF, and then store the actual size in the compressed file info.

But then, you could not store the real size in DataChunk, because there's not enough room. Or you could use a spacial value of wFormatTag, that would trigger various changes in the format.

Anyway, none of these changes would be compatible with existing apps 

Then, instead of redefining two data structures in an incompatible way, why not use mka ?
Title: The 2GB Wav-limit!?
Post by: cliveb on 2005-03-18 09:33:41
Quote
You could always define wFormatTag=1, as if audio data were compressed, set chunkSize to 0xFFFFFFFF, and then store the actual size in the compressed file info.
[a href="index.php?act=findpost&pid=283227"][{POST_SNAPBACK}][/a]

formatTag=1 means *un*compressed. (I take it this is a typo, and that's what you meant). It could work, because it's impossible for a chunk to have a size as large as xFFFFFFFF.

Quote
But then, you could not store the real size in DataChunk, because there's not enough room. Or you could use a spacial value of wFormatTag, that would trigger various changes in the format.

Anyway, none of these changes would be compatible with existing apps 
[a href="index.php?act=findpost&pid=283227"][{POST_SNAPBACK}][/a]

Let me first say that I feel the WAV format shouldn't be messed around with. 4GB gives you over 6 hours of 44.1/16, which seems quite a long time. The only reason for having bigger files strikes me as being able to make very long unattended recordings. The obvious solution is for the recording program to produce multiple WAV files.

However, if what people are trying to arrive at is a way to support single WAV files longer than 4GB while remaining compatible with existing programs, then the most straightforward approach seems to be:

1. Put the first 4GB of samples (minus however many bytes are required for headers) into the data chunk.
2. Put the rest into multiple proprietory chunks (let's call them "more" chunks) after the data chunk. Each chunk has a maximum length of 4GB (because its length field must be an unsigned 32 bit integer, so that these chunks can be ignored by compliant programs).
3. Include an additional proprietory chunk before the data chunk that indicates the presence (and number) of "more" chunks. Each pointer to a "more" chunk could include a file offset as large as we like (eg. using a 64 bit integer).
4. Leave the file_length field in the RIFF header set to *exclude* the "more" chunks. (It couldn't include them anyway, because the file is longer than 4GB). This means that compliant programs which don't know about "more" chunks will still work provided they ignore unknown proprietory chunks (albeit processing only the first set of samples, in the data chunk).

But in summary, here is not the place to be making off-the-cuff design extensions to a mature standard which actually works suprisingly well.
Title: The 2GB Wav-limit!?
Post by: NumLOCK on 2005-03-18 10:07:01
Quote
formatTag=1 means *un*compressed. (I take it this is a typo, and that's what you meant).

Oops..  indeed -- and it proves my lack of practice with wav

Quote
3. Include an additional proprietory chunk before the data chunk that indicates the presence (and number) of "more" chunks. Each pointer to a "more" chunk could include a file offset as large as we like (eg. using a 64 bit integer).

Wouldn't this cause problems with some badly written, unmaintained programs ? I've seen some "optimized" programs which don't... umm.. parse everything 

Quote
But in summary, here is not the place to be making off-the-cuff design extensions to a mature standard which actually works suprisingly well.


Agreed (was trying to discourage him a bit, actually) 
Title: The 2GB Wav-limit!?
Post by: madoka on 2005-03-18 10:35:45
Quote
I think I read somewhere that aiff does not have this restriction. Is this correct?
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=283150")

No, this is incorrect.  AIFFs have a maximum size of about 2 GB, for the chunk size field in the chunk header is a 32-bit signed integer.  Moreover, each AIFF can only have 1 sound data chunk.

[a href="http://www.tsp.ece.mcgill.ca/MMSP/Documents/AudioFormats/AIFF/AIFF.html]http://www.tsp.ece.mcgill.ca/MMSP/Document.../AIFF/AIFF.html[/url]
Title: The 2GB Wav-limit!?
Post by: cliveb on 2005-03-18 12:29:05
Quote
Quote
3. Include an additional proprietory chunk before the data chunk that indicates the presence (and number) of "more" chunks. Each pointer to a "more" chunk could include a file offset as large as we like (eg. using a 64 bit integer).

Wouldn't this cause problems with some badly written, unmaintained programs ? I've seen some "optimized" programs which don't... umm.. parse everything 
[a href="index.php?act=findpost&pid=283243"][{POST_SNAPBACK}][/a]

All bets are off with regard to badly written programs. But all WAV processing programs really should at the very least ignore any chunks they don't recognise. So this extra chunk (which will contain the required 4 byte integer defining its size) should just be skipped. The very large offsets to other chunks that it contains would never be seen.

Actually, come to think of it, this additional chunk before the data chunk isn't necessary: a program using the extra "more" chunks could just notice that they are present and process them. Other programs will ignore them. And even badly written programs that assume the data chunk always comes right after the fmt chunk would still work.

(Question to self: why am I even thinking about this, when my opinion is that it shouldn't happen???!!)
Title: The 2GB Wav-limit!?
Post by: NumLOCK on 2005-03-18 15:39:57
Ah, ok. Good idea..

Then the first, legacy chunk could perhaps be dummy (ie: of zero length), so that legacy programs (buggy or not) will see nothing at all, unless they are upgraded ?

But then, when opening one of these new wave files with an old program and saving it, all newer chunks would get destroyed 
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-18 18:44:50
Quote
But in summary, here is not the place to be making off-the-cuff design extensions to a mature standard which actually works suprisingly well.
[a href="index.php?act=findpost&pid=283236"][{POST_SNAPBACK}][/a]

well, i just find it interesting to *try* and point out possibilities, the replies help my interests.

cheers
Title: The 2GB Wav-limit!?
Post by: Kurt S on 2005-03-18 19:12:57
You could always use Adobe's Audition in the multitrack mode. When the temp file reaches the 4 gig limit, it creates another one. You can keep recording untill your hard drive is filled up. Saving the file will be an issue but you could always save several 4 gig chnks that make up the whole recording session or save as mp3 as someone else mentioned earlier.
Title: The 2GB Wav-limit!?
Post by: ChristianHJW on 2005-03-19 21:21:43
Quote
Anyway, none of these changes would be compatible with existing apps 
Then, instead of redefining two data structures in an incompatible way, why not use mka ?


See ? All these great experts here discussing about the limitations of a format that has been 'invented' when the biggest HDDs were still 120 MB, and none of them even considers to give you an answer to this very valid question  .....

Christian
matroska project admin
http://www.matroska.org (http://www.matroska.org)
Title: The 2GB Wav-limit!?
Post by: Tec9SD on 2005-03-19 22:59:00
Quote
Quote
Anyway, none of these changes would be compatible with existing apps 
Then, instead of redefining two data structures in an incompatible way, why not use mka ?


See ? All these great experts here discussing about the limitations of a format that has been 'invented' when the biggest HDDs were still 120 MB, and none of them even considers to give you an answer to this very valid question  .....

Christian
matroska project admin
http://www.matroska.org (http://www.matroska.org)[a href="index.php?act=findpost&pid=283675"][{POST_SNAPBACK}][/a]
I am uncertain that I understand the  relevance of your reply and how it contributory to the thread or what you are implying?
Quote
[...]
I guess there is no way, since it is a limitation of the format. You could use raw PCM and wrap it into a Matroska container (if supported).
[...][a href="index.php?act=findpost&pid=283017"][{POST_SNAPBACK}][/a]
However, would you please instead respond with an authoritative answer?

Thanks, tec
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-19 23:19:39
Quote
Quote
Anyway, none of these changes would be compatible with existing apps 
Then, instead of redefining two data structures in an incompatible way, why not use mka ?


See ? All these great experts here discussing about the limitations of a format that has been 'invented' when the biggest HDDs were still 120 MB, and none of them even considers to give you an answer to this very valid question  .....

Christian
matroska project admin
http://www.matroska.org (http://www.matroska.org)
[a href="index.php?act=findpost&pid=283675"][{POST_SNAPBACK}][/a]

I answered this question - thought I was getting boring; wav headers could ~surely be redefined to break 4gig barrier without breaking backward compatability. Cliveb's rhetorical solution might be clearer and superior to my own suggestions.

I was not aware of the matroska project as maybe I should be, and it looks good -extensible, I hope it catches on.  Old .wav only reading programs wont be able to open matroska files though will they?
Title: The 2GB Wav-limit!?
Post by: rjamorim on 2005-03-19 23:26:32
Quote
Old .wav only reading programs wont be able to open matroska files though will they?
[a href="index.php?act=findpost&pid=283713"][{POST_SNAPBACK}][/a]


Nope, and, at least for me, that's the point of sticking to WAV standard.
Title: The 2GB Wav-limit!?
Post by: ChristianHJW on 2005-03-22 20:43:40
Quote
See ? All these great experts here discussing about the limitations of a format that has been 'invented' when the biggest HDDs were still 120 MB, and none of them even considers to give you an answer to this very valid question  ..... Christian
I am uncertain that I understand the  relevance of your reply and how it contributory to the thread or what you are implying?


What i wanted to say is, if all the energy that was invested into discussing the limitations of an old, outdated format, and to make it even worse, even more energy to find hacks around them, would be invested into adapting new, better standards like matroska into various important apps like Cooledit, EAC and others, we all move forward a lot faster  .....

Quote
I was not aware of the matroska project as maybe I should be, and it looks good -extensible, I hope it catches on.  Old .wav only reading programs wont be able to open matroska files though will they?


matroska will only 'catch on' if people actually start using it, coders start implementing it. Waiting for others until it finally will become a standard one day, can lead to nowhere ....

Quote
Nope, and, at least for me, that's the point of sticking to WAV standard.


If all people would think like that, we'd all still be using MP3  .....

Christian
matroska project admin
http://www.matroska.org (http://www.matroska.org)
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-03-22 21:04:45
Quote
matroska will only 'catch on' if people actually start using it, coders start implementing it. Waiting for others until it finally will become a standard one day, can lead to nowhere ....

Difficulty is its not backwards compatible with wav, so it will have to be implemented in parallel for quite some years before simplifying things.
A basic Wav compatible format for PCM doesnt have to be very complex to read or write or do beyond 4G so why should the next suggested format discourage refinement of an existing one?
Title: The 2GB Wav-limit!?
Post by: rjamorim on 2005-03-22 22:08:16
Quote
If all people would think like that, we'd all still be using MP3  .....
[a href="index.php?act=findpost&pid=284647"][{POST_SNAPBACK}][/a]


Most people are still using MP3. Me included.
Title: The 2GB Wav-limit!?
Post by: ChristianHJW on 2005-03-23 20:11:55
Quote
Quote
If all people would think like that, we'd all still be using MP3  .....

Most people are still using MP3. Me included.


.... call me a radical revoluzzer then, i dont  .....
Title: The 2GB Wav-limit!?
Post by: wrevans on 2005-04-26 21:50:11
There have been a number of posts about the 2GB size limit for wav files.  I recently was using an automatic sound recorder to record hours of bird song on my laptop. I meant to select mono but accidentally selected stereo resulting in a 2.57 GB wav file that will not play (or open in a sound editor such as Sound Forge).

Is there a way to salvage this information by going in and grabbing samples?
Title: The 2GB Wav-limit!?
Post by: xmixahlx on 2005-04-26 21:58:04
use it as raw audio

you may need to know bits/samplerate/etc.


later
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-04-27 00:46:40
Quote
use it as raw audio

you may need to know bits/samplerate/etc.

I ended up searching a dozen or so websites for a file splitting
utility as promising as this (http://www.freebyte.com/hjsplit)-hjsplit its a 200kB peach

E: oops added link
Title: The 2GB Wav-limit!?
Post by: CiTay on 2005-04-27 14:06:14
Quote
I ended up searching a dozen or so websites for a file splitting
utility as promising as this (http://www.freebyte.com/hjsplit)-hjsplit its a 200kB
[{POST_SNAPBACK}][/a] (http://index.php?act=findpost&pid=293585")


Ah yes, HJSplit. "Very nice looking freeware file splitter" - what a blatant lie. It's ugly, with a horrific usability.

Maybe try this: [a href="http://www.acc.umu.se/~max/maxsplitter/index_free.html]http://www.acc.umu.se/~max/maxsplitter/index_free.html[/url]
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-04-27 16:29:30
Quote
Maybe try this: http://www.acc.umu.se/~max/maxsplitter/index_free.html (http://www.acc.umu.se/~max/maxsplitter/index_free.html)
[a href="index.php?act=findpost&pid=293671"][{POST_SNAPBACK}][/a]


Yeah well, hjsplit is GPL, 200k and explicitly supports filesizes up to 10 gigabytes.
The link rankings for 'file ~splitting' are riddled with dozens of trialwares and old <4gig utilities designed for floppies.
Maxsplitter doesnt deem to mention its max size, and the download is for a 'free edition'
We can take it from your recommendation then that the missing functionality doesnt include very large file handling then? -not being able to tell from anywhere on the website - its just another candidate to try and then (in most cases) delete.
Title: The 2GB Wav-limit!?
Post by: kjoonlee on 2005-04-27 17:09:46
You can use GNU dd and GNU split to split files.

dd to skip the RIFF header, split to split.
Title: The 2GB Wav-limit!?
Post by: xmixahlx on 2005-04-27 18:09:51
oooh, that's lookin' manual, baby...

100th suggestion: shntool


later
Title: The 2GB Wav-limit!?
Post by: c-naptik on 2005-05-08 03:44:06
Quote
oooh, that's lookin' manual, baby...

100th suggestion: shntool


later
[a href="index.php?act=findpost&pid=293707"][{POST_SNAPBACK}][/a]


Hi,

I'm having the same problem here, I just recorded a live mix, 3.29 Gb with N-Track.
I didn't know about the 2 Gb WAV file limit, but I should have.
I got an "error exiting data chunk", something like that, as well as an error involving the 'fmt' chunk.
The data is there though.

I've tried your different solutions but none has worked so far. HJSplit does split my file into smaller ones, but the header must still be corrupted.

I've tried shntool as well, but no luck :

I:\>shntool strip mix.wav
warning: no output format specified - defaulting to wav

shntool [strip]: error: skipping file 'mix.wav' because its header is inconsiste
nt

I'm willing to use dd now, but what should the syntax be exactly? I can't find any useful info online so far.

Thank you very much for your help, it will be greatly appreciated!
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-05-08 07:06:09
Quote
I'm willing to use dd now, but what should the syntax be exactly? I can't find any useful info online so far.

You could use Sox (http://sox.sourceforge.net/) now, to convert the split files into separate wavs again.
Command something like:

sox -r 44100 -s -w -c 1 filename.raw filename.wav

-r samplerate
-s signed integer samples
-w 16bit samples
-c channels

hth
Title: The 2GB Wav-limit!?
Post by: kjoonlee on 2005-05-08 12:24:21
To use GNU dd to skip the header, you need to know the size of the header.

First save a normal short file with the same characteristics. Take note of its size, sampling rate, bitdepth, number of samples, and number of channels.

raw size (bits) == sampling rate x bitdepth x number of samples x number of channels
raw size (bytes) == raw size (bits) / 8
size == header size + raw size (bytes)
header size == size - raw size (bytes)

Let's say header size is 54 bytes.

dd if=big.wav of=stripped.pcm ibs=54 skip=1

This will give you a stripped PCM file, and if it's less than 4GB in size, you should have no trouble loading it as raw PCM in a capable audio editor.

----

If you want to split any file, you can use GNU split. Let's say you want to split a stripped PCM file into several 2GB stripped-PCM files.

split -b 2048m stripped.pcm 'splitpcm.'

This will give you files from splitpcm.aa to splitpcm.zz.

---

Where to get GNU coreutils: google for MinGW, GnuWin32, or Cygwin.

The manual for dd and split: http://www.gnu.org/software/coreutils/manual/coreutils.html (http://www.gnu.org/software/coreutils/manual/coreutils.html)
Title: The 2GB Wav-limit!?
Post by: kjoonlee on 2005-05-08 12:31:24
If pipes are fully supported, without the use of temporary files, you can combine the two processes above.

dd if=big.wav ibs=54 skip=1 | split -b 2048m - 'strippedpcm.'
Title: The 2GB Wav-limit!?
Post by: c-naptik on 2005-05-08 17:23:39
Quote
If pipes are fully supported, without the use of temporary files, you can combine the two processes above.

dd if=big.wav ibs=54 skip=1 | split -b 2048m - 'strippedpcm.'
[a href="index.php?act=findpost&pid=296029"][{POST_SNAPBACK}][/a]


Thank you very much for your help!

I managed to fix it in a different manner in the end :
I noticed than by splitting my file into smaller ones (using HJsplit and that other one), and by renaming the second file say, from .wav.002 into .raw, I managed to load it into Cool Edit as RAW file (16 bits, 44 khz).
I tried with smaller sizes, as I thought the first one would have been corrupted.
It figured out that I could simply rename the whole 3.29 Gb corrupted wave file into .raw and thus load it into RAW file in the audio editor.
From there I was able to save the file back into proper .wav and .mp3.
Now everything is fine

Thanks for all these shell commands though, this may come in handy one of these days, it's no lost knowledge, I'll write it down!
Title: The 2GB Wav-limit!?
Post by: spoon on 2005-10-15 12:39:20
To resurect an old thread, there is a new proposal for a RIFF64 standard (one likely to be used by the broadcasting industry):

http://www.sr.se/utveckling/tu/bwf/prog/RF_64v1_4.pdf (http://www.sr.se/utveckling/tu/bwf/prog/RF_64v1_4.pdf)
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-10-15 14:09:05
cool - Its a backwards compatible format extension that escapes the old wave format at size = 0xffffffff and puts the extended details in a "junk chunk"
Proposed about a month before this thread started.

Nice find Spoon'
Title: The 2GB Wav-limit!?
Post by: Involarius on 2005-10-15 16:47:36
OoOoOoO just like BigTIFF (http://www.awaresystems.be/imaging/tiff/bigtiff.html)
Title: The 2GB Wav-limit!?
Post by: spoon on 2005-10-15 19:24:45
>Its a backwards compatible

Perhaps not for the poorly written programs (ones that expect a 'fmt ' at a certain location, instead of walking the chunks...). Other than that, I like it, and will be implementing it.
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-10-15 21:02:34
It breaks my own little wavreader class, but thats not really difficult
Code: [Select]
  if (fmt!=0x666D7420)
 { System.out.println("  fmt sig not encountered - Failure Likely...");
   System.out.println("  fmtsig val="+fmt+"\n");}
Title: The 2GB Wav-limit!?
Post by: rjamorim on 2005-10-15 21:21:23
Quote
>Its a backwards compatible

Perhaps not for the poorly written programs (ones that expect a 'fmt ' at a certain location, instead of walking the chunks...). Other than that, I like it, and will be implementing it.
[a href="index.php?act=findpost&pid=334627"][{POST_SNAPBACK}][/a]


That's great. Thanks.

I hope other developers embrace it too.
Title: The 2GB Wav-limit!?
Post by: bryant on 2005-10-15 21:27:38
I received information the other day about an alternative solution promoted by Sony that is mentioned in that paper and described here:

http://media.vcs.de/Downloads/Sony_Wave64.pdf (http://media.vcs.de/Downloads/Sony_Wave64.pdf)

Hopefully one standard will emerge. In any event, sounds like a nice enhancement for WavPack soon...
Title: The 2GB Wav-limit!?
Post by: AndyH-ha on 2005-10-15 21:58:06
I have not tested this extensively, nor looked at the documentation, but I discovered a while back that n-Track Studio just keeps going when it reached 2G by starting another file. As far as I could determine, it never dropped a sample, the second file just continued perfectly. This approach could allow unlimited length recordings in a not-so-expensive program. I have the 24 bit version of n-Track Studio.
Title: The 2GB Wav-limit!?
Post by: spoon on 2005-10-15 22:13:11
Ok to retract my previous statement 'Its a backwards compatible' - in it's current form it is not because if you look at the docs, RIFF (first four bytes in the file changes to 'RF64') I have written to them to suggest that it should stay 'RIFF' then old programs will play the files upto 4GB and newer programs should read the 'ds64' chunk and will know the true length.

Wave64 seems to be used a fair amount, not sure I like the change of simple chunks (ie 'fmt ') to full blown UUIDs, seems pointless (are there going to be 100 billion x 100 billion chunk identifiers? no.
Title: The 2GB Wav-limit!?
Post by: bryant on 2005-10-15 22:32:35
Quote
Ok to retract my previous statement 'Its a backwards compatible' - in it's current form it is not because if you look at the docs, RIFF (first four bytes in the file changes to 'RF64') I have written to them to suggest that it should stay 'RIFF' then old programs will play the files upto 4GB and newer programs should read the 'ds64' chunk and will know the true length.
[a href="index.php?act=findpost&pid=334678"][{POST_SNAPBACK}][/a]

I don't think so, just from a quick read. I think the idea is that the file is identical to a standard wav as long as it's less than 4 gig, except that there's room in a dummy chunk (after "fmt") to expand the header when the file size exceeds 4 gig. At that point the RIFF chunk at the beginning is replaced with the RF64 (and the extra space is used so there's no shuffling of the whole file).

I don't think it would be very good if wav files bigger than 4 gig simply appeared to unaware applications as shorter than they really were without any warning for the user. Instead, they should simply be unreadable when they are bigger than 4 gig.
Title: The 2GB Wav-limit!?
Post by: rompel on 2005-10-15 22:42:03
Quote
Ok to retract my previous statement 'Its a backwards compatible' - in it's current form it is not because if you look at the docs, RIFF (first four bytes in the file changes to 'RF64') I have written to them to suggest that it should stay 'RIFF' then old programs will play the files upto 4GB and newer programs should read the 'ds64' chunk and will know the true length.

It seems to me that old software will be broken on >4GB files no matter what you do.  I'd rather have the failure mode be an immediate error rather than silent truncation.  Of course, if you're going to change the file type, why bother with the ds64 chunk?  Why not just expand the necessary fields to 64-bit like Wave64?

Edit: David said it first.

Quote
Wave64 seems to be used a fair amount, not sure I like the change of simple chunks (ie 'fmt ') to full blown UUIDs, seems pointless (are there going to be 100 billion x 100 billion chunk identifiers? no.

I'm with you here.  I saw that for the first time today and it struck me as incredibly stupid.
Title: The 2GB Wav-limit!?
Post by: ChiGung on 2005-10-15 23:35:59
Quote
It seems to me that old software will be broken on >4GB files no matter what you do.  I'd rather have the failure mode be an immediate error rather than silent truncation.
Of course, if you're going to change the file type, why bother with the ds64 chunk?  Why not just expand the necessary fields to 64-bit like Wave64?

I brought another brain cell in on this, and its telling me that since they stated a desire for backwards compatibility, the RIFF sig should be used, until the size exceeds 4 gig (or extra channels etc required), then it can be changed to RF64 and the junkchunk is utilised for extending the file. (Its written in at the start because its much harder to add later if/when itturns out to be needed)
Id like the fmt chunk to stay first. Its just a small but nice consideration if basic formats can be coded simply when possible. Would help students and amature enthuasts get a handle, keeping basics accessible, avoiding unnecessary layers of metafluff.
No more braincells available atm for finding out the proposed new chunk order.