Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Problem with wavpackdll (Read 11940 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Problem with wavpackdll

Hi,

I'm trying to load wv files using wavpackdll in Delphi.

No problem loading the audio data & its various properties. The problem is with the extra RIFF data.

First, the doc says "and the trailer will be available after all samples have been read."
But here, whether I call WavpackGetWrapperData before or after the audio data has been read (note: I read in several chunks until all audio has been decoded, does that matter?), it always return me the header, not the trailer.

I have to force WavpackSeekTrailingWrapper in order to get the trailer (or rather, something that looks like it by its length), even after the audio has been fully decoded.

Second (& this is the biggest problem), WavpackGetWrapperBytes returns me something that roughly looks like the length of my riff chunks, BUT WavpackGetWrapperData seems to point to garbage (& I quadruple-checked the header translation).
My test sample contains named audio markers. I see those names at the end of the wv file, hinting that they're in the trailer (I find this header/trailer system a little confusing btw). But if I check the result of WavpackGetWrapperData, I'm not seeing them, just random data. Any idea?
Thanks.


Note: I have set the OPEN_WRAPPER flag when opening the context.

Problem with wavpackdll

Reply #1
Oops! Nevermind the garbage problem, it was a stupid mistake (of the kind you never suspect first).

However the first (probably harmless) problem is still there, WavpackSeekTrailingWrapper seems to be required whether I've decoded the audio data or not.
Or maybe I misunderstood its usage - but I assumed that if I had called WavpackUnpackSamples until there was no more data, WavpackSeekTrailingWrapper wasn't required.

Problem with wavpackdll

Reply #2
I believe that you have to attempt to read past the end of the audio data before the trailing wrapper will be available (this is not clear in the docs). It is probably a good idea to do this anyway (ignoring the value returned from WavpackGetNumSamples()) because it’s possible that a malformed WavPack file might have more audio than actually indicated in the header and you would not find this out without trying to read past the end. Anyway, this is how the wvunpack command-line utility gets the trailing RIFF data without calling WavpackSeekTrailingWrapper().

Hope this helps...

David


 

Problem with wavpackdll

Reply #3
You're right, that seems to be the case.

So to resume:

-one has to call UnpackSamples until it returns zero or less, then the trailer becomes available

-something that's not in the doc either: if you read the header (before the data), then the trailer will be just that, the trailer. But if you don't read the header, then the trailer will ALSO contain the header. Which is a good thing, it's easier that way. It just doesn't seem to be documented.



Problem with wavpackdll

Reply #4
I also found another pitfall:  in the wrapper data, the 'data' chunk is present (I don't know why it's saved, I assume for maximum transparency), however its chunk length is not zero (it's a value that looks like garbage, because it's not chunk length of the original wav either).

So for those who plan to run a normal wav parser on the wrapper data returned by the DLL, make sure to do a special case for the data chunk.

Problem with wavpackdll

Reply #5
The idea of all that wrapper stuff was to make the WavPack file format mostly agnostic to the source format (at least in the future). So what is stored is simply everything before the audio and everything after the audio, both verbatim. Then the wvunpack program can be “dumb” about what those wrappers mean and simply copy the data and know the source file will be reproduced exactly. And, of course, most applications can simply ignore all that stuff because everything required to get and interpret the audio is stored natively.

So, the data chunk should have the length as it was stored in the source file, and that should be the very last thing in the header RIFF chunk. It will definitely not have a length of zero (unless that’s what the file had, in which case you would have had to use the -i option to encode any audio from it).

Problem with wavpackdll

Reply #6
Yes, I finally realized that the philosophy was to save the original header, including the data chunk without the actual data.

Makes sense. But it was tricky to save it. I'm actually saving to wavpack directly, without a wav file existing, yet I need to store sampler info & markers, thus I have to fake a header with a data chunk of the right length, but with no data, and store it in 2 parts. Works now.

I suppose that extra info should normally be stored in tags, and it would be pretty easy technically, but it's hard to get everyone agree on tags. So, standard wav chunks that everyone can read still beat proprietary sampler/markers tags.

Problem with wavpackdll

Reply #7
-something that's not in the doc either: if you read the header (before the data), then the trailer will be just that, the trailer. But if you don't read the header, then the trailer will ALSO contain the header. Which is a good thing, it's easier that way. It just doesn't seem to be documented.
Yeah, basically what happens is any wrapper data read from the WavPack file is appended onto the data being stored for retrieval by the application. This data is actually cleared by calling WavpackFreeWrapper(). Thanks for pointing this out...I will update this in the docs for the next release.

Problem with wavpackdll

Reply #8
I suppose that extra info should normally be stored in tags, and it would be pretty easy technically, but it's hard to get everyone agree on tags. So, standard wav chunks that everyone can read still beat proprietary sampler/markers tags.

It depends on several things, but if you want to be able to create WavPack files that can be unpacked with the command-line tools and still retain the information, then this is the only way to do it. I am working on a system for 3rd party applications to store proprietary information in dedicated metadata ids inside the WavPack file, but this would of course be lost when converting to WAV.

You might be able to get away with not having to create the RIFF data that comes before the audio data because the library will do that for you, and then you can simply use WavpackAddWrapper() at the end to append trailing stuff. There is currently a minor bug associated with this that the length field of the initial RIFF chunk at the very beginning of the file does not get updated with the full file length, but I have yet to see a program observe this and refuse to continue reading RIFF chunks. In any event, this is being fixed in the next release.

Problem with wavpackdll

Reply #9
You might be able to get away with not having to create the RIFF data that comes before the audio data because the library will do that for you, and then you can simply use WavpackAddWrapper() at the end to append trailing stuff.


Are you sure? Because it's the first thing I tried.
I basically only added the trailer data, nothing up to the data chunk. And my own reader didn't have a problem loading the encoded files, but the command line unpacker did not restore those chunks in a readable way.
It may be the bug you mention, but then wvunpack.exe is affected by it as well.


Quote
There is currently a minor bug associated with this that the length field of the initial RIFF chunk at the very beginning of the file does not get updated with the full file length


Should it be the full length including the non-present audio data, or not? What I'm storing right now includes it, just as it would be in the original wav.


I am working on a system for 3rd party applications to store proprietary information in dedicated metadata ids inside the WavPack file


What do you mean, doesn't the DLL already allow this? I mean, through ID3v1 tags? I know there's a standard list of such tags, but is it really restricted?

IMHO, as it looks like today's tagging systems are (all?) based on random-sized string ID's, I think someone should define a list of ID's that would work (for anyone who'd want to read them) through all of the tagging systems. Could even use only 4-letters IDs so that it'd work for RIFFs as well.


If anyone is interested, here's the info I read out of wavs right now:
-the title & comments, but those already have standard ID3 versions
-the tempo, but this is actually in a kinda proprietary ACID chunk, and it's not really stored in a good way, so this could be improved. There's the ID3 "BPM" tag for this. However it misses an important info: the location of the first downbeat.
-markers, standard RIFF chunks, along with marker purpose IDs
-marker labelling & end positions (for markers to become regions), standard RIFF chunks as well, but it's not really well defined (looks afterthought)
-marker-associated MIDI trigger note, something obscure & with a buggy definition, used by SoundForge
-sampler info (root note, keyzone, fine pitch, etc)

IMHO the markers system would better be stored in a better structure, and with one marker per tag.
The tempo/downbeat could be linked to the marker system in some way, since markers can mark beats.
Sampler chunks are obviously used by samplers, markers by anything, from loop slicers to CD burners.

Markers are stored in integer positions right now in wav files. IMHO double float would be better, because integer markers don't survive resampling (can be important for loop points).

Problem with wavpackdll

Reply #10
You might be able to get away with not having to create the RIFF data that comes before the audio data because the library will do that for you, and then you can simply use WavpackAddWrapper() at the end to append trailing stuff.

Are you sure? Because it's the first thing I tried.
I basically only added the trailer data, nothing up to the data chunk. And my own reader didn't have a problem loading the encoded files, but the command line unpacker did not restore those chunks in a readable way.
It may be the bug you mention, but then wvunpack.exe is affected by it as well.
Hmm, the only thing I can think of is that you must not send the trailer information until all the audio has been sent and WavpackFlushSamples() has been called. I just wrote a little summary of this that I will edit (and test) and put in the next update of the manual:

There are 3 ways of handling the RIFF information when creating WavPack files. The first is the method you are using where you simply provide both the header part and the trailing part yourself. You must do all the formatting yourself and if you decide that you want to update the header part after you have encoded the audio (because, for example, you want to change the data chunk length or the initial chunk length), you must read the first block and use WavpackGetWrapperLocation() to find out where in the block the wrapper is stored. WavPack will not touch any RIFF chunk data that you created. This is a handy method if the application has to also be able to create WAV files because all the code to generate all this RIFF stuff will already be there.

The second way is by far the simplest and is what most people use, and that is to simply do nothing. If you start encoding audio without having sent any wrapper data, the WavPack library will create the header for you. It will have no extra information in it (obviously) and there will be no trailer. If you need to use WavpackUpdateNumSamples() after encoding the audio, the library will also update the header that it created. This method was provided so that most applications do not need to be burdened with all this RIFF stuff and still create fully functional WavPack files.

The final method is basically a hybrid of these two, and I really have not tested it much (but I believe it is the method that Reaper uses). Here, the application starts encoding audio without sending any RIFF data, so the library creates one for you. However, once audio data has been sent, and WavpackFlushSamples() has been called, the application sends some number of trailing RIFF chunks (again, formatted exactly as they will appear in the WAV file). In this way, the application does not have to be concerned about creating (or modifying) the initial RIFF header (with all the boring format stuff) but can still add some application-specific chunks at the end without too much trouble. The minor bug in this that I mentioned is that the length field in the RIFF header that represents the length of the entire file (which I don’t think many programs even look at) does not get updated with this additional length (it only gets updated with audio that was not reflected in the sample count sent to WavpackSetConfiguration()).



Problem with wavpackdll

Reply #11
I am working on a system for 3rd party applications to store proprietary information in dedicated metadata ids inside the WavPack file

What do you mean, doesn't the DLL already allow this? I mean, through ID3v1 tags? I know there's a standard list of such tags, but is it really restricted?
Well, it’s actually APEv2 tags that can have arbitrary fields that you can use for whatever you want, but yes, this is handled by the DLL. However, these are not technically stored in the WavPack file.

What I was referring to was a new mechanism that would allow application-specific metadata to be stored inside the WavPack file. One advantage of this would be that the data could be interleaved with the audio and would be available in streaming environments. I’m not sure exactly what this might be used for, but it could be used for DAW programs to store their metadata without having to come up with something universally portable.

BTW, thanks for adding WavPack support to your application (whatever it is)... 

Problem with wavpackdll

Reply #12
The final method is basically a hybrid of these two


Yes, what I initially tried, and it didn't work but maybe I did it wrong
(I think I do use that chunk length in my parser btw)


The minor bug in this that I mentioned is that the length field in the RIFF header that represents the length of the entire file


Just to be sure: the "length of the entire file" should be the length of the (virtual) wav file, not the wavpack file, right? I mean, WvUnpack doesn't modify that length when unpacking, right?


BTW, thanks for adding WavPack support to your application (whatever it is)...


Well, thanks for WavPack. We're most likely gonna use it to compress samples that come with FL Studio, and let users pack smaller songs to share through the net. We've been relying on OGG (actually OGG as ACM codec) for this so far, but looped samples & sliced/marked samples in general don't survive lossy compression very well.

(talking about this, what'd be really interesting for us would be a lossless/lossy hybrid with lossess compression happening only for specific regions, like around loop points & markers. But that's another story)

Problem with wavpackdll

Reply #13
The final method is basically a hybrid of these two


Yes, what I initially tried, and it didn't work but maybe I did it wrong
(I think I do use that chunk length in my parser btw)

If you use that chunk length in your parser then you would never get to the stuff after the audio.

There could also be something else wrong...I will certainly do some testing before I declare that this method actually works (and if you've got the code already working the other way then that's probably the safest).

Problem with wavpackdll

Reply #14
Just to be sure: the "length of the entire file" should be the length of the (virtual) wav file, not the wavpack file, right? I mean, WvUnpack doesn't modify that length when unpacking, right?
Yes, it's the length of the virtual WAV file. WvUnpack blindly copies that metadata verbatim into the destination with no parsing whatsoever.

Problem with wavpackdll

Reply #15
Well, thanks for WavPack. We're most likely gonna use it to compress samples that come with FL Studio, and let users pack smaller songs to share through the net. We've been relying on OGG (actually OGG as ACM codec) for this so far, but looped samples & sliced/marked samples in general don't survive lossy compression very well.

(talking about this, what'd be really interesting for us would be a lossless/lossy hybrid with lossess compression happening only for specific regions, like around loop points & markers. But that's another story)
Ah, cool! Yes, I can see how WavPack would probably be a better choice for looped samples because the samples may be an arbitrary length and there's no coding delay. Of course, even the lossy mode of WavPack shares these characteristics, so that might be suitable also. And switching back and forth should work fine too...the encoder does not currently support that, but both the format and the decoder do already. Let me know and we can talk about an implementation.