Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: [bug] custom encoder pipe receives -1 as chunk sizes (Read 2857 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

[bug] custom encoder pipe receives -1 as chunk sizes

Consider encoder examples attached as screenshots.

  • Wavpack 5.4.0 fails with "Object not found"
  • flac 1.3.3 (or master) creates a file of 405m 47s length but correct size (whether or not "Properties/Advanced/Tools/Converter/FLAC: Add seektables to encoded files" is set)
  • APE 7.26 (the first APE version to support pipes) produces garbage. However other programs piping to MAC.exe produce correct results (like type or flac.exe -d -c)

I created a custom "encoder" tool which just grabs stdin as binary and stores to file. And I wasn't the first to write one :)
Grabbing the input with the tool showed 3 differences to original wav file (see attach again):
  • RIFF chunk size is 0xFFFFFFFF instead of original
  • same problem for data chunk size
  • LIST (with track metadata) chunk is missing from the output, which is expected more or less. E.g. flac doesn't support --keep-foreign-metadata with piped input, while APE and Wavpack silently copy it and that data isn't seen by any tool afterwards

So the question is, why is the piped output not according to spec? Foobar's intermediate format (can be WAVE or Wave64, I believe) is known, source samples are known from the source file, isn't it possible to set correct sizes based on this information?

Current fixes for conversions:
  • provide -i for Wavpack, as described previously
  • one can't produce flacs w/o seektables with Foobar since removing -S- means "apply default seektables", and that's the same one gets from disabling "FLAC: Add seektables to encoded files" setting. Actually flac 1.3.3 only fixes resulting header after applying seektables, so one can't pipe encoded w/o -S- flacs anywhere
  • APE 7.27 decided to just load the whole input into memory if the RIFF chunk is improperly sized

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #1 –
Quote from: XPEHOPE3 on
Foobar's intermediate format (can be WAVE or Wave64, I believe) is known, source samples are known from the source file, isn't it possible to set correct sizes based on this information?
No because it's possible to use DSPs during conversion, and DSPs can do anything with audio stream. They can add samples, remove samples, change number of channels, etc.

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #2 –
Quote from: lvqcl on
No because it's possible to use DSPs during conversion, and DSPs can do anything with audio stream. They can add samples, remove samples, change number of channels, etc.
So in general case no. But in lossless/audio-MD5 preserving case there is a way, right?

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #3 –
What a news!
Actually, for many years people are successfully using flac with --ignore-chunk-sizes and wavpack with -i (and other tools with appropriate options) with fb2k converter and have no problem with it.
As for Monkey, obviously, it needs to have such option too.

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #4 –
@Rollin
So is it how the circle of uncomfortableness goes?

  • Since no chunk sizes are provided, receiving codec ("C") must support ignoring the sizes
  • "C" would know the size only on EOF
  • "C" would only be able to set sizes, seekpoints, etc., for its own output only at the end of encoding
  • If the encoding is within a pipeline, "C" has no way to set the sizes in time for the start of the next item ("D", e.g. some muxer in video processing) in the pipeline, so "D" must also support "C" without proper size information
In the end of the pipeline someone has to fill sizes, seekpoints for all the endpoint codecs anyway (as seekpoints are needed at the start of the file, I believe)...

BTW, gstreamer's pipeline AFAIR has a mode for playback where the pipeline length is known in advance and pushed to all other pipeline items.

And I think it's easy to add to Foobar's Converter ability to set correct chunk sizes to the common case of DSP-less conversion.

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #5 –
Quote from: lvqcl on
No because it's possible to use DSPs during conversion, and DSPs can do anything with audio stream. They can add samples, remove samples, change number of channels, etc.
But why does it still happen even if I setup conversion with disabled Processing? After all the decoder, e.g. flac -d -c, outputs correct chunk sizes by itself. So there's probably some code to change those to -1, but not under "if (dsp_active)"

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #6 –
Thanks for the help.

I just released an updated Monkey's Audio that encodes on the fly if the header length is -1.

This way it does no reading to memory which I think is a win.

Cheers,
-Matt

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #7 –
Inputs can actually report any length and produce as many samples as they want, regardless of the reported length. The reported length is just a hint.

Speaking of which, from the FLAC decoder test bench revision 5, test item #45, "no total number of samples set", would report to foobar a length of 0, but produce samples until it runs out of data to produce.

Several of my synthesizer based plugins, including MIDI, report a fixed length based on the file's duration, but may exceed this when configured to loop.
 

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #8 –
Quote from: Rollin on
What a news!
Actually, for many years people are successfully using flac with --ignore-chunk-sizes and wavpack with -i (and other tools with appropriate options) with fb2k converter and have no problem with it.
As for Monkey, obviously, it needs to have such option too.
That's not "obvious" as long as a Monkey's file cannot contain more audio than a 4 GB WAVE file can.
Trying to convert bigger files to .ape with foobar2000, will stall the conversion about at that stage; the abort button will keep the converter box open and unresponsive; closing foobar2000 will take forever.

@Peter : consider either a workaround that "enables the abort button" for these cases - or consider to ignore it, hoping that people who try to use this format (nothing else I think? TAK happily produces big files from stdin) are would be willing to kill processes?

Re: [bug] custom encoder pipe receives -1 as chunk sizes

Reply #9 –
Quote from: Porcus on
That's not "obvious" as long as a Monkey's file cannot contain more audio than a 4 GB WAVE file can.
At least there's some code to read Wave64 files as input in Monkey's audio (but that didn't support pipes in v. 7.31)