Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work (Read 1315 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Hello,

I know the subject sounds very specific but basically I downloaded the soundtrack on the official page (https://www.srb2.org/download/soundtrack/) about two weeks ago in FLAC and everything was fine. I tried playing it again today and it just doesn't work. Foobar tells me that the files are corrupted, it plays the music for a split second before stopping.

This doesn't occur with any other FLAC file from any other album.
I have redownloaded the soundtrack, same problem. Other players such as VLC do not have any issue with the files.
I have also reinstalled Foobar, nothing changes. I desinstalled my recent plugins, nothing either.

Does anybody know what causes that ? Is it a problem with the files or with me ? Below results from the integrity verifier

Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #1 –
Looks like a problem with the files themselves.

Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #2 –
Reference FLAC 1.4.0 got more zealous over errors, and since you reinstalled foobar2000, you got one with a newer FLAC build. That may explain part of this. Including, why foobar2000 reports different MD5s. The following is the most recent x64 beta:
Code: [Select]
MD5: D41D8CD98F00B204E9800998ECF8427E
CRC32: 00000000
Error: Corrupted FLAC stream
Error: MD5 mismatch
Error: Reported length is inaccurate : 1:44.489813 vs 0:00.000000 decoded
while the following is foobar2000 v1.6.12 with foo_verifier 1.4.2:
Code: [Select]
MD5: 6FA28EF080CC19A5AC980FDD91BA1DA4
CRC32: AE8ED95F
Error: Nonsensical most significant bits - the file is corrupted

But I think I need to summon @ktf here, as flac.exe version 1.3.2 verifies this file - which means it also is happy about the MD5. That is a pretty clear sign that old FLAC decodes this one as it was encoded, apparently by reference libFLAC 1.3.2 20170101. But other files do err out with version 1.3.2 as well.

I downloaded the WMAL version, converted to flac with ffmpeg and compared. foobar 1.6.12 reports them as bit-identical to the FLAC versions. But that need not say much if the WMAL was converted from the FLAC files.


PS: @ktf, is flac -t flacfile > errfile.log  supposed to redirect all the objections? It doesn't, and there are quite a lot for files like these.

Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #3 –
Quote from: Porcus on
That is a pretty clear sign that old FLAC decodes this one as it was encoded, apparently by reference libFLAC 1.3.2 20170101.
Not necessarily. The encoder and decoder were never tested for overflow, because it is impossible to get overflow when encoding WAVs, as these are already limited. Problematic files like these have been made by programs (DAWs, synths etc.) that directly use libFLAC and supply values that overflow the indicated range. The DAW/synth/whatever is supposed to take care of this, libFLAC didn't check until 1.4.0, but behaviour is undefined in case of overflows. See here:

Quote
Each sample should be a signed integer, right-justified to the resolution set by FLAC__stream_encoder_set_bits_per_sample(). For example, if the resolution is 16 bits per sample, the samples should all be in the range [-32768,32767].

The reason the MD5sum sometimes validates is because the way samples are 'wrapped-around' by the MD5 sum is often the same for encoding and decoding, which means the MD5 'is correct for the wrong data'. Sometimes the overflow is bad enough to cause more problems than just wrap-around, in that case the MD5 will be incorrect.

libFLAC starting with FLAC 1.4.0 will error out on receiving such out-of-range values while encoding. Decoding also checks for them, as was noticed here.

Quote
PS: @ktf, is flac -t flacfile > errfile.log  supposed to redirect all the objections? It doesn't, and there are quite a lot for files like these.
Usually flac does output on stderr, which means flac -t flacfile 2> errfile.log is needed (on posix systems anyway)
Music: sounds arranged such that they construct feelings.

Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #4 –
Quote from: ktf on
behaviour is undefined in case of overflows.
Yes - and I surely appreciate the fact that the decoder screams error.
But then when "plan A" (play back a compliant file) is no longer accessible, then "plan B" is ... how to play it in practice? What should the "-F" do? A reasonable attempt would be to play it back the 1.3.x way. Or at least give an error that suggests to the user to try to decode it with 1.3.x and listen to it? I'm just thinking aloud here.

Because with 1.4.0, flac -dF on the 117 gives a 68 byte file out. And a lot of the errors are like,
*** Got error code 1:FLAC__STREAM_DECODER_ERROR_STATUS_BAD_HEADER
*** Got error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
... and if that means it doesn't even see the frame header, I'd say 1.3.x does "better"? At least to be integrated in a player? Or maybe not?
Some error code 2 (CRC mismatch) as well.

Quote from: ktf on
The reason the MD5sum sometimes validates is because the way samples are 'wrapped-around' by the MD5 sum is often the same for encoding and decoding, which means the MD5 'is correct for the wrong data'. Sometimes the overflow is bad enough to cause more problems than just wrap-around, in that case the MD5 will be incorrect.
To be more specific (the following is a question): Might it be roughly as follows:
(1) DAW has an internal >16 bit format
(2) DAW is supposed to have a 16 bit signal (OK, these are 24-bit files, nominally), but doesn't check that properly
(note1)
(3) DAW calls some FLAC encoding routine: stereo decorrelates (note2) makes a predictor & encodes the residuals (note3), and writes a pair of FLAC subframes that would have been just fine hadn't the input signal been overflowing (that is: "there is no way to detect it is broken until you decode and find out that it is outside the range")
(4) ... and computes MD5 from the input and writes it into the file, and if the verify option is chosen: decodes, runs MD5 on the output and is happy about itself.

Now ... :
note1: meaning, it behaves like "surely I can work in my 32-bit domain, and when my metadata says 16 bits I trust that rather than checking the range of everything"
note2: it is so that already here we might be in the situation where each subframe (side & L/M/R) is fine, meaning that even if everything is verbatim, you must compute back L&R to detect overflow?
note3: ... and as long as we are in the "note2" scenario, encoder won't even notice? (Maybe there are more situations where the encoder will in practice fail to notice, if it needs a word of at least 17 bits (25 for 24) it would likely go 32, and then it would have to do the same check that the DAW failed at, at least if the predictor is good enough?)
note4: so here is the question: if we are in the "note 2" scenario (or maybe even more of them), is there any good reason that MD5 should match, other than "OK, so MD5 might not be calculated on the actual signal after wrapping, but hey at least you have a confirmation with a weaker hash"? Likely the answer requires more knowledge about MD5 collisions than mine.


Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #5 –
Quote from: Porcus on
Quote from: ktf on
behaviour is undefined in case of overflows.
Yes - and I surely appreciate the fact that the decoder screams error.
But then when "plan A" (play back a compliant file) is no longer accessible, then "plan B" is ... how to play it in practice? What should the "-F" do? A reasonable attempt would be to play it back the 1.3.x way. Or at least give an error that suggests to the user to try to decode it with 1.3.x and listen to it? I'm just thinking aloud here.

The problem here is that there is no way to retrieve the original file. Sure, FLAC 1.3.x does play it back, but with errors. There is no way to fix these errors. If the way FLAC 1.3.x play back these files is deemed OK, then these files can be decoded to WAV, that makes the wrap-around permanent. Then re-encoding to FLAC gives a valid file.

Quote
note1: meaning, it behaves like "surely I can work in my 32-bit domain, and when my metadata says 16 bits I trust that rather than checking the range of everything"

So technically speaking, it is usually possible to restore the signal if the overflow wasn't too bad. Usually, because that is only true if the overflow happens somewhere else than at the very start of a frame. The LPC+Rice coding stage doesn't really care about bit depth, they can store whatever you'd like. The problem lies in the warm-up samples and other samples stored directly, like a verbatim subframe. See for example the screenshot here: https://hydrogenaud.io/index.php/topic,123219.msg1017891.html#msg1017891 As you can see that file contains samples exceeding the [-1.0,1.0]. This only works if none of these samples are a warm-up sample.

Now, the reason FLAC 1.4.x doesn't have a mode to restore these files is because it is unclear how they should be restored. Should they wrap around like FLAC 1.3.x does? Should they clip/clamp at min/max level, as the DAW presumably would have done if it were programmed correctly? Should it try to preserve the peak somehow with a limiter?
Music: sounds arranged such that they construct feelings.

Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #6 –
Quote from: ktf on
The problem here is that there is no way to retrieve the original file. Sure, FLAC 1.3.x does play it back, but with errors. There is no way to fix these errors. If the way FLAC 1.3.x play back these files is deemed OK, then these files can be decoded to WAV, that makes the wrap-around permanent. Then re-encoding to FLAC gives a valid file.
Totally agree as far as there is no "-F" or similar invoked. Myself I actually keep a small handful of lossless-format files rather than having a "fake valid".
But the moment user resorts to something like -F, then it is all about pragmatism I guess. Could be that you will be bogged down in requests for -F1, -F2, where the highest level tries to save everything even if that requires to decode 16-bit to midway and then write a new 32-bit WAVE file. Urrr, you don't want to implement float ;)

Anyway, the posts here including the one you linked to, have revealed a cause for a feedback to the user that ... y'know, it is possible to get out something that is the same as you heard before updating your fb2k.


Quote from: ktf on
So technically speaking, it is usually possible to restore the signal if the overflow wasn't too bad. Usually, because that is only true if the overflow happens somewhere else than at the very start of a frame.
Good point. (I take it that you mean "subframe"? So in the beginning of a frame encoded L&R, and "overflow" by definition means that one of them is overflowed or both, so signal corrupted; and in M&S etc you could in principle be lucky still.)
So if the encoder is fed a 16-bit signal like here, and let's just assume dual mono: it runs linear prediction analysis, fine let's use order seven for this frame, and then it would start to write the warm-up samples ... and if one of these turns out as 1<andsixteenmorebits> the encoder - presuming it does not check - will do whatever it takes to get sixteen out of them, ditching the MSB or something.

But then, that would not be likely to give matching MD5? If the subframe is stored verbatim I understand it could be subject to the same wrapping/truncation upon decoding, but in an LPC frame (including a fixed-predictor), a wrong warm-up sample would propagate through to wrong values of the subsequent samples - in a different way than when reading the original PCM, which just isn't encoded that way?

Re: Sonic Robo Blast 2 (specifically) soundtrack FLAC files do not work

Reply #7 –
The way MD5 is calculated is by lining up samples. As the upper bits are assumed zero, it only copies the bits deemed significant.

FLAC internally works with samples with 32-bits where possible. However, MD5 calculation doesn't work with individual sample values, it works with a bytestream. So, the samples are 'clipped' specifically for MD5 calculation. This shouldn't be a problem, because sample values shouldn't overflow.
Music: sounds arranged such that they construct feelings.