Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC v1.4.x: Strange decoding error (Read 6092 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

FLAC v1.4.x: Strange decoding error

I happen to have a FLAC file from an unknown source, which behaves differently when being decoded with flac versions before and after v1.4.x. According to its StreamInfo, it was created with flac v1.3.2.
When testing this file with "flac -t" with any version of flac I have (that is v1.3.[1-4]) before v1.4.x, it reports OK.
The same file tested with any version of flac after v1.4.x, it reports this:
Code: [Select]
05_Eldorado.flac: *** Got error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
05_Eldorado.flac: ERROR during decoding
                                  state = FLAC__STREAM_DECODER_END_OF_STREAM
Latest foobar's "verfiy integrity" also reports an error:
Code: [Select]
Item: "...\05_Eldorado.flac"
MD5: D80F69ABFF24166034F1D8BC761F62D4
CRC32: FEBFC4C6
Error: Corrupted FLAC stream
Is there a way to investigate where that mysterious error is within the file? I could not hear any problems after decoding with flac133.
And has the algorithm of verifying changed along the versions?

Re: FLAC v1.4.x: Strange decoding error

Reply #3 –
So, as suggested, this is probably a file with a ID3v1 tag. If you want to make sure, try to re-encode the file with for example `flac input.flac -o test.flac`. If the ID3v1 tag is the cause, re-encoding finished without an error (for now at least, this is actually a bug)
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.x: Strange decoding error

Reply #4 –
The re-coding works without an error.
But there is no ID3v1 tag at the end of the file - although there seems to be garbage at the end of the file.
I'm by no means an expert for FLAC frames and how they should look, but the end of the file looks very different from rest of the audio data block (tons of 0xFF)

Re: FLAC v1.4.x: Strange decoding error

Reply #5 –
Are you by any chance able to send only this last bit of the file? Maybe I can make sense of it.

Anyway, FLAC doesn't know about ID3v1, so whether there is a valid tag or garbage makes no difference for its behaviour. As the file passed testing with earlier versions of FLAC and re-encodes correctly, we can only conclude this is no unreadable or otherwise malformed FLAC frame at the end.

Another way to test is by using FLAC 1.4.x with flac -tF file.flac. If after spitting out errors it says 'ok' the MD5 checksum checks out. Otherwise it will let you know the MD5sum is incorrect, so it is still possible to test the MD5sum of these kind of files with new FLAC versions.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.x: Strange decoding error

Reply #6 –
Meanwhile I'm quite sure that garbage at the end of the file is the problem.
I could reproduce the error message when I added some (four) random bytes at the end of a good file.
The very end of the malicous file is like this:
Code: [Select]
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

02072C60  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072C70  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072C80  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072C90  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072CA0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072CB0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072CC0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072CD0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072CE0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF F0  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿð
02072CF0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D00  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D10  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D20  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D30  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D40  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D50  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D60  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D70  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D80  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072D90  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072DA0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072DB0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072DC0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072DD0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072DE0  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF  ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
02072DF0  47 44 FF F8 C9 18 E0 BD 98 6E 00 00 00 00 00 00  GDÿøÉ.མn......
02072E00  66 96 FF F8 C9 18 E0 BD 99 69 00 00 00 00 00 00  f–ÿøÉ.ཙi......
02072E10  F1 E3 FF F8 C9 18 E0 BD 9A 60 00 00 00 00 00 00  ñãÿøÉ.ཚ`......
02072E20  C8 79 FF F8 C9 18 E0 BD 9B 67 00 00 00 00 00 00  ÈyÿøÉ.ཛg......
02072E30  5F 0C FF F8 C9 18 E0 BD 9C 72 00 00 00 00 00 00  _.ÿøÉ.ཛྷr......
02072E40  BB 4D FF F8 C9 18 E0 BD 9D 75 00 00 00 00 00 00  »MÿøÉ.à½.u......
02072E50  2C 38 FF F8 C9 18 E0 BD 9E 7C 00 00 00 00 00 00  ,8ÿøÉ.ཞ|......
02072E60  15 A2 FF F8 79 18 E0 BD 9F 0B 1F 07 00 00 00 00  .¢ÿøy.ཟ.......
02072E70  00 00 FB 79 43 C6 F8 FD AE 32 56 09 EA AA BD 91  ..ûyCÆøý®2V.ꪽ‘
02072E80  55 1D 0E C2 E3 54 1F AD 2A 65 DF F9 F8 47 89 B2  U..ÂãT..*eßùøG‰²
02072E90  6E 57 38 86 60 44 56 78 83 82 83 AE 99 4A 25 20  nW8†`DVxƒ‚ƒ®™J% 
02072EA0  44 21 35 AE A8 7A 17 C1 7C 4D 7D E6 CE E3 9B C4  D!5®¨z.Á|M}æÎã›Ä
02072EB0  45 D2 9D 8C 70 69 24 12 85 E3 E6 1E 62 2E 1A 4E  EÒ.Œpi$.…ãæ.b..N
02072EC0  39 E2 72 83 04 85 00 84 5D 33 D2 BC 68 A9 68 B0  9ârƒ.….„]3Ò¼h©h°
02072ED0  8C 65 31 EA 2B 46 05 30 43 AE 28 6A 2F 9E 58 67  Œe1ê+F.0C®(j/žXg
02072EE0  87 10 AB 30 37 15 08 22 1E 5C CC 68 6D 28 3B D8  ‡.«07..".\Ìhm(;Ø
02072EF0  86 44 CA 19 A3 B1 94 44 33 B7 CB 56 B5 97 9E 51  †DÊ.£±”D3·ËVµ—žQ
02072F00  04 29 C7 26 7D 64 F9 A7 6B 08 20 CE 41 FA FE 38  .)Ç&}dù§k. ÎAúþ8
02072F10  D4 64 15 B5 8D FE 44 FA E8 AD B4 15 05 E4 C5 8C  Ôd.µ.þDúè.´..äÅŒ
02072F20  97 D1 42 88 97 B0 88 68 93 16 F2 8F 0E 84 30 8B  —ÑBˆ—°ˆh“.ò..„0‹
02072F30  F8 F8 AC 67 33 91 1A BF 84 5E 26 02 F0 C0        øø¬g3‘.¿„^&.ðÀ
But the problem is not the 0xFFs as I assumed. When I deleted the bytes from address 0x02072E74 (43 C6 F8...) [after the last "good" frame header 0xFF 0xF8] to the end of the file (202 Bytes), flac14x tested it OK.

Quote
Anyway, FLAC doesn't know about ID3v1, so whether there is a valid tag or garbage makes no difference for its behaviour.
According to the documentation @ xiph it does:
Quote
As a convenience, the reference decoder knows how to skip ID3v1 and ID3v2 tags. Note however that the FLAC specification does not require compliant implementations to support ID3 in any form and their use is strongly discouraged.
The -tF test did as you expected: There was an OK after the "FLAC__STREAM_DECODER_END_OF_STREAM" error message.
Btw. If a Flac file didn't have an MD5 hash in the StreamInfo block, how does flac justify its test being OK?
 

Re: FLAC v1.4.x: Strange decoding error

Reply #7 –
Thanks for all the replies. I found this open issue, so it's already in the works...  ;)

Re: FLAC v1.4.x: Strange decoding error

Reply #8 –
Already in the works if it is the same thing happening.
It is probably a good idea if you could give ktf a copy of the file, because if you have a slightly different issue, one might want to fix that as well while at it.

As for the specification ... is it even compliant to just throw a chunk of arbitrary payload at the end?
I mean the doc is pretty clear about f**c that shit!, but the spec does not outright prescribe what is to happen after the last audio frame?

Re: FLAC v1.4.x: Strange decoding error

Reply #9 –
Quote from: sundance on
Thanks for all the replies. I found this open issue, so it's already in the works...  ;)
I'm not sure what you expect to be fixed about what you're dealing with. There is no going back to the situation as it was before, because this is actually the result of fixing a security issue.

Quote from: sundance on
But the problem is not the 0xFFs as I assumed. When I deleted the bytes from address 0x02072E74 (43 C6 F8...) [after the last "good" frame header 0xFF 0xF8] to the end of the file (202 Bytes), flac14x tested it OK.
No clue what it is, not a ID3v1 indeed.

Quote
Quote
Anyway, FLAC doesn't know about ID3v1, so whether there is a valid tag or garbage makes no difference for its behaviour.
According to the documentation @ xiph it does:
Quote
As a convenience, the reference decoder knows how to skip ID3v1 and ID3v2 tags. Note however that the FLAC specification does not require compliant implementations to support ID3 in any form and their use is strongly discouraged.
This is outdated as is clear now, and it doesn't really say what happens. The thing is: FLAC used to quit decoding a file when it thought it was done, which implicitly skips an ID3v1 tag if the file is otherwise valid. It doesn't detect anything about such a tag, it just used to quit right before it. ID3v2 is indeed something that is detected and skipped.

Quote
Btw. If a Flac file didn't have an MD5 hash in the StreamInfo block, how does flac justify its test being OK?
No clue what it does actually. Should look whether it says ok then, if it does, it needs fixing.

Quote from: Porcus on
As for the specification ... is it even compliant to just throw a chunk of arbitrary payload at the end?
I mean the doc is pretty clear about f**c that shit!, but the spec does not outright prescribe what is to happen after the last audio frame?

The spec says the following

Quote
A FLAC bitstream consists of the fLaC (i.e. 0x664C6143) marker at the beginning of the stream, followed by a mandatory metadata block (called the STREAMINFO block), any number of other metadata blocks, then the audio frames.
This specifically excludes ID3v2 as I read it, because when an ID3v2 is inserted, the fLaC marker is no longer at the beginning of the stream. There is nothing specifically that says that a FLAC file MUST end after the last audio frame, but there isn't anything in there that allows it either.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.x: Strange decoding error

Reply #10 –
Quote from: ktf on
This specifically excludes ID3v2 as I read it, because when an ID3v2 is inserted, the fLaC marker is no longer at the beginning of the stream.
ID3v2.4 can actually be after the audio. At the very end, or even: after audio, before ID3v1. But that is maybe nearly as scarce as attempts to tag it with APEv2 ... oops  :-X

Quote from: ktf on
There is nothing specifically that says that a FLAC file MUST end after the last audio frame, but there isn't anything in there that allows it either.
Maybe that is good enough - and that it is better to instruct decoders to failsafe against whatever possibly-harmful might be put there, than to give the impression that there won't ever be anything?

Re: FLAC v1.4.x: Strange decoding error

Reply #11 –
Quote from: Porcus on
ID3v2.4 can actually be after the audio. At the very end, or even: after audio, before ID3v1. But that is maybe nearly as scarce as attempts to tag it with APEv2 ... oops  :-X
OK, didn't know. In that case: FLAC knows how to skip ID3v2 when it is at the start of the file. Perhaps I should include some code to warn users about that when they use flac -t.

Quote
Maybe that is good enough - and that it is better to instruct decoders to failsafe against whatever possibly-harmful might be put there, than to give the impression that there won't ever be anything?
There are quite some things listed in the security considerations: players are encouraged to make no assumptions about pretty much everything and properly fuzz an implementation. Parsing of files is very prone to security errors, that is not something exclusive to FLAC.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.x: Strange decoding error

Reply #12 –
Quote
I'm not sure what you expect to be fixed about what you're dealing with. There is no going back to the situation as it was before, because this is actually the result of fixing a security issue.
Maybe I wasn't clear enough here.
I'm not expecting this behaviour (= throwing a stream error when there is garbage, even if it was an ID3v1 tag - which could be considered garbage in a FLAC file) to be corrected. I prefer anything strange to be reported when I ask flac to test a file. It would be nice to have one or the other commented enhancements, e.g. verbosity of errors messages:
Quote
Thing that need to be done:
  • flac needs to detect when it has not reached end-of-file despite encoding the target number of samples. Currently it is not possible, not even with -F, to decode a file of which the total number of samples falls short without editing that number manually
  • flac needs to detect and warn the user about ID3v1 tags
  • flac needs to be able to re-encode files with an unknown number of samples
@ very low priority!

Re: FLAC v1.4.x: Strange decoding error

Reply #13 –
Quote
Btw. If a Flac file didn't have an MD5 hash in the StreamInfo block, how does flac justify its test being OK?
No clue what it does actually. Should look whether it says ok then, if it does, it needs fixing.
This is what happens when a FLAC is tested that has no MD5, that is all 16 bytes are 0x00:
Code: [Select]
flac 1.4.1
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

silence_noMD5.flac: WARNING, cannot check MD5 signature since it was unset in the STREAMINFO
ok
There is no difference across flac versions.

Code: [Select]
METADATA block #0
  type: 0 (STREAMINFO)
  is last: false
  length: 34
  minimum blocksize: 3584 samples
  maximum blocksize: 3584 samples
  minimum framesize: 392 bytes
  maximum framesize: 1255 bytes
  sample_rate: 44100 Hz
  channels: 2
  bits-per-sample: 16
  total samples: 44100
  MD5 signature: 00000000000000000000000000000000
METADATA block #1
  type: 3 (SEEKTABLE)
  is last: false
  length: 18
  seek points: 1
    point 0: sample_number=0, stream_offset=0, frame_samples=3584
METADATA block #2
  type: 4 (VORBIS_COMMENT)
  is last: false
  length: 131
  vendor string: reference libFLAC 1.4.1 20220922
  comments: 4
    comment[0]: TITLE=One Second of Silence
    comment[1]: ENCODERSETTINGS=-7
    comment[2]: DATE=2022
    comment[3]: ORGANIZATION=sundance
METADATA block #4
  type: 1 (PADDING)
  is last: true
  length: 1964

Re: FLAC v1.4.x: Strange decoding error

Reply #14 –
BTW, there were some FLAC encoders that added 4608 of garbage samples to the end of files. CUETools has option to detect and fix such files.

Re: FLAC v1.4.x: Strange decoding error

Reply #15 –
I actually meant I was wondering whether FLAC would still output 'ok' while there were errors when an MD5 sum was not available and errors were found.

I just tested this, and saw the following behaviour

  • When a file has an MD5 sum and trailing garbage, running flac -tF will output errors, and finish with 'ok'
  • When a file has an MD5 sum that doesn't check out, running flac -tF will output errors, and finish with 'ERROR, MD5 signature mismatch'
  • When a file has no MD5 and encounters errors, running flac -tF will output errors, and finish with 'ok'

I think that last bit is not as it is supposed to work. It should only output ok when either the MD5 checks out OR no errors are encountered.
Music: sounds arranged such that they construct feelings.