Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC Reliability (Read 4879 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

FLAC Reliability

I am noticing a trend with my FLAC audio collection -- files consistently become corrupt given time.   I find many people bragging about recoverability of FLAC files, I question this though.  If a FLAC file mysteriously becomes invalid due to FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC as an example (I see this many times when I expect my files to be valid for the life of my media storage.  When I do encounter this error that results in loss of my FLAC, am I guaranteed that ignoring this error during decode followed by re-encoding the FLAC will result in lossless audio?

Re: FLAC Reliability

Reply #2 –
Quote from: Octocontrabass on
Quote from: Dryst on
files consistently become corrupt given time.
That's not normal. Your computer may be broken..
HDD/SSD failing and needing to be replaced for @Dryst?

Re: FLAC Reliability

Reply #3 –
Quote from: Dryst on
I am noticing a trend with my FLAC audio collection -- files consistently become corrupt given time.   I find many people bragging about recoverability of FLAC files, I question this though.  If a FLAC file mysteriously becomes invalid due to FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC as an example (I see this many times when I expect my files to be valid for the life of my media storage.  When I do encounter this error that results in loss of my FLAC, am I guaranteed that ignoring this error during decode followed by re-encoding the FLAC will result in lossless audio?

Something in your system is failing and needs replacement or repair.  You need to be keeping proper back ups and you need to make sure those work and are correct regardless of what format you're using, period.  There is no exceptions to this.  FLAC has some error checking abilities to make this easy to check that your files are correct and can be decoded back to original uncompressed PCM stream.  If you're getting errors like this, you're losing bits and pieces of your collection that can never be recovered.  FLACs don't become corrupt on their own over time, it's a very reliable format if you use common computer sense in keeping proper back ups.  The whole FLAC can handle a flipped bit is great if you're streaming or broadcasting but that's it, if it's happening to actual files that playback off a device or bits and pieces of your collection, it means you're losing your collection.  Lost bits are no longer lossless because you lost something and simply re-encoding will not fix anything nor will it recover anything that was lost, it'll just make the error message go away but it won't fix the actual problem with your failing system as it'll simply continue anyway and might actually make things even worse!

Re: FLAC Reliability

Reply #4 –
Quote from: Dryst on
If a FLAC file mysteriously becomes invalid due to FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC as an example
This can be caused by something other than hardware failure, for example:
https://hydrogenaud.io/index.php/topic,123259.0.html
https://hydrogenaud.io/index.php/topic,123219.msg1017955.html#msg1017955

Basically, software that make use of the more recent flac library are more picky about how metadata is stored.

Re: FLAC Reliability

Reply #5 –
Storage is not infallible I've had corruption, it's one of those things you need to mitigate if the data is critical it's not flac-specific. Two things you should do to mitigate:
  • Backup
  • Use a resilient filesystem like ZFS, BTRFS or BcacheFS. Filesystems that have checksumming and copy-on-write. Some/all of error detection, correction, redundancy, recoverability

Re: FLAC Reliability

Reply #6 –
I am on the road right now and cannot check:

* Best-case fond hope first - what error message does what version of the reference decoder give when the it encounters ID3?
Newer decoders got more zealous, and will scream at files that earlier ones would let pass.
If you try foobar2000, it will tell you if the MD5 matches. If it does, then you might be lucky. It isn't hard to remove ID3.

* As others have pointed out, the FLAC format itself does not corrupt over time, but the FLAC format can only detect, not repair - and more: a corrupted frame (typically 4096 samples) will be lost.
If you don't have a backup, then you have small chances at recovering. If there is just a single error, then maybe MAYBE if it is a CD rip, CUETools can repair (it has redundancy audio data in an external database). I don't remember how many consecutive errors it can handle, but 4096 is seven or eight CD frames.

* The drive might be dying yes. But, the error might also have occurred upon writing by faulty RAM. Or by some tagging application.
Also, the error might have occurred by faulty data transfer - I once had that issue when my mobo failed (on USB first).

 * Anyway, what to do. Layman suggestion:
(1) Run a memory test on the computer. If that is faulty ... uh-oh. Lots of other files could be written faulty to the drive.
(2) See what backups you have. If the drive is about to fail, then you want to give priority to reading what you already have.
(3) Test the backup too. In "the most lucky case" with ID3, they might even have been there all the time.
(4) Copy what you don't have backup of.
(5) Make another copy of the copy. One to work with, one to leave untouched until you are done.
(6) Test what you copied.
Now you have an idea of how much might be damaged.

Re: FLAC Reliability

Reply #7 –
Thank you for all of your comments.

Note that I do have back-ups and I use TeraCopy to copy + verify to my USB3 SSD backup.  I do this routinely.  Additionally, I primaraily make online purchases from sites that allow re-downloads (bandcamp, JunoDownload, 7Digital.  I absolutely try to avoid HDTracks at all cost.  They follow an old-fashioned purchase model that their competitors have long abandoned.

I also stopped using AudioTester, because it will show FLAC files validate fine when FLAC.EXE -t shows a failure.

Re: FLAC Reliability

Reply #8 –
Quote from: bennetng on
Quote from: Dryst on
If a FLAC file mysteriously becomes invalid due to FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC as an example
This can be caused by something other than hardware failure, for example:
https://hydrogenaud.io/index.php/topic,123259.0.html
https://hydrogenaud.io/index.php/topic,123219.msg1017955.html#msg1017955

Basically, software that make use of the more recent flac library are more picky about how metadata is stored.

I routinely use the latest available FLAC -- 1.4.3 currently on my WIndows 11 machine.  I have FLAC 1.4.2 installed on my SFTP server that contains my music library.  I use RaiDrive to map my SFTP server to a Windows drive.  fb2k plays content from this mapped SFTP drive.

I am currently running FLAC.EXE -t against all FLAC files right now on my server.  It will be finished in about three more days.  I routinely run this, and all content currently passes.  When I encounter any failures, I immediately replace failed content with my backed-up copy.

I was previously using pCloud, but I have given up on pCloud.  I now rent an SFTP server that I use for cloud storage.

Re: FLAC Reliability

Reply #9 –
Files should not change after they have been written, so something is very wrong on your system. Replacing the files from backups is not the answer. Find the cause.

Re: FLAC Reliability

Reply #10 –
Posts above are sharing the same issue experienced by others, so I am not alone.  It very well could be a bug.

Re: FLAC Reliability

Reply #11 –
I have as the developer of dBpoweramp been involved with FLAC since it was invented, never once have I seen Flac introduce an error into the encoding. We are talking in that time around 1 billion files encoded with dBpoweramp

Re: FLAC Reliability

Reply #12 –
FLAC is very good at detecting errors, as mentioned above correcting it MAY be possible if it is a properly done CD rip. The important thing is that you know when you have an error and can have the opportunity to correct things however possible.


All data is susceptible to corruption over time. Even moreso  now that SSDs are overtaking HDDs in popularity and people are using them for “archival storage” - something they are not equally well suited for.

The simple “keeping my files on a USB drive” doesn’t really cut it for data integrity. Unfortunately this is what I’m stuck with for the time being, but I plan on migrating all my data to a BTRFS RAID 6 array as funds allow.

Re: FLAC Reliability

Reply #13 –
@Dryst and others:
* If I remember correctly (again, I'm not at home and cannot test), reference FLAC 1.4.x will scream at two types of errors that might pass AudioTester or FLAC 1.3.x:
. One is ID3 tags. If you have gotten those - by some tagging application - and that is the only issue you have, then you are lucky.
. When a 16 bit FLAC exceeds 16 bits (or similar with other bit depths). Then it isn't encoded from say .wav - there have been such cases when some audio manipulation tool has created those in floating-point and written them to say, a "16 bit" FLAC without checking that the numbers are small enough for 16 bits. In those cases one can only guess what was "intended".
. (I don't remember if AudioTester will report if an MD5 is missing.)
* The "lucky best case" could be caused by subsequent tagging with an application that is willing to write ID3 tags to FLAC files. There are portable or in-car units that "require" that ...
* Unfortunately, retagging using a buggy application might destroy audio, if it writes tags into where the audio should start.
* I have had cases where some FLAC encoder made a nasty error: full drive.

Re: FLAC Reliability

Reply #14 –
This is clearly a case of storage failing. This happened to me. Once I went and ran the integrity check test from foobar2000 on some FLAC laying around for ages on my old HDD. Several tracks were corrupt, but bad blocks were starting to appear. I ran the HDD test and the test did not pass the device. So I backed up the non affected albums and replaced the drive. Re-ripped the affected ones. Never in more than 20 years using FLAC, I've seen corruption "out of the blue", neither in a Linux system nor in a Windows system. This is hardware for sure. Consider getting a SSD with good reputation such Samsung EVO series or Intel's SSD.

Re: FLAC Reliability

Reply #15 –
Quote from: krafty on
This is clearly a case of storage failing.
Could be, but not "clearly" so.

A software update that invalidates file formats, which previous versions of the software accepted, could produce these results without any alteration of the file.  This can easily happen when there is widespread disagreement on precise details of file specification.

This is why I take the policy of not fixing what ain't broke, and only updating software when absolutely necessary – in full knowledge of the attendant risk.
It's your privilege to disagree, but that doesn't make you right and me wrong.

Re: FLAC Reliability

Reply #16 –
Quote from: Dryst on
When I do encounter this error that results in loss of my FLAC, am I guaranteed that ignoring this error during decode followed by re-encoding the FLAC will result in lossless audio?
As any version of the flac command line will tell you, flac comes with ABSOLUTELY NO WARRANTY.  This is free software. So, no guarantee, ever. Without any further information on your specific case, there is not much to say about it really.

A few more remarks:
- The FLAC format has no error-correction of any kind build in. It does quite a good job at detecting them.
- Recent versions of the FLAC command line tools have improved: they detect errors earlier versions don't. I consider this a feature, not a bug, because they are errors nonetheless.
- I agree that the FLAC command line tool could be more specific in explaining what went wrong, specifically.

Quote from: fooball on
This can easily happen when there is widespread disagreement on precise details of file specification.
Not sure why you mention this, because there is no such disagreement for FLAC. In fact, the FLAC format has recently been meticulously documented and the found ambiguities are benign and well understood.
Music: sounds arranged such that they construct feelings.

Re: FLAC Reliability

Reply #17 –
Quote from: ktf on
Quote from: fooball on
This can easily happen when there is widespread disagreement on precise details of file specification.
Not sure why you mention this, because there is no such disagreement for FLAC. In fact, the FLAC format has recently been meticulously documented and the found ambiguities are benign and well understood.
Really?  OK, perhaps I should have said implementation or interpretation of the specification.  By this I mean, as an example, the possibility that a tagging application might write ID3 tags instead of Vorbis tags.  Is that not playing free and loose with the FLAC specification?

However, regardless of what you say and no doubt the best intentions of the FLAC community, as a career engineer I can attest to how difficult it is to word exact details of specification totally unambiguously.  You only have to look at the subtle incompatibilities in HDMI equipment to realise that manufacturers accidentally (or deliberately!) interpret the details differently, but can still demonstrate conformance to the HDMI specification.

It's your privilege to disagree, but that doesn't make you right and me wrong.

Re: FLAC Reliability

Reply #18 –
When it comes to the "best-case" scenario - where error/warning messages are not triggered by anything in the audio, not even in the leading FLAC metadata, but solely due to ID3 at the end - then it is surely up to discussion whether the (level of) ambiguity in the spec is "benign"; Also, given how ID3 found its way into .mp3 I would probably take it that some applications would append it even if the spec disallowed it, so the benevolence-or-not is maybe a moot question.

Re: FLAC Reliability

Reply #19 –
I recommend you to use a disk diagnostic tool such as CrystalDiskInfo (or anything you trust) to check its actual state, just to be sure. File corruption by no apparent reason is one important signal of storage failing, no matter it's an old HDD or a brand new SSD. I've almost lost my entire collection because of a failing HDD. Lesson learned: Always backup your stuff, even if you feel you don't need it at all!

Another common cause of file corruption is malware.

Ah, and don't try to fix anything until you know what's the cause of the issue!
 

Re: FLAC Reliability

Reply #20 –
I found a solution to this issue.  The solution is to stop using FLAC 1.4.x.  When I use versions no higher than 1.3.3 I no longer face this issue.

This may explain why it is absolutely impossible to purchase a FLAC download encoded using FLAC 1.4.x.  No one sells FLAC encoded with this.  This highest you will find is 1.3.3 from every site I have ever found that sells FLAC downloads.

This issue is resolved using FLAC 1.3.3.

Re: FLAC Reliability

Reply #21 –
Quote from: Dryst on
I found a solution to this issue.  The solution is to stop using FLAC 1.4.x.  When I use versions no higher than 1.3.3 I no longer face this issue.
No solution, but a work-around. These files are corrupt, but FLAC 1.3.3 is unable to detect the corruption. FLAC 1.4.x is able to, and will tell you that.

Quote
This may explain why it is absolutely impossible to purchase a FLAC download encoded using FLAC 1.4.x.  No one sells FLAC encoded with this.  This highest you will find is 1.3.3 from every site I have ever found that sells FLAC downloads.
This has nothing to do with it. FLAC 1.4.x is simply too new, it takes a while before software is updated, and files aren't usually re-encoded.

Quote
This issue is resolved using FLAC 1.3.3.
The issue is worked around using FLAC 1.3.3, but the files are still corrupt.
Music: sounds arranged such that they construct feelings.

Re: FLAC Reliability

Reply #22 –
Was this the same issue as https://hydrogenaud.io/index.php/topic,126633.0.html ? 

@Dryst :  As pointed out by the original FLAC dev in https://hydrogenaud.io/index.php/topic,61792.msg568080.html#msg568080 there was a known bug in Reaper, where in fact decoding with an old flac version that ignores the corruption, would magically resolve it - assuming it is precisely that kind of error. And from the thread on top, some of those files also fail MD5, which means it isn't precisely that. 

The longer story of that bug is as follows: a .flac file stores numbers. Integers. Within a certain range. That range is specified like "16 bits" or "32 bits", but it is "physically possible" to store a too big number even if the initial flag says "nothing here exceeds 16 bits". If you do write a too big number there ... you have done something wrong.
Old flac trusted you to write the right thing, so it would treat your numbers like a "16 bit odometer" would. You know, a motorcycle odometer with 3 digits: tell it "1007", and it will go full circle and translate it to "7". But flac 1.4 will scream and say, you promised me a signal that would fit in 3 digits and now you tell me you have "1007".
So flac 1.4 has surely found something that is wrong. But what? It could be that a small chunk of the file has been overwritten with some arbitrary garbage. It could be that your promise of "3" digits should have been "4". Or indeed: at least one Reaper version did sometimes store a "1007" thing when it should have stored "7".

Re: FLAC Reliability

Reply #23 –
Quote from: Dryst on
This may explain why it is absolutely impossible to purchase a FLAC download encoded using FLAC 1.4.x.  No one sells FLAC encoded with this.  This highest you will find is 1.3.3 from every site I have ever found that sells FLAC downloads.

I rarely purchase digital downloads, only when physical media is unavailable.  I was looking for a couple albums released only a couple years ago.   I was surprised to see them encoded with FLAC 1.2.x

Re: FLAC Reliability

Reply #24 –
Quote from: ktf on
it takes a while before software is updated, and files aren't usually re-encoded.
Don't know about all online stores, but Bandcamp at least in past was re-encoding when updating used version of FLAC.