Skip to main content ↑ ↓

Notice

If you are using a Hotmail or Outlook email address, please change it now, as Microsoft is rejecting all email from our service outright.
Topic: Protecting audio files from bit rot? (Read 13438 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Protecting audio files from bit rot?

I don't know how many people here are familiar with the term "bit rot" but it's basically silent corruption of data.  Something like a malfunctioning hard drive controller or a loose cable can cause the bits to get flipped, corrupting your data.  If bit rot has occurred, a backup won't save your files because you will just be replacing the old good backup with a new bad one.  The two most common file systems these days - HFS+ (Mac OS X) and NTFS (Windows) do not protect against bit rot data corruption

While bit rot is silent corruption, people have discovered it by hearing the damage it has done to their audio files.   Ever heard an old MP3 with a very tiny blip of static? Guess what, that's 1 frame broken screwing up a few milliseconds worth of audio data.  Other annoying noises like an MP3 that started with a chirp, or a track that has a "click" in it, bad pop noises, or a song that skips ahead a few seconds are caused by bit rot.

So I am curious if anyone here has taken preventive measures to protect their audio files?  Are you storing your music library on newer, experimental file system on a different computer?  Maybe you zip all your albums and store them on external media so you can checksum them if the album has problems later on.  Or maybe you don't care at all and will re-rip your songs or albums if you find problems in the future.

Re: Protecting audio files from bit rot?

Reply #1 –
Backups are really the only thing you can do to protect against hardware failure.

Re: Protecting audio files from bit rot?

Reply #2 –
Use audio formats with checksum support, like flac and wavpack. It cannot prevent or avoid data corruption but you don't need to listen to the files one by one to verify file integrity. There are tools like this:

http://www.foobar2000.org/components/view/foo_verifier

That's one of the reasons why I don't archive audio files in wav format.

Re: Protecting audio files from bit rot?

Reply #3 –
Multiple back ups is your best defense before it ever starts to happen.  Always check files before you back up a recent change and if you notice problems with corruption, find the cause of it.  In a lot cases it maybe necessary to either wipe the drive or replace it depending on the cause of corruption.

Also check your back ups for signs of failure periodically.  Disguard any back up that's gone bad and replace it with new back up that's free of problems ASAP.

Offline back up with several external devices both on site and off site that are often not connected to any computer or network, is better because the possibility of malware attacks such as ransomware as well as fire and natural disasters.

Re: Protecting audio files from bit rot?

Reply #4 –
What a load of audiofail BS.  "Bit rot" *facepalm* .  There is nothing specific between you audio files and other data you dont need to safeguard it in any special way.

all the same methods apply

Backup
Data verification chekcsum/hash
Error recovery

You might want to look into .par files
Sven Bent - Denmark
 

Re: Protecting audio files from bit rot?

Reply #5 –
Quote from: sven_Bent on
What a load of audiofail BS.  "Bit rot" *facepalm* .  There is nothing specific between you audio files and other data you dont need to safeguard it in any special way.

all te same mthod apply

Backup
Data verification chekcsum/hash
Error recovery

You might want to look into .par files


Bit rot (based on the context of the original poster) is failure of the storage medium, it's another term for when a storage medium begins to lose small bits of information quietly.  He was talking about data degradation and worried about losing his collection to it.  Not any different from someone worrying about losing their photo collection to silent data corruption or any other personal or valuable data.

Terms get thrown around a lot and lots of times people can get confused with them.  Bit rot actually refers to software rot and many confuse it with data rot.

Re: Protecting audio files from bit rot?

Reply #6 –
The problem is that OP brings it up like its something special about audio. Its simple data corruption like any other data corruption just as likely/unlikely like anything else ( just like you say).  But  OP thinks that backups does not work because somehow the BIT's are rotted and copying over backup does not overwrite the "Bad bits" with good bits.
My point is there is nothing special to worry about in regards to audio data. and all the same methods apply that applies with any other kind of data.
If you blindly copy over you good backup with a new version. without knowing if it good or bad, then you backup method is wrong, it has nothing to do with audio data itself.
Which is also why i advised him to look into .par file which will be able to restore corrupted parts of files.  compared to storage consumption is far better than having multiple copies of the same files.
Sven Bent - Denmark

Re: Protecting audio files from bit rot?

Reply #7 –
Sven, actually he has a point, if you don't check that your files are good before doing a backup, and write the new backup over the old one, as many people do when backing up to a hard drive or even the cloud.  If a file has become corrupted without your knowledge, and you subsequently do a backup and overwrite an old, good backup, you have destroyed the last good copy of the file.  All too common because of the way most automatic backup software is set up.

Re: Protecting audio files from bit rot?

Reply #8 –
Quote from: sven_Bent on
What a load of audiofail BS.  "Bit rot" *facepalm* .  There is nothing specific between you audio files and other data you dont need to safeguard it in any special way.

all the same methods apply

Backup
Data verification chekcsum/hash
Error recovery

You might want to look into .par files

I never said there was a difference between audio files or other data like pictures.  I mention audio files because this is a forum dedicated to music...  In my music library I have found audio files that have been damaged in different ways by bit rot, but have replaced most.  You don't know it happens because it's silent corruption.  You won't know if you have a bad copy of a song or an album if your computer automatically backups your music library to a hard drive.

I am curious what your method is.

Re: Protecting audio files from bit rot?

Reply #9 –
You quoted him suggesting you look into .par files. Did you do that?  While you're at it, try zfs if you haven't already.

There are smart copy programs available. fb2k has a file integrity plugin.

If your automated backup system doesn't do:
Quote
Data verification chekcsum/hash
Error recovery
then you should stop using it.

Re: Protecting audio files from bit rot?

Reply #10 –
Quote from: deathcoreRULES on
Quote from: sven_Bent on
What a load of audiofail BS.  "Bit rot" *facepalm* .  There is nothing specific between you audio files and other data you dont need to safeguard it in any special way.

all the same methods apply

Backup
Data verification chekcsum/hash
Error recovery

You might want to look into .par files

I never said there was a difference between audio files or other data like pictures.  I mention audio files because this is a forum dedicated to music...  In my music library I have found audio files that have been damaged in different ways by bit rot, but have replaced most.  You don't know it happens because it's silent corruption.  You won't know if you have a bad copy of a song or an album if your computer automatically backups your music library to a hard drive.

I am curious what your method is.


I don't think I've ever seen an audio file damaged by bit rot. You'd have to be fairly unlucky to have an audible difference, or have a hard drive that was rapidly failing.  

Re: Protecting audio files from bit rot?

Reply #11 –
Hard drives, as well as optical media, are protected by cyclical redundancy checksums, or CRC, as well as some error correction codes. When a sector has been demolished by hardware failure, it won't simply result in flipped bits, it will result in attempts to recover the data, followed by outright read errors. This is more likely to happen, if at all, either due to defective hardware, or aging hardware.

"Flipped bits" may occur if the file is being read, then a bit flips on the way to memory, or in memory, but only if it's then rewritten to the file that way. I guess maybe you can try to guard against this happening by having a board capable of using ECC memory, and install some ECC memory, but then you're looking at crashes or other memory errors instead of silently losing your files. I still think this sort of thing is incredibly rare, though.

Now, flash memory, like in USB drives, memory cards, and solid state drives, can suffer from bit rot, if they are not kept powered on occasion. I'm not sure if they need to be explicitly refreshed, or simply powered on regularly. And I think I more recently heard that the bit rot factor doesn't really creep into the game until you're looking at near end of life devices, approaching unusable due most of their capacity already hitting its maximum write cycle count, and reallocations eating into the usable capacity. I could be wrong, though. I don't think anyone has done extensive testing of this, feel free to research this on your own.

Re: Protecting audio files from bit rot?

Reply #12 –
Quote from: kode54 on
Now, flash memory, like in USB drives, memory cards, and solid state drives, can suffer from bit rot, if they are not kept powered on occasion. I'm not sure if they need to be explicitly refreshed, or simply powered on regularly. And I think I more recently heard that the bit rot factor doesn't really creep into the game until you're looking at near end of life devices, approaching unusable due most of their capacity already hitting its maximum write cycle count, and reallocations eating into the usable capacity. I could be wrong, though. I don't think anyone has done extensive testing of this, feel free to research this on your own.

Flash memory is just starting to come into play with larger and larger capacities.  Time will tell us a lot about newer emerging technology such as solid state drives over traditional mechanical ones.  I be very interested in what the long term things that can happen with this type of technology is, without finding out the hard way on my own that is.  I'm not very trusting of it for backup purposes because of some of that stuff that has been reported.  I certain use flash drives and SD cards a lot though but those get plugged in a lot or used as storage on media players and phones.

Re: Protecting audio files from bit rot?

Reply #13 –
I remembered when I was still using Windows XP, there were really some audio and video files in my harddrive getting noticeably corrupted, but I suspect I could have ignored those error like some of the chkdsk reports or lost clusters due to power failure or unexpected reboots. Couldn't remember if the disk/partition was using FAT32 or NTFS though. Never happened again since I upgraded to Windows 7.

In case of CDR data corruption the drive will simply retry and spin like crazy and throw a CRC error.

A good thing about audio file format with internal checksum support is the checksum will not be affected by metadata so updating tags will not change the checksum. However it seems that checksum is not popular among lossy formats, but at least wavpack supports it.

Re: Protecting audio files from bit rot?

Reply #14 –
Many file formats have checksums built-in that can be used to detect bit rot with the right tools.  If I may shamelessly plug my own freeware, I have written a couple bit rot detectors available here:

https://Mediags.codeplex.com

The Mediags console program will sweep your directories and report bit rot in .flac, .mp3 (LAME only), and other file type by verifying all available checksums.  This is possible because these formats store internal checksums of their data.  Formats such as ALAC cannot be checked this way because they have no internal checksums.

The site above also features the UberFLAC product (console & WPF) which will do the same checks as Mediags on your .flac files
and will also do deep analysis of an EAC rip which includes verifying .flac files against the CRC-32 values in the EAC log.

These tools are intended for archivists who care about data integrity down to the bit.  These apps require .NET 4 and are purely portable.

Re: Protecting audio files from bit rot?

Reply #15 –
FWIW, I've never been disappointed (so far) by using separate audio drives and/or partitions, thus keeping audio data apart from the system drive.

Re: Protecting audio files from bit rot?

Reply #16 –
Quote from: saratoga on
Quote from: deathcoreRULES on
Quote from: sven_Bent on
What a load of audiofail BS.  "Bit rot" *facepalm* .  There is nothing specific between you audio files and other data you dont need to safeguard it in any special way.

all the same methods apply

Backup
Data verification chekcsum/hash
Error recovery

You might want to look into .par files

I never said there was a difference between audio files or other data like pictures.  I mention audio files because this is a forum dedicated to music...  In my music library I have found audio files that have been damaged in different ways by bit rot, but have replaced most.  You don't know it happens because it's silent corruption.  You won't know if you have a bad copy of a song or an album if your computer automatically backups your music library to a hard drive.

I am curious what your method is.


I don't think I've ever seen an audio file damaged by bit rot. You'd have to be fairly unlucky to have an audible difference, or have a hard drive that was rapidly failing.  
Quote from: bennetng on
I remembered when I was still using Windows XP, there were really some audio and video files in my harddrive getting noticeably corrupted, but I suspect I could have ignored those error like some of the chkdsk reports or lost clusters due to power failure or unexpected reboots. Couldn't remember if the disk/partition was using FAT32 or NTFS though. Never happened again since I upgraded to Windows 7.

In case of CDR data corruption the drive will simply retry and spin like crazy and throw a CRC error.

A good thing about audio file format with internal checksum support is the checksum will not be affected by metadata so updating tags will not change the checksum. However it seems that checksum is not popular among lossy formats, but at least wavpack supports it.
Things go wrong with computers. When things go wrong with hard disks, it can cost us our data. If that data is primarilly music, well... it's music. That is not saying that there is anything  special about data files that happen to contain music.  We, the end users, are not necessarily expected to know and use the precisely correct technical terms.

Quote from: Kees de Visser on
FWIW, I've never been disappointed (so far) by using separate audio drives and/or partitions, thus keeping audio data apart from the system drive.

What, in case spreadsheet data starts leaking numbers into the audio? ;)

It might make sense to keep filesystems that regularly have data added to them, but relatively rarely have it deleted or moved around. I don't have any justification for that: it is intuitive, which means I could easily by BSing!

I have had one audio file that went bad, in that it was playable from end to end, but the sound was heavily distorted. I never found out how that happened, or if (quite probable) it was something that I did.

Fast-failing hard disks are all too common. What I suspect is that it starts as slow failure, nothing that the system cannot handle, and thus is not noticed. Then it escalates and data becomes unreadable. Any hard disk that starts to make odd noises or give errors should be replaced urgently. 
The most important audio cables are the ones in the brain

Re: Protecting audio files from bit rot?

Reply #17 –
Quote from: Thad E Ginathom on
Quote from: bennetng on
I remembered when I was still using Windows XP, there were really some audio and video files in my harddrive getting noticeably corrupted, but I suspect I could have ignored those error like some of the chkdsk reports or lost clusters due to power failure or unexpected reboots. Couldn't remember if the disk/partition was using FAT32 or NTFS though. Never happened again since I upgraded to Windows 7.

In case of CDR data corruption the drive will simply retry and spin like crazy and throw a CRC error.

A good thing about audio file format with internal checksum support is the checksum will not be affected by metadata so updating tags will not change the checksum. However it seems that checksum is not popular among lossy formats, but at least wavpack supports it.
Things go wrong with computers. When things go wrong with hard disks, it can cost us our data. If that data is primarilly music, well... it's music. That is not saying that there is anything  special about data files that happen to contain music.  We, the end users, are not necessarily expected to know and use the precisely correct technical terms.


It seems you quoted something that I didn't write in my original post. Maybe they are actually part of your reply?

Re: Protecting audio files from bit rot?

Reply #18 –
The key thing to remember is that on mechanical hard drives (and likely flash memory) is that read and ECC errors are often followed by more errors and then soon after by hardware failure:

https://www.backblaze.com/blog/hard-drive-smart-stats/

If you look at that data, as soon as a hard drive has even one ECC error, it becomes likely to fail completely, and if it has more than one, it's likely to fail even sooner.  

For most people "bitrot" is only going to be noticed when the system stops booting or the drive refusing to spin up, at which point all data will be lost.  For this reason you really need a back up.  If you're worried about your backup's integrity, use a format with CRC or compute an MD5 hash of the backed up files.

Re: Protecting audio files from bit rot?

Reply #19 –
Greynol already told you the answer. ZFS (or btrfs). I've been bitten by bitrot before when I had a drive silently corrupt hundreds of pictures. I didn't catch it until I randomly looked through the pictures. Once found I went back through my backups and found all the backups except my very oldest (and due to be destroyed) copy had all been quietly copied with errors. I was saved only by the fact I was lazy about destroying my oldest backup that one time. I switched to a file system that offers automatic fault detection and correction. ZFS. Problem solved.

Re: Protecting audio files from bit rot?

Reply #20 –
Lots of reasons a person might experience audio anomalies such as the OP describes, but I'd think if the files themselves contained such faulty data, it would be plainly visible using tools like Audacity.

Thinking there are a couple of likely possibilities:

1. Data was always faulty, but maybe previous playback systems were more fault-tolerant, else listener simply didn't notice before.

2. Something in the digital delivery and/or playback chain isn't right.

When I experience such things, it's generally when I'm streaming tunes over the internet, and the provider seems to be experiencing network problems. Sometimes switching to another station and back in order to clear out the buffer is all it needs, else I tune in later in the day.

Experimental filesystems: Anyone who knows what that means knows not to store important data there.

Malware: Get protected! This includes Mac OS X: I encountered my first OS X-specific malware last year, it was a pain to clean out, and this was only one of the more "benign" sorts of adware, not something really insidious, like ransomware.

Re: Protecting audio files from bit rot?

Reply #21 –
Quote from: schmidj on
Sven, actually he has a point, if you don't check that your files are good before doing a backup, and write the new backup over the old one, as many people do when backing up to a hard drive or even the cloud.  If a file has become corrupted without your knowledge, and you subsequently do a backup and overwrite an old, good backup, you have destroyed the last good copy of the file.  All too common because of the way most automatic backup software is set up.

We must be reading different stuff. Deathcore says backups does not protect against it
I am the one bring up the fact that it only if you are doing backups the wrong way.
"If you blindly copy over you good backup with a new version. without knowing if it good or bad, then you backup method is wrong, "
right above your post. So I don't know why you want to point my own point out to me...
if you keep destroying your backup with at new version you are logically only able to restore from your last backup. basically the entire backup method id done bad, if this is how you do it.

Most storage media on the hardware level invisible from the user has ECC built into it and will report once these goes into extreme usage and backups might be needed. AHEAD of time for bits being lost/flipped/wrong. it is also on you internet data transfer protocol
And if you make sure you computer is working properly this is not a problem for bit rot.

Quote from: Thad E Ginathom on
Things go wrong with computers. When things go wrong with hard disks, it can cost us our data. If that data is primarilly music, well... it's music.

If you loose important data because you drive goes bad. your backup methods is wrong.
If you loose you important data because you house goes on fire. you backup method is wrong.
almost anything less than total annihilation of earth, you can do a proper backup against

My ordinary data is on a raid5. one drive can go wrong and i can recover it.  really important data that i can never retrieve again i also have on a removable  USB HDD, in case the house goes on fire and i need to grab it quickly with me, or if 2 drives should fail i have a backup to recover it
I have it on a  optical media roughly 7400 KM/s away away that i update roughly once a  year.
If my house get nuked and the entire state of Texas is a burning inferno.  my important data is still intact.

There is nothing new to do this, it has always been that way that something my go bad. and common sense of proper backup will save you.
To bring up such an old, solved and unchanged topic  and in a way where it only audio data is brought up. yes it looks like a huge pile of audiofail. When DeathCore goes into deep details about one bit in a frame bla bla it seems like he thinks it only about audio data. its hard to see he is talking about general data when he only talk about specific data. and again. Its nothing new....

The chances for this Bit rot t is very small (outside of actual disc rot).  for it to happen if you are during proper backups is almost impossible.
Quote from: deathcoreRULES on
I never said there was a difference between audio files or other data like pictures.  I mention audio files because this is a forum dedicated to music...  In my music library I have found audio files that have been damaged in different ways by bit rot, but have replaced most.  You don't know it happens because it's silent corruption.  You won't know if you have a bad copy of a song or an album if your computer automatically backups your music library to a hard drive.

I am curious what your method is.

When you bring up a topic but only bring it up in a very specific sub context ,its really hard to see that you mean the general and not the specifics are you pinpoint out. when you say backups does not work on audio data since you are only talking about audio data. that is a different towards the normal data cause in general  backups works,
So assuming the general very old knowledge that backups works against data corruption and you trying to say it does not work against audio data. Can you kinda see how its easy seeing it as  if you think audio data need special attention?
The alternative was that I would think you don't even know backups works against data corruption. But i didn't think you would know so little to backups.

if you regularly have bitroot. you need to find the source of it. it is NOT something that happens in a ordinary system. its a fault.
mostly because most people accept unstable computer as its a ordinary system.
if this is a recurring effect you need to test your hardware
- memory test (memtest86 or press f7 during windows load and change it to extende test)
- CPU test ( Prime95 or linpack. anything lesse is not gonna max stresstest you CPU
- HDD test (HDD factory has test software to use

To proff you against data corruption. you need to apply a proper backup/restore method
checksum/hash files will help you identify  good/bad data.
If you blindly copy data that has not changed you are doing it wrong and inviting mistake of overwritten good data with bad data
Jave more than one line of backups. i you are really  saa nervous about a single bit flip , which is soo rare to happen. you need multiple backupsfrom multipel diffrent times 1 yyeards old. 1 month old  1 week old c
Par files will help you restore corrupted data incase something goes wrong with you backups as well. use it for anything you want to longtime storage.

Still it a very rare thing to happen if you are using a working properly computer
Sven Bent - Denmark

Re: Protecting audio files from bit rot?

Reply #22 –
Quote from: deathcoreRULES on
Ever heard an old MP3 with a very tiny blip of static? Guess what, that's 1 frame broken screwing up a few milliseconds worth of audio data.

I was always under the impression that such blips were not really silent corruption, rather someone missing the last segment of a p2p download, possibly due to copying before as the p2p client was about to finish, but had not yet written to file the last few bits.
Also I suspect that artifacts at the start are often non-audio written by buggy software, and interpreted as audio by some player.

Anyway, this is not really on-topic to the big question.

Quote from: deathcoreRULES on
So I am curious if anyone here has taken preventive measures to protect their audio files?  Are you storing your music library on newer, experimental file system on a different computer?  Maybe you zip all your albums and store them on external media so you can checksum them if the album has problems later on.

No reason to use .zip for checksum if the audio is lossless - FLAC and WavPack are checksummed formats. For lossy files, one is stuck with the codec. (Is there BTW any simple utility that calculates md5 from audio and writes to tag, and can verify?)

I am not so sure if "experimental" is an appropriate term for the the file systems in question (zfs/btrfs) ... unless you were thinking of something else?
High Voltage socket-nose-avatar

Re: Protecting audio files from bit rot?

Reply #23 –
Quote from: 4season on
Lots of reasons a person might experience audio anomalies such as the OP describes, but I'd think if the files themselves contained such faulty data, it would be plainly visible using tools like Audacity.

I absolutely do not want to rely on inspecting every second of every file manually.
High Voltage socket-nose-avatar

Re: Protecting audio files from bit rot?

Reply #24 –
Quote from: bennetng on
Quote from: Thad E Ginathom on
Quote from: bennetng on
I remembered when I was still using Windows XP, there were really some audio and video files in my harddrive getting noticeably corrupted, but I suspect I could have ignored those error like some of the chkdsk reports or lost clusters due to power failure or unexpected reboots. Couldn't remember if the disk/partition was using FAT32 or NTFS though. Never happened again since I upgraded to Windows 7.

In case of CDR data corruption the drive will simply retry and spin like crazy and throw a CRC error.

A good thing about audio file format with internal checksum support is the checksum will not be affected by metadata so updating tags will not change the checksum. However it seems that checksum is not popular among lossy formats, but at least wavpack supports it.
Things go wrong with computers. When things go wrong with hard disks, it can cost us our data. If that data is primarilly music, well... it's music. That is not saying that there is anything  special about data files that happen to contain music.  We, the end users, are not necessarily expected to know and use the precisely correct technical terms.


It seems you quoted something that I didn't write in my original post. Maybe they are actually part of your reply?


"Things go wrong with computers. When things go wrong with hard disks, it can cost us our data. If that data is primarilly music, well... it's music. That is not saying that there is anything  special about data files that happen to contain music.  We, the end users, are not necessarily expected to know and use the precisely correct technical terms." --- mine

Yes sorry, looks like I screwed up there. In which case I probably made other mistakes too. Apologies to anybody else if I accidentally put words in their mouths or attributed their quotes to another perosn.
The most important audio cables are the ones in the brain
 
SimplePortal 1.0.0 RC1 © 2008-2021