Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: LONG-TERM audio archiving strategy? (Read 80822 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

LONG-TERM audio archiving strategy?

Reply #50 –
I never knew of Parchives until I ran into this thread. A great tool IMO, but I have a question (only slightly OT).

I want to add error recovery information to my CDs. Any error recovery system should take the typical errors of the medium into account, for CDs this is holes, scratches etc.
Now imagine you have a single scratch on your cd, going from the inside to the outside like a radius, maybe not all the way. IIUC, if you used a block size that is larger than one rotation of the CD (45kb on the outside if I calculated right) this will destroy all recovery blocks on the way, leaving the par recovery process useless even though 95% of the data might be read correctly. Even if you set the blocksize smaller, there will still be far larger dropouts than necessary. Very small blocksizes (2kb) might be a solution, but the efficiency of the format drops rapidly there, so you might need to save 100mb of recovery data to recover a maximum of 30 mb. Plus, I did a small test with 4kb blocksize, burned the files on a cd and scratched it as described (but trying to spare the TOC). Of about 1500 blocks in the recovery file only 300 were still usable. Maybe a different drive would give better results copying from a damaged CD (I'll do some more testing), but it's not a good rate.

So to my question, is there an error protection scheme that is better suited for CDs?

LONG-TERM audio archiving strategy?

Reply #51 –
I'll add my desires for a long term backup archive solutions, the product which meets all or most of the below requirements in an easy to use and flexible manner, will get my money (which so far I have not found).


I would like to see something which makes it easier to 'archive' a library to cd/dvd.

Here is how I imagine it:

I would like to start archiving files to DVD (or any potential media: tape/dat, ext drives/raid arrays, etc).

I would like to burn these tracks and keep a record of what track is burned to what media. (disc id's etc)

I would like a field or status somewhere to indicate that the track is archived (and where).

I would like a burn wizard to be part of the 'archiving' process.

I would like a context menu to add or 'send to' the selected node/tracks to the 'archive'.

I would like to add tracks to the 'archive' and then tell me when I have enough tracks to fill a whole disc (cd or dvd selectable or whatever media space).

I would like to include a backup snapshot of the library database in each burned archive. Obviously the newer the burn, the more recent the library database would be. This would be for recovery purposes...

I would like par2 (or quickpar) support.

http://parchive.sourceforge.net/docs/speci...ticle-spec.html

Meaning it would (optionally) automatically use a par2 client and burn the associated par2 files to the media.

This would be a huge increase in reliability as with parity data stored it would be possible to restore even in the case of scratched discs (albeit at cost of extra space used).



Recovery/import wizard:
Should your library/files ever get lost/corrupted/crashed, it should allow you to insert the most recent 'archive' media, import the library and restore any/all tracks select from the disc and all previously archived discs.

LONG-TERM audio archiving strategy?

Reply #52 –
There is integrated error recovery in CDs.  Some software, like CDRoller, will allow you to reread until you read the data correctly.  It works quite well, even in "holed" cds.

LONG-TERM audio archiving strategy?

Reply #53 –
Quote
' date='Aug 24 2006, 22:04' post='424116']
There is integrated error recovery in CDs.  Some software, like CDRoller, will allow you to reread until you read the data correctly.  It works quite well, even in "holed" cds.


Thanks for the hint to this prog, i will try it out.
However, I'm aware that CD-Roms come with error protection already. That's fine but as we know, this error protection often fails, so I'd like to add another layer for important/backup data as many people do by adding PAR files.

LONG-TERM audio archiving strategy?

Reply #54 –
Quote from: wylistener on
And to Beto's point, contracting with Iron Mountain solely for backups of a personal CD collection may be a tad overkill.

Yes, that is what I meant. I was looking to it under my case's point of view. For other people it may pay off, but not for me. It was not my intention to offend anyone.

LONG-TERM audio archiving strategy?

Reply #55 –
IMO, use dvdr to backup with par files and just not use any dvdr. I only buy dvdrs that have been made in Japan since that is where the best dye plants are. Those discs have been known to last 10 years or more easily. If you buy the bargain basement stuff, well, you get what you payed for and that is junk. To me, that is probably the cheapest solution since tape backup can be expensive. If you keep the discs in a dry dark place then they should last for years easily. Light I heard, can degrade the dye on the disc over time. Taiyo Yuden dvdrs are probably some of the best out there. Some companies rebadge them like Sony and usually put professional quality on the packaging even though they are just regular Taiyo Yudens. I can buy 50 Sony Taiyo Yuden spindle for a litlle over $24 at Wal-Mart. That is 235gbs at a cost of 9.79 cents/gb or just under 50 cents a disc!

LONG-TERM audio archiving strategy?

Reply #56 –
I have a collection of about a thousand CDs that I have ripped to tagged flac files with album art on a 320 GB hard drive. I bought a second 320 GB hard drive for backup and had planned on storing the backup drive in my safe deposit box. Is there any special packing of the hard drive that i should consider? Woudl there be any reason to be concerned about the safety of the drive at the bank (magnetic fields, etc.)? How often would I need to check that drive to assure the electronics don't rot and that the files are still there. Would annually be enough? Does this seem like a strong backup philosophy?

LONG-TERM audio archiving strategy?

Reply #57 –
Quote from: cpalcott on
I have a collection of about a thousand CDs that I have ripped to tagged flac files with album art on a 320 GB hard drive. I bought a second 320 GB hard drive for backup and had planned on storing the backup drive in my safe deposit box. Is there any special packing of the hard drive that i should consider? Woudl there be any reason to be concerned about the safety of the drive at the bank (magnetic fields, etc.)? How often would I need to check that drive to assure the electronics don't rot and that the files are still there. Would annually be enough? Does this seem like a strong backup philosophy?


If you are going that way, I would put the HDD in a antistatic bag and protect it from shock. If possible, vacuum seal it. HDDs have moving parts and like everything else that has moving parts they can break down even from lack of use even though the possibility of that is remote. I don't think that a magnetic field would be strong enough to effect the data on the disk at all if there was one at the bank. I would still recommend putting your data on DVDR or DVDR DL. The media doesn't have any moving parts and if you buy good media ,like I said ealier, then you won't have a problem. Just something to consider.

LONG-TERM audio archiving strategy?

Reply #58 –
Careful, Hard Disks are nown to have 'stiction' problems. See my prior posting.

It may be less frequent today, but due to the denser bits on the HD, riskier.

LONG-TERM audio archiving strategy?

Reply #59 –
I have spent way too much time ripping and organising my music collection.

I have RAID5 in my fileserver.
I have also RAID5 primary backup in my neighbors house. We back up everything every week.
My secondary backup is on DVD-R.
I have 10% redundancy stored on another set of DVD-Rs.

LONG-TERM audio archiving strategy?

Reply #60 –
I just found an interesting project that looks like exactly what I was looking for, i.e. an error correction designed with CD/DVD media protection in mind (for example, sector-sized recovery blocks, because this is the normal "unit of failure" on CDs and DVDs). Plus, it's still actively developed and comes with the capability to rip defective CDs, so you don't need a special recovery program. This is actually a really huge feature because the program can flag unreadable sectors (as indicated by the drive) as erasure, thus increasing the efficiency of the correction algorithm.

http://dvdisaster.sourceforge.net/en/index.html

I didn't do any real life tests with this yet, but I'll prepare a test CD now and give it a try.

LONG-TERM audio archiving strategy?

Reply #61 –
Thanks for the info, MedO! I will look forward to your test. If it really is good, then I think I will simplify things and archive to CD-R's / DVD-R's.

LONG-TERM audio archiving strategy?

Reply #62 –
OK, here is my result. Don't take it too serious, though.

Short version:
I burned a CD with error recovery information (25% redundancy), scratched it from both sides, read it again and was, after a few hours, able to reconstruct the original image. Pictures of the scratched CD:
Scratches on the lower side (a bit exaggerated because of the flash)
The biggest hole in the data layer
more holes

A few remarks:
I stopped the scratching at the right time; a bit more and the CD would probably not have been completely recoverable.
Thanks to the nice reading algorithm for scratched CDs, I was able to read 75% of the CD in a few minutes. However, for recovery about 80% were needed, and the remaining 5% searching for readable sectors took a few hours.
I originally wanted to scratch the CD with a wide radial scratch. The problem is that this gives read errors every few sectors, and because of that it takes ages to read a cd damaged like this. The special reading algorithm of the program would not have helped there, because it's designed for a situation where large parts of the CD are error-free.

All in all, the program looks very neat to me. I had some trouble with two old CD-drives, though, they could not be used for reading an image with this program. Also, when using the second ecc-method, the author warns that some drives may not read the "augmented" CDs. I had no trouble with this, and even the old drives were able to access the data on the CD. A plus again: the program is available for Linux as well (it's in Debian unstable for example), with exactly the same UI and features.

So I recommend to give it a try. Maybe you'll stick with it. I guess I will 

HINT: It is really helpful to read the information on the website/documentation first, to understand how this program works.

LONG-TERM audio archiving strategy?

Reply #63 –
By the way, is it a bad idea to store backup CDs in a spindle?

LONG-TERM audio archiving strategy?

Reply #64 –
I think so. The label-side of the CD underneath may stick to the data-side of the CD above it. Or even micro-scratch it.

LONG-TERM audio archiving strategy?

Reply #65 –
You know what? Whatever the result of the discussion here...

... will be distilled into the wiki.

Heck, check the wiki's Topic Index... there's a red-colored link right at the start of the index... a page waiting for us to fill in...

And if you've checked the history... that link is actually the first link created in the "General Guides" subsection! But no one has ever filled it...

... until now that is ... so let's keep discussing this topic healthily, and I'll try distilling this whole thread into that there wikipage.

LONG-TERM audio archiving strategy?

Reply #66 –
An unhelpful comment...

I think all the tests that both us, and the professionals, do to simulate ageing will prove inaccurate with time.

The effects of extreme light / heat / pressure / scratching are one thing we can test. I know they're supposed to let you predict 100 years of normal storage within a few hours/days of extreme testing, but if/when/where they don't, it will be too late by the time we know about the problem!

Active data management seems like the best bet. Then if you have two copies, at least you'll know when one has failed (rather than both!), and you can spot slow failures earlier, rather than later.

How normal people are supposed to find the time/money for this, I don't know.

Cheers,
David.

LONG-TERM audio archiving strategy?

Reply #67 –
Quote from: pepoluan on
Careful, Hard Disks are nown to have 'stiction' problems. See my prior posting.

It may be less frequent today, but due to the denser bits on the HD, riskier.


I think many modern HDs, retracts their heads off the disk surface. At least this should be the case for 2.5" HDs, conceived for portable equipment (laptops). Not sure for 3.5" HDs. With this disks stiction shouldn't be an issue.

Personally I backup my music (and more stuff) on external HDs (250GB LaCie USB) using Beyond Compare 2.

Sergio
Sergio
M-Audio Delta AP + Revox B150 + (JBL 4301B | Sennheiser Amperior | Sennheiser HD598)

LONG-TERM audio archiving strategy?

Reply #68 –
Quote from: smz on
Quote from: pepoluan on
Careful, Hard Disks are nown to have 'stiction' problems. See my prior posting.

It may be less frequent today, but due to the denser bits on the HD, riskier.
I think many modern HDs, retracts their heads off the disk surface. At least this should be the case for 2.5" HDs, conceived for portable equipment (laptops). Not sure for 3.5" HDs. With this disks stiction shouldn't be an issue.
Well, IIRC the heads still rest within the area of the platters, though no longer on a data-storing area. Stiction may still happen.

LONG-TERM audio archiving strategy?

Reply #69 –
Quote from: pepoluan on
Well, IIRC the heads still rest within the area of the platters, though no longer on a data-storing area. Stiction may still happen.

I don't think so. Read this excerpt from http://www.fujitsu.com/downloads/COMP/fcpa...e-single_wp.pdf:

Quote
To enhance the durability of HDDs, manufacturers utilize load/unload technology, in
which the heads of the hard disk drives retract from the platter during idle time as well
as power off. This feature protects the head from hitting the disk when not in operation,
which could otherwise be a potential cause of damage.

Cheers!

Sergio
Sergio
M-Audio Delta AP + Revox B150 + (JBL 4301B | Sennheiser Amperior | Sennheiser HD598)

LONG-TERM audio archiving strategy?

Reply #70 –
Best options (considering level of effort and price) to me are:

RAID 1 or RAID 10, with something like Vantec for easy removal of drives.  A 3 or 4 set rotation, with at least one set kept offsite, the second (of 4) at a separate offsite (for the very paranoid).  Finally, at least a monthly tape backup.

Now, my horror stories;.  Had a RAID 1 system at home, UPS'd, thought I was covered, then had the power supply go.  It popped chips on the drives PCB's as well as toasted the Controller!  Lost it all, BUT had tape backup and was able to recover.  Second story, had a house fire and lost everything.  So, offsite storage (possibly a safety deposit box!) is critical.  Be sure to put those digital photos in there as well.  Finally, had a 60GB notebook drive in a USB2 enclosure I took around to listen with friends.  Plugged into a separate USB hub that a friend subsequently powered with the wrong AC adapter; fried 60 GB of music.  That is a lot of work to replace, tagging, directory structure, scans, etc.  An easy backup that I did not do.

LONG-TERM audio archiving strategy?

Reply #71 –
Quote from: _eclectic_ on
Best options (considering level of effort and price) to me are:

RAID 1 or RAID 10, with something like Vantec for easy removal of drives.  A 3 or 4 set rotation, with at least one set kept offsite, the second (of 4) at a separate offsite (for the very paranoid).  Finally, at least a monthly tape backup.

Now, my horror stories;.  Had a RAID 1 system at home, UPS'd, thought I was covered, then had the power supply go.  It popped chips on the drives PCB's as well as toasted the Controller!  Lost it all, BUT had tape backup and was able to recover.  Second story, had a house fire and lost everything.  So, offsite storage (possibly a safety deposit box!) is critical.  Be sure to put those digital photos in there as well.  Finally, had a 60GB notebook drive in a USB2 enclosure I took around to listen with friends.  Plugged into a separate USB hub that a friend subsequently powered with the wrong AC adapter; fried 60 GB of music.  That is a lot of work to replace, tagging, directory structure, scans, etc.  An easy backup that I did not do.


Except for the fire (sad thing to happen), all the trouble was with HDs failing. This is exactly the reason why drives, even in a RAID, are a bad idea for backup. If the controller or moving parts of the drive fail, you'll need to pay a professional data rescue service to get the data from the platter. I think CDs are great because they do not have any moving parts that can fail, but tapes are simple enough as well. It's very unlikely for a CD to fail completely except in case of disaster, loss or theft, but for these cases you'll need offsite copies anyway.

A tip for everyone using CD backup or just generally interested in data recovery: do not throw away old CD drives if they still work. One of these drives might be able to read a CD the other ones give up on.

LONG-TERM audio archiving strategy?

Reply #72 –
If you ask me, the best is 2 different storage media (HD & DVD) both kept offsite (in case of fire which would wipe out all your backup no matter what raid was used).

LONG-TERM audio archiving strategy?

Reply #73 –
Quote from: pepoluan on
Quote from: smz on
Quote from: pepoluan on
Careful, Hard Disks are nown to have 'stiction' problems. See my prior posting.

It may be less frequent today, but due to the denser bits on the HD, riskier.
I think many modern HDs, retracts their heads off the disk surface. At least this should be the case for 2.5" HDs, conceived for portable equipment (laptops). Not sure for 3.5" HDs. With this disks stiction shouldn't be an issue.
Well, IIRC the heads still rest within the area of the platters, though no longer on a data-storing area. Stiction may still happen.

I remember reading that one common problem was with the drive spindle seizing.

LONG-TERM audio archiving strategy?

Reply #74 –
Hey guys, I don't mean to start any flamewar here, but in my experience people complain about HDDs and tape failing and data loss resulting from it. I have never heard a person claiming they lost a music CD or DVD due to it failing. Not among people I meet, not on forums.

It just seems like things suggested in this thread will create the problem, instead of solving it.

The reason I rip music is to have it all conveniently accessible without any need for disk juggling, not because I fear for its loss. In fact my reasoning is that if my HDD fails, I can always re-rip from source.
The Plan Within Plans