I have been using Monkey's audio for almost a year. I am obsessed with sound quality and use the highest settings in EAC with AccurateRip to rip all my CDs. Recently, I was furious to find out that the f****** Monkey's Audio does not provide any error correction/detection. This means that all my hard work could have been going down the drain. Could anyone please tell me what APE encoder/decoder can I use instead of that piece of shit Monkey's Audio? I read about WavPak, but am not sure how it works. I assume it encodes APE, but I have been unable to install Seek's frontend. Please recommend me something ASAP.
There is no 'quality' settings in lossless and error correction has nothing to do with this either. Any monkey encoder is as lossless as any wavpack or flac encoder.
Anything that encodes/decodes APE will no doubt be using the Monkey's Audio SDK, although it sounds like foobar uses adapted code, so it is possible it is a better decoder than MAC.EXE
WavPack creates WV file - completely different format.
You should do some reading in the wiki (http://wiki.hydrogenaudio.org/index.php?title=Lossless). Follow the link through to the comparison page and make your decision.
It is always possible that you could stick with APE and use alternative/additional error recovery, like PAR2.
APE writes an MD5 of the source audio data, it is upto the program you are encoding to, to decode the audio file and check it was written without error.
Excelsius... you're wrong in several things :
First, MAC does have some form of error detection, it was discussed recently here. It just seems that it is dependant on the decoder, and of course, detection doesn't mean correction.
Next, MAC is open source (since only around 2~3 years IIRC), but afaik, there are no other variants that encode to .APE, so it's the only software to create them (not counting the commandline and library interfaces).
Wavpack is a good lossless codec. It is not as fast as MAC but it offers several other features that make it good, as well as having an active development.
About frontends, i can't suggest any, since i don't use them.
@shadowking : that answer of yours was a bit precipitated, wasn't it?
As far as I know, there are currently no lossless format providing error correction. When an error occurs, decoders can't correct it. But some of them could skip the error; such skipping usually imply the loss of a small musical part.
The amount of loss is bigger with Monkey's Audio (compared to flac or wavpack) and it increases with the strength of the compression (you can loose several seconds with -c5000/insane profile). In some cases, the decoder can't even resync the stream; in other words you can loose the whole part of the file located after the error.
[JAZ]> you can't say that WavPack is not as fast than Monkey's. It doesn't mean anything if you don't explicit the encoding profiles. By defaut WavPack is clearly faster than Monkey's on both encoding and decoding side - but the ratio isn't as good then.
Data integrity on hardware or file system level is not the responsibility of a lossless audio codec IMO.
Data integrity on hardware or file system level is not the responsibility of a lossless audio codec IMO.
File organization isn't the responsability of the encoder either, but all of them are providing tagging support. It's called a
feature and features are playing an important part in the choice of lossless format
Data integrity on hardware or file system level is not the responsibility of a lossless audio codec IMO.
Ahh. That is what I am trying to say. Nothing will save you from crummy hardware. Shouldn't blame the codec as the data is no longer lossless. No codec or any software can correct catastrophic hardware failure - they can report faults, maybe try to correct it but software is only as good as the hardware.
[JAZ]> you can't say that WavPack is not as fast than Monkey's. It doesn't mean anything if you don't explicit the encoding profiles. By defaut WavPack is clearly faster than Monkey's on both encoding and decoding side - but the ratio isn't as good then.
You're right. I was talking for a similar compression ratio.
First of all I want to state that "Excelsius XS" should choose his words with more care.
Monkey's Audio sure is everything but crap! If you don't like what it does, write your own lossless compressor that can reconstruct files from your scratched and stained CDs.
What I really hate are people who cannot even write a "hello world" sample in any programming language but have the audacity to talk of excellent software as a piece of shit. Shame on you!
Wow, I did not know that higher compression equaled more errors, Guru. I have been using MAC's "High" compression, but from now on I will only use the "Fast" mode. So are you saying that flac has better error prevention than APE? If so, I can switch to flac because I have been using it on and off. The problem is that APE has a much better tagging system that works great. I tried the tagging system with flac about a year ago and it sucked.
For all you pro's out there, please give me some information about preserving my files. I have amassed a large classical library of about 8000 files and growing (about 100GB). I am going to keep this library forever and want to know what can I do to preserve the integrity of the files. As I said, I use EAC to rip to .wav and then use MAC to convert the .wav to .ape. The thought that I could copy a perfect file from EAC and then lose the integrity just because I encoded it by MAC is horrible. If there are no "lossless" programs that have complete error correction, then maybe we should stop using that damn word "lossless." In my vocabulary, it can either be completely lossless or lossy. How can you dare calling something lossless even if you are losing a fraction of the data from that given file? Amazing.
And what would you recommend about the hardware? I read that some people suggest making a back up on a dvd, but I don't agree. While the disks of hard drive are almost hermetically sealed in a metal case, a dvd is in the open and a single scratch on the label side will cause permanent data loss. Even if the hard drive dies, you can pay someone to take out your data. It might be expensive (up to $1000), but at least you know that you could get your data back, especially if the data is worth over 100K.
the integrity just because I encoded it by MAC is horrible. If there are no "lossless" programs that have complete error correction, then maybe we should stop using that damn word "lossless." In my vocabulary, it can either be completely lossless or lossy. How can you dare calling something lossless even if you are losing a fraction of the data from that given file? Amazing.
Following your definition even copying of a text file from for instance a harddisk to an usb stick is lossy, because someone could break the latter into pieces...
The thought that I could copy a perfect file from EAC and then lose the integrity just because I encoded it by MAC is horrible.
Amongst so many others (), how on earth did you arrive at this conclusion?
First of all I want to state that "Excelsius XS" should choose his words with more care.
Monkey's Audio sure is everything but crap! If you don't like what it does, write your own lossless compressor that can reconstruct files from your scratched and stained CDs.
What I really hate are people who cannot even write a "hello world" sample in any programming language but have the audacity to talk of excellent software as a piece of shit. Shame on you!
I don't have to choose my words with care and it is not up to you to say what people can and cannot say in this forum. Apparently you're the only one "offended" by something I said that is not even offensive. Maybe I can't write a program, but when a program is called "lossless" I expect no data loss. You probably wouldn't care anyway, but I have spent almost four years painstakingly building a large library and now I feel betrayed and some of my work is trashed because someone decided to call their program "lossless" without making all the specifications clear.
And by the way, if I thought the same way you did, then I would have to say that over 90% of the worlds population, including you, are idiots because they have no knowledge of basic physics, but I don't think that way because I know that everyone is different and has a right to their own opinion, whether right or wrong.
"lossless" means that de decoded audio is bit-identical to the original wave file, not more and not less. If the carrier of encoded data gets some (some!!) corrupt sectors that have not been automatically moved to the hotspare area of the disk (if it was stored on a hard disk), maybe some formats will be able to correct more loss than APE. But my experience is: one corrupt sector does never come alone!
Keeping your archive forever means:
# read all data carriers every few years, check for errors, copy those with errors to new ones.
# move your archive from time to time to an up to date storage medium if you don't want to fumble around with obsolete hardware
"lossless" never ever meant the ability to correct errors introduced either by hardware failures nor by media failures. "Complete error correction" belongs to the realm of fairy tales, not to reality because it is simply impossible!
Even if complete error correction is impossible, at least it seems that flac is better than APE in terms of errors. I really like APE, but given that it has more potential for errors than flac I have no choice but to switch. Maybe we need new software that uses ape or flac encoders but also incorporates an elaborate error correction, much like EAC.
Even if complete error correction is impossible, at least it seems that flac is better than APE in terms of errors. I really like APE, but given that it has more potential for errors than flac I have no choice but to switch. Maybe we need new software that uses ape or flac encoders but also incorporates an elaborate error correction, much like EAC.
If your files are being corrupted then you have a serious problem in your hardware. Check the RAM and always test the files after compressing.
I really like APE, but given that it has more potential for errors than flac I have no choice but to switch.
Again, how is it that you arrived at this erroneous conclusion?
My hardware is excellent. I have a custom brand new computer. I am not saying that my files are getting corrupted, I'm say that I MIGTH be getting little errors every time I compress and decompress my ape files since MAC does not detect errors.
Regarding the errors in APE I read about it in wikipedia where is compares all the lossless software and basically says that MAC has no error detection. Guruboolez also confirmed that you get less errors with flac and wavpak just a few posts above.
What EAC does is completely different to what occurs on corrupt data media.
Audio data is stored with a very weak error correction. in case of more serious damage the CD reader has to guess some samples!
The error correction implemented in filesystems can correct some errors - or it discards further reading with an error message. Normally the erroneous data won't even get to your lossless decoder of choice, unless you use something like "unstoppable copier" to read a CD. This will fill up unreadable sectors with zeros.
What you (not "we", at least not "I") need is some level of RAID system that distributes all data plus correction information over several disks so that 1 disk may get lost.
Again, error detection is done by the filesystem.
Guruboolez also confirmed that you get less errors with flac and wavpak just a few posts above.
No, he didn't.
I hate to break it to you, but you're terribly mistaken about most everything you've said so far.
What guruboolez is trying to tell you is that *IF* your ape files happen to get corrupted, you'll not be able to get as much data back from them as you would *IF* your flac files happen to encounter the same corruption.
This
does not mean that Monkey's Audio introduces errors while encoding.
Error correction in EAC and error correction as it relates to data are two totally different things. It is pointless to even compare them. BTW, despite what is shown in the EAC GUI, EAC performs no error correction.
Regarding the errors in APE I read about it in wikipedia where is compares all the lossless software and basically says that MAC has no error detection.
I see no such mention about lack of error detection in wikipedia. It does talk about error robustness which is not the same thing (and what it says isn't exactly correct, BTW); otherwise please provide a link to support your claim.
I also suggest you acquaint yourself with TOS #2. You've violated this twice already.
.... I'm say that I MIGTH be getting little errors every time I compress and decompress my ape files since MAC does not detect errors....
From Monkey's Audio FAQ:
Could you explain the error detection features in MAC?
The first thing MAC does when it compresses a frame (a small chunk of a file) is figure a 32-bit CRC for it. Then, it encodes that value at the front of the frame. When it decompresses a frame, it figures a CRC on the decompressed data and compares it to the original CRC.
This way, it compares the original data with the decompressed data. This is to say, if something is wrong in prediction (or anti-prediction) the original will not match the decompressed one.
Right now, CRC's are only used on the audio frames. In a future release, MAC will also CRC the WAV header and footer bytes also. (normally around 44 bytes a file)
Regardless, if a frame passes the CRC test, there is only about a one in billions chance of it being corrupt. If a file passes a verify, it means that all of the audio data is fully intact (every frame passed).
The only problem could ever be if there were disk I/O errors during encoding, so that what MAC thought it was supposed to encode wasn't what was really on the HD. HD's have error detection and correction, so the chances of this are about null. Also, once you enter disk I/O errors into the realm of possibilities, you can never really be safe.
I agree that 32 bit CRC is not the best error detection money can buy, but the chance that only one frame is randomly changed and produces the same CRC32 is negligible to me.
I don't have to choose my words with care and it is not up to you to say what people can and cannot say in this forum.
Yeah, that's up to the TOS:
http://www.hydrogenaudio.org/forums/index.php?showtopic=3974 (http://www.hydrogenaudio.org/forums/index.php?showtopic=3974)
So please go read it.
I read it. I haven't violated anything, including #2, which is a very subjective rule. If a member claims that my post does not conform to his/her aesthetic standards, I can claim the same thing about that member. If you don't mind, I'd like to get back to the discussion...
greynol, what you say about APE makes it seem better, but it's still a problem that you can't get as much out of a corrupted APE than from FLAC. The word "Corruption" seems to be an ambiguous. It seems that if you define it scrupulously enough, it would mean that files in a computer get "corrupted" all the time and APE file, or any file for that matter, will eventually become completely useless because every little "unnoticeable" problem will add up and as time goes to infinity the corruption function will approach infinity as well. Is there anyone in here who knows what "corruption" really means? Is there a mathematical function to describe it? Considering that computers are based on 0's and 1's (linear algebra) I would think that there COULD be a corruption formula. That's intriguing and could elucidate a lot of aspects of this ambiguous discussion.
I smell a troll rather than an idiot here.
Excelsius XS,
Check it out, I'll lead you to the water, but it's up to you if you want to take a drink.
If you don't want to use Monkey's Audio because it can't recover as much data as flac or Wav Pack and because you are paranoid, that's your business. But don't think you're going to get away with trashing a perfectly good piece of software just because you are ignorant.
Excelsius XS,
... Monkey's Audio ... can't recover as much data as flac or Wav Pack ... don't think you're going to get away with trashing a perfectly good piece of software just because you are ignorant.
You're right, I'm the ignorant one. MAC can't recover the data, but it's "PERFECTLY GOOD." That about says it. By the way, you just violated TOS#2.
...nor can flac or Wav Pack recover corrupted data. I guess by your reasoning they aren't perfectly good either?
I suggest you read over this thread a little bit harder.
Perhaps instead of saying that you are ignorant I should have said that you don't know what you're talking about; it is painfully obvious.
Again, there are no lossless format which (currently) offers any possibility to restore the corrupted datas.
I'm not an expert but I guess that such feature would necessary take some space and therefore decrease the compressing ratio.
If you're so anxious about corruption (which should only appear on rare situation) I suggest you to go with dedicated solution such as PAR2.
edit> greynol was faster.
Maybe we should make a thread or a poll to ask some of the people who are really good at programming to develop or perfect the lossless front. Are there any programmers in here? I am sure that many people would sacrifice compression to have a more error-free file quality. Since storage becomes cheaper and cheaper, maybe it would be best to just store the files in wav format. I know that currently the industry is trying hard to popularize SACD and DVD-audio, which will have a much better quality that current CDs. Just the fact that a vinyl has a much higher quality than any CD or DVD-audio says a lot about how far back we're in terms of audio technology.
How would you insure that your waves don't get corrupted?
I still don't think you're grasping the concept here.
EDIT: Now I *know* you still aren't grasping the concept here.
Just the fact that a vinyl has a much higher quality than any CD or DVD-audio says a lot about how far back we're in terms of audio technology.
Uh-oh! Now he's just violated TOS #8.
seanyseansean, I'm beginning to think you're right.
I smell a troll rather than an idiot here.
Well, i wasn't sure but...
maybe it would be best to just store the files in wav format. I know that currently the industry is trying hard to popularize SACD and DVD-audio, which will have a much better quality that current CDs. Just the fact that a vinyl has a much higher quality than any CD or DVD-audio says a lot about how far back we're in terms of audio technology.
"MAC can't handle errors well" has been a good topic for trolling and here is another one.
If he is a troll, he isn't very good or entertaining.
greynol, enough is enough. If you make one more post that is similar to what you've been saying I will see that the moderators put you back where you belong, since it seems they have been quick to respond to me.
I'll answer you this time: you cannot be 100% that any file remains pristine, but at least if you save the file in wav and lose a fraction, you won't lose as much as you would if you had an APE. APE is compressed and you would loss more data for the equivalent sized loss.
By quoting TOS#8 you just slandered me because you have no idea what i am talking about. It is perhaps beyond your comprehension, but vinyl indeed has a better quality. In simple terms, the reason is that digital audio approximates the waves in a quantized manner, whereas vinyl is engraved with the exact waveform of the music (it is not approximated).
Wrong, wrong, wrong. That waveform claim has been refuted so many times it's not even funny anymore.
Store your online files on a RAID-5 array, and your offline files on optical media enhanced with PAR2 files or dvddisaster correction, and then use whichever lossless codec you want. And stop worrying about the codecs.
-brendan
The easiest way to avoid errors is to stop using files.
From your posts it seems that there is no simple solution. Oh well. I'll try to come back and address this issue in a forum next year. By that time perhaps we'll all be using SACDs or DVD-audios and audio file storage will be perfected. I'll see if using PAR2 will not be time consuming because this is the first time I am hearing about it.
And Firon, I'm not going to repeat this again. I don't care what you have heard or what you have thought you have heard (or read what some kid has posted). CDs throw out a lot of the data that vinyls have. I have seen the data and the graphs myself. I have a friend who has been making home stereo and amplifiers for 37 years and he knows that too. There was also an editorial in New York Times about this. I don't what your opinion is, but do not misinform the people who come to get some information from this forum.
Many, many audio engineers (such as the creator of MP3 and AAC) who know far more than most of us have proven that your previous claim is essentially junk. If you don't want to believe that, suit yourself.
YOU are the one misinforming people here.
Oh, and if you want to test accuracy on a vinyl, try recording a sine wave on it and look at the output with an oscilloscope. Then do the same with a CD.
Let's not forget the harmonic distortions and the fact that there is an analog noise floor (which is not very low) with limits the "resolution" on vinyl.
Back to keeping your files safe, creation of PAR2 files is slow, but it is very reliable and very useful. With good quality media and taking care of your discs (ie keep them in dark places and in cases), the PAR2s should be more than sufficient for making sure your data is fine in the long term (sans fire or destruction)
I am sure that many people would sacrifice compression to have a more error-free file quality.
You are still seriously missing the point. Lossless codecs
do compress WAVE files and create
no errors. However, post-encoding errors
can occur (Edit: I refer to disc scratchs and bad sectors on hard drives here - corruption that can happen to
any file type and are unrelated to the encoding process). Some formats, such as WavPack and FLAC,
may be better at decoding through those errors (people's views seem to vary).
No codec can recover those errors.
As I, and someone else, have already suggested, I would look at PAR2 parity data if you want some help recovering corrupt data completely.
You need to start listening, and doing some reading. This thread is close to being closed for trolling.
You need to start listening, and doing some reading. This thread is close to being closed for trolling.
I know I've pretty much given up on it, but not before supplying a link:
http://www.quickpar.org.uk/ (http://www.quickpar.org.uk/)
Synthetic, when I said that I indeed did not understand at what point the errors were introduced. I already know what you said. And storing my files on 35 DVDs is not my idea of a safe back up. I have been trying hard to keep my CDs scratch free, but I haven't been able to do that. If I accidentally drop the DVDs or if there is an earthquake,the label side will get damaged and I'll have a pile of plastic garbage. Raid-5 is a temporary and expensive solution for now that I might try.
I don't know what do you mean by trolling, but this thread is over anyway. Thanks to everyone who gave me some useful information or leads.
Firon, if you don't mind please provide the links to the proof that you have found. I am open minded as long as there is scientific evidence to back it up. MP3 was invented by about 5 guys at the Fraunhofer Institute in Germany. If you have a link that shows that the institute says that CDs have better quality than vinyls I will be surprised.
Lets recap:
Monkeys has 2 levels of error detection, a CRC for the smaller 'frames' and an MD5 which can be compared against the whole decoded audio data (very few programs will do this, probabbly because the crc32 will catch 99.9999999% of errors).
When compared to FLAC and wv Monkeys will sometimes (depending on compression mode) recover less data than the others, if the file has become corrupted.
No current audio codec, lossless or wav will recover and stay lossless if there are errors in the datastream.
Invest in a firesafe, removable hard disk and DVD-Rs (ie two types of backup) and you will be safe from earthquakes, fire and user missuse. Better still upload your audio to one of hte online datastorage backup places.
And Firon, I'm not going to repeat this again. I don't care what you have heard or what you have thought you have heard (or read what some kid has posted).
For someone without serious knowledge you sound very arrogant.
CDs throw out a lot of the data that vinyls have. I have seen the data and the graphs myself. I have a friend who has been making home stereo and amplifiers for 37 years and he knows that too. There was also an editorial in New York Times about this. I don't what your opinion is, but do not misinform the people who come to get some information from this forum.
what data did you see there? was ist the difference between CD and vinyl?
Of course they are not equal because vinyl players introduce some wow&flutter, harmonic distortions, non harmonic distortions, groove noise, tracking angle errors, crackles, pops and so on.
CD throws them all away, that's right
Argh, I can't believe I'm even posting in this rediculous thread of misinformations!!!
As said before, it's not up to a *file format* to provide *error correction*. Although most filesystems DO have error correction/detection, and it would be very rare that files gets corrupted without an internal CRC error shows up.
Have you never heard of backup? There are many ways you can securely backup the data. Say, CD/DVD's doesn't last forever, but they last long enough if you keep them safe! But keeping them on such a media, I would definately use some kind of error correction data - I've seen projects before, that stores x% recovery data on media, or you could just duplicate every backed up DVD.
A combination of both an offline harddrive and backed up DVD's could be the best solution though.
AND FOR CHRIST SAKE STOP YELLING ABOUT INTEGRITY ON SPECIFIC FORMATS! IT'S POINT/USELESS IN THIS CASE!!!
...and vinyl is crap
http://www.skepticforum.com/viewtopic.php?t=85 (http://www.skepticforum.com/viewtopic.php?t=85)
Reading into that thread, there's references to books on basic sampling theory and so on that will explain everything (if you don't believe what Mr. Johnston says). And he is one of the co-inventors of MP3 and MPEG-2 AAC.
There's more I could probably dig up, but I'm not really going to bother at 6am.
http://www.skepticforum.com/viewtopic.php?t=85 (http://www.skepticforum.com/viewtopic.php?t=85)
Reading into that thread, there's references to books on basic sampling theory and so on that will explain everything (if you don't believe what Mr. Johnston says). And he is one of the co-inventors of MP3 and MPEG-2 AAC.
There's more I could probably dig up, but I'm not really going to bother at 6am.
I'm afraid no amount of links can help in this case...
By quoting TOS#8 you just slandered me because you have no idea what i am talking about. It is perhaps beyond your comprehension, but vinyl indeed has a better quality. In simple terms, the reason is that digital audio approximates the waves in a quantized manner, whereas vinyl is engraved with the exact waveform of the music (it is not approximated).
Sorry but you don't have the faintest idea of what you are talking about. Your waveformat claim is a very common misconception of people who are not familiar with digital sampling theory and the nyquist-shannon sampling theorem (http://en.wikipedia.org/wiki/Nyquist-Shannon_sampling_theorem).
You can reconstruct the original wave if sampled at least twice the highest frequency. The typical hearing range for humans goes from 20Hz to 20kHz. CDs are sampled at 44kHz which makes possible to reconstruct waves up to 22kHz above the hearing threshold so you do not lose anything from the waveformat.
Please don't argue against this because otherwise everyone will be sure you are a troll.
As said before, it's not up to a *file format* to provide *error correction*. Although most filesystems DO have error correction/detection, and it would be very rare that files gets corrupted without an internal CRC error shows up.
Are you really sure?
Double checking with wikipedia, GPFS and ZFS only provide Error Correction.
Never seen someone so paranoid about data integrity that he would question the format of the data, and I thought A removeable hard drive in a cupboard would be enough for the average joe.
As said before, it's not up to a *file format* to provide *error correction*. Although most filesystems DO have error correction/detection, and it would be very rare that files gets corrupted without an internal CRC error shows up.
Are you really sure?
Double checking with wikipedia, GPFS and ZFS only provide Error Correction.
Never seen someone so paranoid about data integrity that he would question the format of the data, and I thought A removeable hard drive in a cupboard would be enough for the average joe.
Few filesystems actually provide error correction, because we are again looking at the wrong place. Error correction should be media dependent to be able to correct typical errors for that media, so in reality (on all drives you could remotely call modern) error correction for hard-disks is handled by the disk/controller internally, independent of the file system.
If a sector is read with wrong checksum, the drive uses ECC data to try to correct it and the sector is "reallocated" so that all further requests to it are redirected to a spare "healthy" sector. All this generally happens without any intervention or knowlege from the operating system, but you can use a SMART monitor to find out how many sectors your drive has already reallocated.
Unfortunately there is not much you can do to get better protection on that layer, so adding an ECC feature to filesystems does make sense.
You need to start listening, and doing some reading. This thread is close to being closed for trolling.
I know I've pretty much given up on it, but not before supplying a link:
http://www.quickpar.org.uk/ (http://www.quickpar.org.uk/)
Quickpar/par2 are good recommendations.
However, as I mentioned earlier dvddisaser is (IMHO) the application the thread-opener needs. The tool specifically address bitrot of data files on optical media. And it's engineered specifically to help you toward that goal. And it is Free Software.
Read the website, it has very useful information and gives multiple proven techniques for reducing your long term data-loss risks:
http://dvdisaster.berlios.de/en/ (http://dvdisaster.berlios.de/en/)
-brendan
If I accidentally drop the DVDs or if there is an earthquake,the label side will get damaged and I'll have a pile of plastic garbage. Raid-5 is a temporary and expensive solution for now that I might try.
FYI, only CDs have the severe label side vulnerability. The metal layer for DVDs is actually in the middle of the wafer, equidistant from both the read side and the label side. If you think about it, here's a good proof: explain why dual sided DVDs exist, but dual sided CDs do not.
-brendan
Not to mention that CDs and DVDs have multiple layers of built-in error correction. A scratch through the data layer isn't necessarily going to result in data loss.
I thought of bringing this up earlier, but considering how many misconceptions the OP had, I really didn't know where to begin.
EDIT: spelling/grammar
EDIT2: But you're right brendan; you have to go pretty deep to scratch the data layer on a DVD.
EDIT3: errr, forget it, I've already been put in my place.
Yes, most magnetic and optical media store, at the bit-level, both error detection and error correction data. The drives themselves can detect and correct the raw data read off of the disc/disk. Most contemporary magenetic media has the ability to "refresh" failing data and can even rewrite and/or reallocate the corrected data back onto the media. Optical drives generally cannot do this, so their bitrot tends to accumulate faster.
You can add an additional layer of error detection/correction by using an array that includes redundancy.
In addition, the PATA/SATA/SCSI/USB/Firewire/etc. connections generally include error detection on the communications and will retransmit several times if you have a timing/power/interference/cabling issue.
Once the data gets onto the main board, you are at the mercy of bad RAM, bad software, good software with esoteric bugs, cosmic rays, gremlins, the NSA and fat fingers. No file format can protect you against that.
There is only one thing that can: backups.
-brendan
...good software with esoteric bugs
...and to go full-circle, does MAC 3.99 have (or does anyone have any reason to think that it has) any esoteric bugs?
Sure it has them. If you encode files at full moon you must not forget to dance naked around an oak tree, else you will hear voices in your music
DVD-Rs would be a better choice than CD-Rs, since they have better error correction capabilities (though I don't know how good is it in the long term vs a CD), provided you don't scratch it up or destroy the disk.
That, and there's more space to waste on redundancy.
Speaking of DVDisaster, I need to try it out, it's a very cool tool.
As said before, it's not up to a *file format* to provide *error correction*. Although most filesystems DO have error correction/detection, and it would be very rare that files gets corrupted without an internal CRC error shows up.
Are you really sure?
Double checking with wikipedia, GPFS and ZFS only provide Error Correction.
Never seen someone so paranoid about data integrity that he would question the format of the data, and I thought A removeable hard drive in a cupboard would be enough for the average joe.
I'm not sure that NTFS/FAT/ISO9660 has correction, but it has indeed a very strong CRC check! It choke when I tried to restore many of my old data CD-R's. According to thread opener, Windows (etc) would just disregard any kind of CRC check and copy the files corrupted.
Well, here are my 2 cents:
I also use MAC for archiving my rips. It gives slightly better compression than WavPack and FLAC, the tagging is fine, and it is fast ENOUGH.
I had one cd image which I compressed with mac 3.97 that did not decode with mac 3.99, out of around 20, but it decoded fine on MAC 3.97, I reencoded, and was satisfied. The MAC gui has a verify, which check the files after packing. In case of doubt, I would suggest to check the files before archiving.
The error DETECTION for MAC is more AS robust AS FLAC, since, as posted above, it has an md5 of the audio data and a CRC per frame, but I am not sure about WavePack (CORRECTED ACOORDING TO INFORMATION BELOW BY jcoalson). I am not sure which method the MAC front-end uses, but I gess it uses both. So as far as error DETECTION goes, I suggest to stick to MAC. Besides, there are ways to decode through errors. You get a glitch on the audio, but most of it is fine. For fast checks, I use QuickSFV (http://www.QuickSFV.org) to generate MD5 sums of the files, and check after I burn.
I use par2, quickpar (links above) and and ICE ECC (http://www.ice-graphics.com/ICEECC/IndexE.html) to generate error correction files. ICE ECC is faster than PAR2, and you can generate correction for full directories, but if the filenames aren't the same when it checks, it doesn't recognize the files. It is a problem specially with long filenames, which nero truncates when burning. Also, if you change the file name or directory structure before burning, or miss one file, it will complain when regenarating the file. However, PAR2 is easyer to use at command line, which is fine to to program that uses so much time. Make one script (or bat file), let it run, and go to sleep. I have not been able to use ICE ECC from command line, but I didn't try very hard. And PAR2 files will be recognized even if the name is changed. Besides, ICE ECC has a very handy option, to fit the amount of recovery blocks to a certain size, say DVD or CD. I didn't know about dvdisaster, I discovered on this (lengthy) thread, but seems worth a look. I still prefer to keep recovery data on the DVD, thought, than make it after I burn.
I burn my rips to a DVD, for a lot of reasons. But when looking for my 100 and counting back-up DVD's, I am considering buying an external HD case, and a big drive. In the long run, will be cheaper. I store a copy of ICE ECC executable (it works without installation) and the MAC command line encoder on the DVD, in case they dissapear.
I think RAID's are cool but overkill for what you want, and I particularly don't trust them very much. The biggest problem is if you'll be able to read the media, be it DVD, CD, or even SATA drives for very long. I once had to find a 5-1/4" drive to read a floppy, and it was quite hard. I believe that with DVD's or HD's you won't have to change the medium so often in your lifetime, but one day you'll have to move the data to a new medium.
This post is long, the thread is even longer. I suggest to make a new thread about which is the best medium and recovery strategy for archiving.
The error DETECTION for MAC is more robust than in FLAC, since as posted above it has an md5 of the audio data and a CRC per frame.
please do not add yet more misinformation to this thread. FLAC was the first format to have MD5 checking (it had it from the beginning), and it has frame CRCs (from the beginning). later MAC added md5 checksums ala FLAC.
Josh
I have no reason to doubt tht MAC stores an md5 checksum of the decoded data, but I have never worked with a program that actually uses this (EDIT: unless MAC uses this upon decoding or verification).
The MAC 4.01 beta GUI does have the "Quick Verify" option. But based on how quickly it works, there is absolutely no way that the md5 checksum it uses is for the decoded data. Rather, I am certain that it is a checksum on the encoded data.
I do find this useful since it is a very fast way to verify whether an ape file was corrupted. I've gotten into the habbit of verifying what I burn to DVD and MAC's quick verify has made this very easy.
EDIT:
I also use MAC for archiving my rips. It gives slightly better compression than WavPack and FLAC, the tagging is fine, and it is faster then both.
I hope you're not trying to say that MAC can encode or decode faster than WavPack or flac, cause that most certainly isn't true!
The error DETECTION for MAC is more robust than in FLAC, since as posted above it has an md5 of the audio data and a CRC per frame.
please do not add yet more misinformation to this thread. FLAC was the first format to have MD5 checking (it had it from the beginning), and it has frame CRCs (from the beginning). later MAC added md5 checksums ala FLAC.
Josh
Sorry, I was just following from another post on this thread. And I did not know that FLAC had CRC's stored for each frame, it wasn't mentioned before on this thread either.
Now, a question: when you test a flac file (from command line, flac -t <flacfile>) is the CRC of each frame verified, or just the md5 for the audio stream?
Most likely both are tested at the same time.
Now, a question: when you test a flac file (from command line, flac -t <flacfile>) is the CRC of each frame verified, or just the md5 for the audio stream?
both, otherwise why have them? test mode is the same as decode mode except no output file is written.
I'm not sure that NTFS/FAT/ISO9660 has correction, but it has indeed a very strong CRC check! It choke when I tried to restore many of my old data CD-R's. According to thread opener, Windows (etc) would just disregard any kind of CRC check and copy the files corrupted.
"Normal" Iso-CDs have a third (or second, depending how you count) layer of error protection/detection: Every sector stores 2KiB of user data, plus a 4-byte checksum and 276 Bytes of ECC data. This is an addition to the 8 Bytes ECC data per frame (one frame contains 24 bytes of user data) that Audio-CDs already have for C1/C2 error detection/correction. So a read error from a CD is quite unlikely to be missed by a good drive, and most small errors (especially scratches from inside to outside, small holes) wil be correctable. Circular scratches that destroy long parts of the track in a row are much more problematic, so the tip printed on the box is important: Do not use circular motions when cleaning a cd.
Some of you listed some possible solutions. None of them are that simple and are pretty time consuming. I'll try to see if I can implement them. Using raid 5 or something similar seems to be the simplest solution. Anyway, thanks for the info.
I hope that CDs will be obsolete very soon and we can finally start using higher quality DVD-audio or SACDs. That will mean I will have to rebuild my entire library yet again. Right now I am in the process of rebuilding it because when I started about four years ago there was no one to tell me to use lossless format. As a result I now have several thousand crappy mp3s (at 320). Sometimes you have to hate technology!
As a result I now have several thousand crappy mp3s (at 320). Sometimes you have to hate technology!
OMG here we go... again...
I hope that CDs will be obsolete very soon and we can finally start using higher quality DVD-audio or SACDs.
(...)
Sometimes you have to hate technology!
You seem to know what you're talking about.
Right now I am in the process of rebuilding it because when I started about four years ago there was no one to tell me to use lossless format.
How can you be sure that your lossless files will be "really lossless" and not just "Monkey lossless"? SCNR
Btw, didn't you say that "this thread is over anyway" in post #42? So, stop the trolling now.
Some of you listed some possible solutions. None of them are that simple and are pretty time consuming. I'll try to see if I can implement them. Using raid 5 or something similar seems to be the simplest solution. Anyway, thanks for the info.
I hope that CDs will be obsolete very soon and we can finally start using higher quality DVD-audio or SACDs. That will mean I will have to rebuild my entire library yet again. Right now I am in the process of rebuilding it because when I started about four years ago there was no one to tell me to use lossless format. As a result I now have several thousand crappy mp3s (at 320). Sometimes you have to hate technology!
Dude, we join forums to share verifiable knowledge and to learn from each other. Please adopt an attitude of learning, or else this forum can tell you that your presence is not needed. As it is, the only thing we are learning from you now is that trolls come in all shapes and sizes.
Btw, didn't you say that "this thread is over anyway" in post #42? So, stop the trolling now.
C'mon, it's not fair to tell him to stop just because he started to reuse different stuff to refuel the dying embers of this thread!
Anyway, thanks for the info.
Please adopt an attitude of learning, or else ...
Well .. he said thank you.
Using raid 5 or something similar seems to be the simplest solution.
Then again it's the most expensive and noisy solution (but that can be OK for some)
I hope that CDs will be obsolete very soon and we can finally start using higher quality DVD-audio or SACDs.
That's the trolling again ... I'll bite just once.
I hate to burst your bubble but the Hi-Rez audio formats are dying, there might stay an niche market for them for a while. Download Music shops are more likely to replace CD with mostly lossy (often DRMed) "crap" as you would call it.
Download Music shops are more likely to replace CD with mostly lossy (often DRMed) "crap" as you would call it.
It's sad that only a few shops sells lossless files. Then again you can question why 24bit/96khz files are not available, since we don't have the CD medium. Another thought; could lossy 96khz files encoded from a 24bit source sound better than the CD medium we are used to? Personally i've not been able to do such tests myself yet
What's your problem people? You'll turn anything I say into personal matters. Only a low life can constantly attack someone, especially in a pack, just to feel better. If "trolls" come in all shapes and sizes, this thread has indeed showed it, which ironically included one of your moderators. Grow up. Thanks for the warm welcome, but no thanks. Good bye.
And off to Head-Fi he is...
Using raid 5 or something similar seems to be the simplest solution.
Then again it's the most expensive and noisy solution (but that can be OK for some)
Ah, not so much any more. The Infrant ReadyNAS NV+ (four drive SATA RAID-5) is ~$600 for the empty enclosure and with the NV+'s new fanless PSU, it's pretty darn quiet (assuming you purchase relatively quiet drives).
-brendan
And off to Head-Fi he is...
Do you think the "Excelsius" over there is "our" Excelsius XS"?
If yes, then seanyseansean was not right in post #26
(the other one destroyed his tweeters by listening too loud - but of course his speakers are cap.)
Sometimes the only thing I want to do is cry.
What's your problem people? You'll turn anything I say into personal matters.
No, you are. We are merely trying to enlighten you but you insist in not listening.
You should know that when you repeatedly post unverified information you are breaking the forum rules and most likely that will be disputed by other members.
Thanks for the warm welcome, but no thanks. Good bye.
WE'LL MISS YOU To be honest, I'm surprised this thread lasted so long.