Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Transferring Flac Collection over 802.11G (Read 9575 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Transferring Flac Collection over 802.11G

Ive got a win2K3 server setup and am going to transfer around 200+GB of Flac & Ogg files to it from my HTPC in the living room. The HTPC is currently connected via 802.11g (125). I get really good transfer speeds but I am worried about corruption. Question is should I worry about corruption transferring this way? I transfer 1-2GB movies all the time. But this is my sacred Flac collection and the copies being trasnferred are backups.


I tried using and external hard drive but for some reason when I plug it in to the Win2K3 server the server freezes up.

* I am going to install a gigabit card on the HTPC in a week or two, should I just wait for that?

Transferring Flac Collection over 802.11G

Reply #1 –
Quote
I get really good transfer speeds but I am worried about corruption. Question is should I worry about corruption transferring this way? I transfer 1-2GB movies all the time. But this is my sacred Flac collection and the copies being trasnferred are backups.
[{POST_SNAPBACK}][/a]

Use something like the GUI Windows FlacTester application ([a href="http://www.vuplayer.com/other.htm]http://www.vuplayer.com/other.htm[/url]) to verify your FLAC files once they are transferred for MD5 checksum errors. That should give you more peace of mind.

Transferring Flac Collection over 802.11G

Reply #2 –
I wouldn't worry too much about corruption at all – no more so than for a more traditional wired network. In fact I wouldn’t be much more worried about transferring your data via 802.11g than I would be transferring them between 2 hard drives on the same machine (actually if data corruption did occur transferring between 2 hard drives would be a worse case than transferring over any modern network).

Network standards are designed to be error tolerant (that is have the ability to detect and correct errors) wireless networks doubly so - they have to be, given the data transferred could be anything (e.g. medical data) it has to reach the destination entirely intact, bit identical to source. Hence there are several layers of protection in place between your data and the actual transfer. Worst case there is an error that is detected but can’t be corrected, so the receiver will inform the sender and the sender will resend the data.

But if you want that extra piece of mind (understandable  ) you could test the resultant files as suggested by guest0101 and/or even go as far as to wrap the flac files up in a container supporting checksums (e.g. rar), which would tell you if the file had received errors in transit. You also may want to check up on the wireless network hardware involved – depending on your hardware you may be able to select a more secure method of transfer rather than a faster method of transfer (again, not that it’s likely to make much difference – especially not if signal strength is good).

Transferring Flac Collection over 802.11G

Reply #3 –
Yeah, wireless has some good ways of looking after data integrity, it's at least as good as wired. That doesn't mean errors cannot slip through, there is a specific error rate, but i'm not sure what it is, a quick google didn't help much. It's likely that you need to push many terabytes before you see an error coming through undetected though. As Guest0101 said, check your files before you delete the originals.
Veni Vidi Vorbis.

Transferring Flac Collection over 802.11G

Reply #4 –
I have had a corrupted file once whilst copying/moving from 1 HD to another HD via wireless lan, iirc the connection broke down by some reason.
So, I recommend to copy instead of moving.
If during moving the file gets corrupted, it is lost with high probability.
If you copy the file, you have a backup.
After successful copying, you can erase the original, if u wanna.

Transferring Flac Collection over 802.11G

Reply #5 –
Quote
I have had a corrupted file once whilst copying/moving from 1 HD to another HD via wireless lan, iirc the connection broke down by some reason.
So, I recommend to copy instead of moving.
If during moving the file gets corrupted, it is lost with high probability.
If you copy the file, you have a backup.
After successful copying, you can erase the original, if u wanna.
[a href="index.php?act=findpost&pid=304554"][{POST_SNAPBACK}][/a]


The files are copying if you transfer over a wireless network as well.

There's no risk involved at all.

Transferring Flac Collection over 802.11G

Reply #6 –
Quote
I am worried about corruption. Question is should I worry about corruption transferring this way? [a href="index.php?act=findpost&pid=304462"][{POST_SNAPBACK}][/a]


Try using robocopy.exe (stands for robust copy -- part of the winserver 2003 rtk, I think...go to the microsoft site and search for "robocopy.exe.")  It's what I use for slinging flac files around between machines.

Robocopy is a command line tool that has better verification and error recovery options than xcopy (or the explorer shell, for that matter.)  It also works well over slow or intermittent connections.

Transferring Flac Collection over 802.11G

Reply #7 –
Thanks for much for all the replies! I have to say I have been pretty impressed with the transfer speeds over 802.11g so that is good news.

I am going to check out RoboCopy for sure as I transfer large files all the time.
Regarding "wrap the flac files up in a container supporting checksums". How would I go about doing this? I know how to pack a .rar with flac files, would I just create a checksum and include it in the folder?

Thanks again!

Transferring Flac Collection over 802.11G

Reply #8 –
Quote
Thanks for much for all the replies! I have to say I have been pretty impressed with the transfer speeds over 802.11g so that is good news.

I am going to check out RoboCopy for sure as I transfer large files all the time.
Regarding "wrap the flac files up in a container supporting checksums". How would I go about doing this? I know how to pack a .rar with flac files, would I just create a checksum and include it in the folder?

Thanks again!
[a href="index.php?act=findpost&pid=304647"][{POST_SNAPBACK}][/a]


Rar has a data-integrity test itself. If you also include a recovery record you can repair the file if an error does occure.

Transferring Flac Collection over 802.11G

Reply #9 –
Its not something to worry about... everything gets checked on transmission automatically. Stuff like accidental data corruption when a file is being read or written to during normal day-to-day operations is much more common because there is no correction then.

Transferring Flac Collection over 802.11G

Reply #10 –
Quote
...and/or even go as far as to wrap the flac files up in a container supporting checksums (e.g. rar), which would tell you if the file had received errors in transit.
[a href="index.php?act=findpost&pid=304543"][{POST_SNAPBACK}][/a]

The FLAC format itself, stores a MD5 signature of the original unencoded data in the file header during encoding. Later, when the file is decoded or FLAC´s Test feature is used, the MD5 signature is compared with the file, to check for corruption... -Martin.
 

Transferring Flac Collection over 802.11G

Reply #11 –
Gotcha so no "real" need. Much thanks! Transferring as we speak!

Transferring Flac Collection over 802.11G

Reply #12 –
Quote
The files are copying if you transfer over a wireless network as well.

There's no risk involved at all.



Theory & Practice/experience...
As I wrote above, make sure, you copy files (independent if via wires or wireless), not move them !

Transferring Flac Collection over 802.11G

Reply #13 –
If you don't want to wrap the files Microsoft has a command line tool which can output MD5 or SHA1 hashes for files of any type.

Quote
Its not something to worry about... everything gets checked on transmission automatically. Stuff like accidental data corruption when a file is being read or written to during normal day-to-day operations is much more common because there is no correction then.
[a href="index.php?act=findpost&pid=304673"][{POST_SNAPBACK}][/a]
Ethernet uses CRC for error checking which isn't as robust as SHA1 or MD5 and errors can slip through. I've had files corrupted during transfer accross wired and wireless connections before.

Transferring Flac Collection over 802.11G

Reply #14 –
I took a look at some of my resources and it seems that the 802.11 specs would dictate an uncorrected bit error probability of 2.5e-22. The error rate of the physical channel is much higher (around 1e-6) but packet CRCs and other error correction mechanisms catch the vast majority of these errors.

Now, you have around 1.6e12 bits of data to transfer over the wire. We can assume that the probability of an uncorrected error in any bit is independent of errors in other bits (this is not quite true, but is good enough for now). So - the probability that at least one bit in the set will be incorrect is 1-(1-2.5e-22)^(1.6e12). This is a pretty small number.

However, running something along the lines of (I'm sure there are Windows equivalents of these commands, otherwise you could try Cygwin) "find -name "*" -exec md5sum '{};'" on the directories on both ends might be worth it. A diff tool can tell you if any of the lines differ.

Transferring Flac Collection over 802.11G

Reply #15 –
Quote
However, running something along the lines of (I'm sure there are Windows equivalents of these commands, otherwise you could try Cygwin) "find -name "*" -exec md5sum '{};'" on the directories on both ends might be worth it. A diff tool can tell you if any of the lines differ.


I am going to do some comparisions between the two sets of flac files.  The information I have gotten from this thread has eased my mind a bit because over the last few months I have done alot of transferring of files. So far I have not had any problems.

Transferring Flac Collection over 802.11G

Reply #16 –
Quote
I am going to do some comparisions between the two sets of flac files...
[a href="index.php?act=findpost&pid=304920"][{POST_SNAPBACK}][/a]

There is no need to compare the two sets of files... Just copy the files, and when done, you open FLAC frontend, and add all the directories with the FLAC´s that you have copied, and then you just click "Test". Now all the FLAC´s will be decoded and an MD5 checsum will be calculated for each decoded FLAC, and that MD5 checksum will be compared against an internal MD5 checksum of the original unencoded data stored inside the FLAC´s, so this test is all that´s needed to check for corruption... -Martin.

Transferring Flac Collection over 802.11G

Reply #17 –
Quote
Quote
I am going to do some comparisions between the two sets of flac files...
[a href="index.php?act=findpost&pid=304920"][{POST_SNAPBACK}][/a]

There is no need to compare the two sets of files... Just copy the files, and when done, you open FLAC frontend, and add all the directories with the FLAC´s that you have copied, and then you just click "Test". Now all the FLAC´s will be decoded and an MD5 checsum will be calculated for each decoded FLAC, and that MD5 checksum will be compared against an internal MD5 checksum of the original unencoded data stored inside the FLAC´s, so this test is all that´s needed to check for corruption... -Martin.
[a href="index.php?act=findpost&pid=304953"][{POST_SNAPBACK}][/a]



Thanks for the clarification.
It seems like with the Flac Front End I can only do one folder at a time. Am I missing something?

Transferring Flac Collection over 802.11G

Reply #18 –
Quote
Thanks for the clarification.
It seems like with the Flac Front End I can only do one folder at a time. Am I missing something?
[a href="index.php?act=findpost&pid=305091"][{POST_SNAPBACK}][/a]

You can drag and drop multiple folders into FLAC frontend... -Martin.

Transferring Flac Collection over 802.11G

Reply #19 –
I would suggest using a file comparison program like Beyond Compare for this purpose. After the transfer you can compare the MD5 checksums automatically. The two group of files do not have to be simultaneously accessible to the same computer. You can take a snapshot (including MD5 checksums) of a group of folders -> transfer the snapshot file -> compare the snapshot to another group of folders on another computer.
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.

Transferring Flac Collection over 802.11G

Reply #20 –
Thanks for the link, I will check it out!

What do you guys prefer when backing up your flac files?  A .bak file or duplicate of the raw files themselves?

What I want to do is set it up so that whenever I add new flac files to my HTPC it is detected and syncs it with the backups on the server. I was going to use either Windows built-in backup program but I was affraid of using a .bak file instead of the actual files. I was also looking at a program called SURESYNC.

http://www.spursuits.com/suresync/suresync.htm

What format do you guys store your backups in?

Transferring Flac Collection over 802.11G

Reply #21 –
The software I recommended earlier, Beyond Compare, does synchronization too. I think it's best to have an offline drive which you sync with the online drive periodically. Beyond Compare would do this very well. If you choose to have two online drives I think the failure probability (of one of them failing) is higher. The only advantage of that solution is clearly that system would be more up-to-date in case of a failure. But also remember that sometimes you might perform operations on your hard drive by mistake (or virus can infect, etc), which you don't want replicated on the backup drive. This would be problematic for always online backup solution.
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.

Transferring Flac Collection over 802.11G

Reply #22 –
Thanks for the idea Atici, I am going to install and run it tonight.

Transferring Flac Collection over 802.11G

Reply #23 –
I just wanted to say that Beyond Compare is a great program! I have writing scripts and adding filters today and ran my first few tests. There are alot of sitautions where this app be a good solution.

Transferring Flac Collection over 802.11G

Reply #24 –
Quote
I just wanted to say that Beyond Compare is a great program! I have writing scripts and adding filters today and ran my first few tests. There are alot of sitautions where this app be a good solution.
[{POST_SNAPBACK}][/a]

Yes it is indeed.  I got confused, why do you need scripts and filters for?  Let me know your experiences and how you solved the backup problem. Also by default Beyond Compare do not compare CRCs of the files, I'd suggest turning it on for higher confidence (if copy failed or not).

I know of [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=31590]this thread[/url] which you might also find interesting.
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.