Hi all,
I am looking for a Windows (Seven) backup/sync solution adapted to all my audio files. Let me explain.
My music library consists in over 40,000 music files (800 GB), and it's saved on an external HD at regular intervals.
The problem is that I regularly change the tags of many files at once using foobar.
With a classic backup solution, every time a tag has been changed, the entire audio file must be re-saved ... even though all the "audio" part has remained unchanged and it would have been enough to save the "tag" part of the file!
Thus, I'm looking for a solution capable of backing up the whole file when needed, or only the "tag" part when only tags habe been modified.
On Linux there are solutions like rsync for this kind of advanced features.
On Seven I do not know what can exist. So I hope you can help me find the right solution.
FYI, currently I use SyncBackPro, but it does not seem to work the way I want (or maybe there is a setting that I have not found??).
BTW, I'm not really a geek, so I'm looking for a really EASY TO USE solution, not some command line thing with lots of parameters.
The idea would be to have a windows GUI that would allow me to synchronize ("mirror" mode) a source folder (e.g. C:\AUDIO) with a destination folder (e.g. D:\AUDIO), and that will NOT copy 800 GB of data every time I change ONE single tag on all my audio files. I hope you get the idea.
Last but not least, the solution would have to handle file names and tags involving sometimes accented characters (é,è, ê, ï, etc.).
Thanks in advance.
What you are looking for exists, but is useless in your situation. You are backing up to local storage, so tools like rsync would actually be slower if used in the way you are thinking.
Tools like rsync are useful when you have two computers and a slow network between the source and destination. Doing differential copies on local storage is a waste of time and involves more disk access.
If you can use the command line then robocopy is simple and included with Windows 7. Open a command prompt and run the following:
robocopy /mir C:\audio D:\audio
It will make D:\audio a perfect copy of C:\audio (a mirror). It will only copy files that are different. Be aware that it will also delete files from the backup if they are missing from the source.
It will still copy every file if you change a tag in each one, but again that is faster then figuring out the differentials on local storage.
I use your very same setup as well as SyncBackPro. Sometimes I change "things" and I have to recopy 300GB at once.
When you change the tag the file changes and of course you have to recopy it, how does rsync work? How can it copy only the tag without copying the file if the tag is inside the file?
washu, I see your point, but my concern is not about the time it takes, but about the writing operation.
I want a backup solution that will write as less data as possible, even if it takes more time.
Like I said, I need a solution that will NOT write 800 GB of data every time I change ONE single tag on all my audio files. That is my choice.
Besides, I'm not really on local storage. All my audio files are on a NAS, and the NAS drives are mounted as network drives under Seven. My example with C: and D: was just an example.
So the real backup/sync would be as follows : X:\AUDIO (network drive) ----> F:\AUDIO (external hard drive).
SyncBackPro does not copy everything even if I change one file. It does copy only the changed files, of course. But that is not enough for me.
I don't need a solution that will only copy files that are different. I need a solution that will copy only tags when only tags are changed, and that will copy the whole file only when there's no other choice.
eahm, I suggest you find some doc about rsync (EDIT : here for instance : http://rsync.samba.org/tech_report/ (http://rsync.samba.org/tech_report/) ). I can't explain you precisely how it works, but I can promise you that it works exactly that way.
Why would you care about the writing? It's not going to hurt anything. In fact, it is safer as you eliminate a point of failure (the diffing program). Not that rsync has had any such bugs that I'm aware of, but keeping it simple is always better.
If you are worried about drive life then what you are proposing is worse, as the drives will have to do a full read and then at least a partial write. A full write is less work.
Having your data on a NAS doesn't change anything. It's still going to be way faster and less work to simply copy the files instead of diffing them, unless your network is horribly slow.
If you really must go down this route then you can get versions of rsync for Windows if you look around. I'm going to strongly advise that you do not go this route. Besides it being a worse solution, you do not appear to have the technical level required to use it properly (no insult intended). You said yourself "I'm not really a geek", and rsync on windows has some technical issues due to it being built for UNIX systems.
I periodically back up my music (and other data) to an external HD and have found this tool to be well suited for the purpose:
http://synkron.sourceforge.net/ (http://synkron.sourceforge.net/)
One word of caution, in setting up folder pairs (e.g. master/slave), make sure "Do not back up updated files" is checked, otherwise it does a redundant backup of files you've deleted/overwritten/etc. in a local synkron folder (unless of course you really want this behavior).
washu, you still don't get what he needs. The solution he needs is actually better than you think. Your opinion is too strong, some people just don't care about opinions.
Remedial Sound, I suggest you to test SyncBackPro. I use it for my customers as well, I ensure you it's the best Backup/Sync tool right after Symantec Backup Exec. Also check the prices.
Hi guys,
Remedial Sound, I have checked Synkron's doc, and I don't see it using any delta-transmission algorithm like rsync. Is there something you can tell me about it ?
@all : I am currently testing Liciel Rsync ( http://code.google.com/p/licielrsync/ (http://code.google.com/p/licielrsync/) ) which seems to be pretty much exactly what I've been looking for. I suggest you guys check it out.
I've done a first test with a 6.95 GB folder. First backup of all the files, then I changed ONE tag on all the test files (with foobar), and I did a new backup. Here are the results of the second backup :
sent 12.20M bytes received 14.46M bytes 23.52K bytes/sec
total size is 6.95G speedup is 260.51
... so, considering the small amount of sent/received data, I think it really works ! What do you think ?
so the disk-trashing/overall longevity-uptime question's could be:
Is it wiser to:
a. run a delta-aware sync to "dumb" share (that is: rsync not running in daemon mode)
b. run a "dumber" utility like pathsync on a "dumb" share http://www.cockos.com/pathsync/ (http://www.cockos.com/pathsync/)
c. replace the "dumb" share with some nix server and use rsync in daemon mode
?
p.s. if updates to backup procedure are expected, like more machines syncing with some share - then being aware of rsync limits, afaik it is not really bidirectional.
Also check DeltaCopy and Syncrify. http://web.synametrics.com/SyncrifyVsDeltaCopy.htm (http://web.synametrics.com/SyncrifyVsDeltaCopy.htm)
They seem to be more complete backup solutions, I think I am going to switch two/three companies to Syncrify and see how it runs.
Smok3, I think that for the intended purposes, there's no need for any bi-directional software. IMHO it's best to have all audio in ONE location, and simply backup it to another location for security purposes (or even for nomad use). By doing so, you can modify or even delete all the copied data, but the "reference audio" remains stored in one location and will be copied again during the next backup. So my method is : "if I want to modify anything in my music library, I need to do it within the "reference audio", because any modification in the "copied audio" will disappear with the next backup".
As for your previous question, honestly I have no idea. I hardly understand the question anyway, sorry.
eahm, Synkrify is a paid solution, so I guess I'll stick with Liciel Rsync...
wcs13, it's 100% free for personal use.
...but you will need to install the Server, the Client and Java. I think it's more for businesses.
I've done a first test with a 6.95 GB folder. First backup of all the files, then I changed ONE tag on all the test files (with foobar), and I did a new backup. Here are the results of the second backup :
sent 12.20M bytes received 14.46M bytes 23.52K bytes/sec
total size is 6.95G speedup is 260.51
... so, considering the small amount of sent/received data, I think it really works ! What do you think ?
The tool is not reporting the total I/O, just the parts it worked on.
Assuming I did the math right:
You "read" 14.46M bytes at 23.52K bytes/sec. That's about 630 sec.
6.95 GB transfered in 630 sec is about 11.8 MB/sec, around the realistic speed of a 100 mbit network.
So the tool is reading all your files over the network and then writing just the changes. You've saved no time and put more load on your drives, NAS and CPU. You have introduced an additional point of failure and complication.
Have you confirmed it works with your filenames? UNIX tools wrapped in cygwin don't always handle that properly.
You're doing something drastically wrong if you're regularly changing tags on all the files in your audio collection. I can understand it to some extent as you figure out what's what, but at some point you should find that your collection becomes a very static entity.
washu, I hear you. However, I don't agree with you. I prefer reading 800 GB and writing a very small % of it, rather than writing 800 GB. I don't see why there should be more load on the drives. Writing is more stressful than just reading.
BTW, yes, I have confirmed it works with my filenames.
JJZolx, you are off topic but I will answer you. I've been improving my music library for the last few years, and yes, it has become a very static entity. So you are right, all my tags are very stable now.
The thing is that very recently, I have felt the need to add a new tag field to ALL my music library. FYI it's the Dynamic Range tag, automatically generated by the foobar component.
This sort of thing happens once in a while. You just need to add a new tag to all your music, even if for all the other tags it has become a very static entity, that's all.
So you are right, all my tags are very stable now.
The thing is that very recently, I have felt the need to add a new tag field to ALL my music library. FYI it's the Dynamic Range tag, automatically generated by the foobar component.
This sort of thing happens once in a while. You just need to add a new tag to all your music, even if for all the other tags it has become a very static entity, that's all.
And before that, when was the last time you retagged every file in your library? If you're doing it so often that you really need this type of utility, then I'd be surprised. Backing up the whole 800 GB library to disk on the rare occasion that you edit every single file should be trivial for any disk drive. I think your concerns for the health of your hard drives are wildly unfounded.
washu, I hear you. However, I don't agree with you. I prefer reading 800 GB and writing a very small % of it, rather than writing 800 GB. I don't see why there should be more load on the drives. Writing is more stressful than just reading.
Again I'm not trying to be insulting, but your computer knowledge is obviously very limited. Writing is not more stressful than reading. Additional head movements on your drives is way more stressful then either. You are at the very minimum doubling the number of seeks your drive has to perform. More likely you are increasing them 50X or more as just writing large files would be mostly sequential. You are turning an easy sequential write job into a seek intensive read-write job. Also if your tag changes make the file sizes differ by more than a block then doing this your way will heavily fragment the data. This would cause excessive head movements every time you accessed your data, not just when you updated it.
I'm not saying you will kill your drives, but if anything will do it prematurely it is excessive head seeks. It will also kill performance, you don't notice because you are only on 100 mbit. Upgrade to gig-e in the future and it will hurt your potential transfer greatly.
washu, you are wrong about rsync - that's not how it works. The numbers mean what they say:
- 6.95GB is the total size of the files synchronised
- 12.20MB is the amount sent, this is actual file update data
- 14.46MB is the amount received, all checksum data from the far end
- the rate (23.52K bytes/sec) is simply the amount actually transferred divided by the time taken for the total synchronisation, much of which is time spent creating checksums at each end of the link, and checking the checksums at the sending end.
I use rsync to synchronise around 6TB of data regularly, and I can assure you that it only sends the changes over the network, not the whole 6TB. How can I be so certain - well, I monitor the link utilisation, which has odd spikes of a few seconds at ~250Mbit/s when data is actually transferred, interspersed with long periods of almost no network utilisation but high levels of CPU utilisation at both ends of the link while checksums are being calculated. Also, the whole thing takes around 15-30 minutes on a typical day, which is nowhere near enough to transfer 6TB.
Ouroboros:
What you say is completely correct, IF and ONLY IF rsync is being used on both ends and it is talking over the rysnc protocol. In this case it is not. His NAS is not running rsync, only his Windows PC is. If only one end is using rsync then it has no choice but to read the data in its entirety in order to do the comparison. In this case it is reading it over SMB from the NAS.
Also, you obviously aren't modifying the whole 6 TB each time like wcs13 is. For rsync to even do the compare on 6 TB of changed files in 30 min you would need storage capable of delivering about 3.7 GB/sec. A high end SAN might be able to do that, otherwise it's just not happening.
We don't know how his external HD is connected. It sounds as though it might even be a local disk.
Your disk I/O throughput numbers assume that you are running rsync in non-default mode. In default mode it checks the file size and date/time, and only checks the actual file if the size / date /time have changed. Consequently it reads nowhere near the full 6TB, it simply reads the inodes and directory files, around 1% of the total disk size. Consequently the 3.7 GB/s drops to 37 MB/s - easily achievable.
It's only if you use the -I option that you have to read the whole of the 6TB.
I understand both of you, thank you for your comments.
washu is right in saying that currently my NAS is not running rsync. But in fact... my NAS HAS rsync installed, it's simply not running right now, because I don't know how to install rsync (or a rsync-like) in my Windows part and make both rsync talk over the network. If someone knows how to do this (is there a tuto somewhere ?), I think we'd all be happy !
FYI my NAS is simply a home-made Debian server with no GUI, only a command-line interface. It was installed by a friend of mine. I'm not really in touch with him anymore, but I can still try to type some Unix commands and see what happens, if someone helps me of course...
Ouroboros:
We are actually on the same page here, we are agreeing but saying it differently. Yes, in most cases rsync will do exactly what you said and only have to read a tiny % of the files. That is why I said "you obviously aren't modifying the whole 6 TB each time". This isn't specific to rsync, any good file sync program will check sizes and date/time to avoid copying unmodified files.
However, in wcs13's case he is modifying every file each time. Even though a tag update is a tiny change, rsync still has to scan every file in it's entirety to find out what the change was. Unless rsync is being run on both the source and destination, rsync must read the whole file using whatever disk or network transport is being used. The differences in the end are small, which is why rsync reports such small numbers. It doesn't change the fact that the whole data set was transferred anyway.
Unless the rsync protocol is being used then it doesn't gain any time or bandwidth savings over any other sync tool.
wcs13:
If you have a real PC running debian then that does change things a bit. Most of the time when I hear "NAS" I assume a consumer grade limited device.
Assuming the PC isn't really slow, then it can probably do disk transfers far faster than your network works, so you could actually save some significant time. The bottleneck will likely shift to your USB hard drive.
It's been a while since I've setup rsync on debian, (I'm more into BSDs), but I'll see what I can find. A quick Google shows several tutorials so that should at least get you going. I don't mind trying to help out, if you run into some specific problems I'll try to help out.
This doesn't change my caution about drive wear. The I/O load will be the same or greater. If you are concerned about that then you should look at upgrading your network to gig-e for faster transfers and just use a normal sync program.
@washu.
Got it! Yes, we are both saying the same thing - almost.
Even if you only run rsync at one end, you still don't transfer the whole file to do the initial checking. You use whatever file access protocol you are running to check the date / time / size, and only if they don't match do you need to transfer the whole file for checking. However, it defeats the object of using rsync entirely - if you are transferring the whole file back to the source to check for changed blocks you might as well just transfer the whole file to the destination.
For clarity, you don't need to use the rsync protocol, but you do do need to use the rsync algorithm at both ends to get the benefit.
@wcs13
Set up an rsync daemon on the NAS box. I used this tutorial (https://help.ubuntu.com/community/rsync#Rsync_Daemon).
Then from the Windows machine you run something like:
rsync -avh /Localdirectory/ wcs13@NAS::NAS/Remotedirectory
If you have a real PC running debian then that does change things a bit. Most of the time when I hear "NAS" I assume a consumer grade limited device.
Yes, it's a real hand-mounted PC. It's not really slow, so it shouldn't be the bottleneck. The external hard drive shouldn't be either, since it's not USB but eSATA.
It's been a while since I've setup rsync on debian, (I'm more into BSDs), but I'll see what I can find. A quick Google shows several tutorials so that should at least get you going. I don't mind trying to help out, if you run into some specific problems I'll try to help out.
I think rsync is already set up on Debian, as a daemon. Could you provide me with some way of checking that out ?
This doesn't change my caution about drive wear. The I/O load will be the same or greater. If you are concerned about that then you should look at upgrading your network to gig-e for faster transfers and just use a normal sync program.
Network upgrade to Gig-E is planned but not immediately. BTW, if the network gets upgraded, then I think the Debian PC will become the new bottleneck (it's not really slow, but it's not lightning fast either ).
Set up an rsync daemon on the NAS box. I used this tutorial (https://help.ubuntu.com/community/rsync#Rsync_Daemon).
Then from the Windows machine you run something like:
rsync -avh /Localdirectory/ wcs13@NAS::NAS/Remotedirectory
Ouroboros, I have some questions for you :
- What is this command supposed to do exactly ? Thanks in advance.
- How could I possibly type this command without installint some rsync clone first ?
- If this command creates a "link" between the two intended folders, how can I make that link "permanent" ?
- If this link is created, can I use LicielRsync to backup the folders efficiently, or must I use something else ?
Sorry for all the dumb questions, I'm just trying to figure this out.
Just in case, I remind you that my sync must go in ony ONE direction : from the NAS (where all the "master audio files" are located) to the Windows PC (where only backups are located), not the opposite. Thanks.
I would like to reiterate the warnings about rsync on windows. At least when I tried I had trouble with long paths and filenames with special characters. That is true for many of the other backup software I tried too.
In the end I landed on goodsync. As far as I know that also only copies changed parts of the files.
- What is this command supposed to do exactly?
rsync -avh /Localdirectory/ wcs13@NAS::NAS/Remotedirectory
It creates / updates an archive copy of /Localdirectory onto /Remotedirectory on the machine named NAS, on which there is an rsync account in your name. I'm fairly certain that if you want to do it the other way round (make changes on the NAS then sync them back to the local machine) then you simply reverse the directories, but you should do some reading and experimenting on a smaller subdirectory e.g.
rsync -avh wcs13@NAS::NAS/Remotedirectory/ /Localdirectory
Note that you only need the trailing slash on the source directory
- How could I possibly type this command without installint some rsync clone first ?
Isn't that what LicielRsync is? I've not used it, but I assume that's what it does. It may give you a GUI to hide the detail, in which case you need to read the manual / notes for LicielRsync
If this command creates a "link" between the two intended folders, how can I make that link "permanent" ?
On a linux system you add a cron entry to run the program once a day.
- If this link is created, can I use LicielRsync to backup the folders efficiently, or must I use something else ?
As I said, I've never used it, but that's certainly the impression you get from reading the link.
You should also look at Syncrify, which seems to provide an alternative while still retaining the use of the rsync algorithm and basic architecture.
I would also suggest that you pay attention to the warnings you have been given about disk usage. While rsync (and similar) reduce the amount of traffic transferred very efficiently, I'd be really surprised if they don't rewrite the entire file. I can't imagine that it is feasible to write a program to understand the disk block/cluster structure of every possible file system, then work out which disk block/cluster relates to the changed rsync file block, and only rewrite those changed blocks/clusters. It's more likely that it reads the local file into memory, applies the rsync changes, then writes the entire file back to the disk. In other words, if you change the tags on all of your files, while you may only transfer the 1% changes you still rewrite all of the files.
Another potential issue to watch out for is that rsync when used on Windows does not create file timestamps accurately. This is because Windows actually has far more accurate file timestamps than generic UNIX. In many cases this is not an issue, but it can screw up other sync/backup tools that were designed to work with Windows natively.
Ouroboros:
Getting way off topic here, but rsync does only write changed data to the disk if it can. It just says "write data X at position Y" and lets the file system sort that out into blocks. That's why I mentioned the issue of fragmentation, the file system may not put the changes into a contiguous block. Where as writing out the whole file as a normal sync program does has a much higher chance of getting it into one piece on disk. Using rsync for wcs13 task will very likely make fewer writes in terms of bytes (but not in number), it will make the disk work much harder by increasing the number of head seeks greatly.
i'm also interested in audio file backup solutions and the informative discussion here made me wonder if there isn't a way that would be more specifically tailored towards audio files. after all virtually everything discussed here would work exactly the same with any kind of files and the methods described don't care that the files are audio. the way i see it audio files have 3 distinct "parts" to them: the filename, the metadata (tags) and the actual audio data. generally audio data is static and doesn't change, while metadata and filenames can change, be it because of typo corrections, additions or naming/tagging scheme changes. this makes the usual changes not only very small relative to file size, but also limited to a very distinct, well defined and easily identified portion of the file, which makes a lot of difference compared to a change that could happen anywhere, which is the case rsync type apps have to work off.
so is it not possible to have a backup solution that cleverly ignores the vast majority of data since it's static and only works on the changing parts? you can obviously read & write just the tags (the tagging apps do it) and do a simple rename without re-reading/writing the whole file, but are there any low level file system/hdd inner workings or other factors that would make such a backup scheme not worth the apparent substantial gains? i realize you'd probably need a monitoring part at least for the filename changes, but i'd be more than happy to fire up the monitoring app, do my renaming on the source and have it applied to the backup destination instead of watching gigs of data crawl over usb/network because of a silly typo correction.
I used to sync my music collection (and other data) to an external backup drive using SyncToy (http://www.microsoft.com/en-us/download/details.aspx?id=15155) in echo mode.
Later I moved to the cygwin port of rsync.
But then I finally got to building my ZFS NAS, using inexpensive and low power hardware running OpenIndiana (FreeBSD would have been an option too). It's feature set makes ZFS the ideal technology for the task IMHO. Best-in-class data integrity features and instant snapshots that can be streamed incrementally over any SSH connection to mention just a few things. I'm very happy with the solution and would recommend it to anyone that is at least a bit of a geek.