Small Office Backup with Rsync | January 23, 2007

When setting up our new office, I wanted to ensure everything was backed up correctly. I asked around for backup solutions, but the options were overwhelming. As we were a new company, I didn’t want to spend a huge amount of money on complicated software or hardware solutions, so in the end we went with something that’s already built into the operating system–Rsync.

This small but powerful command line tool forms the basis for a lot of Mac backup solutions, which are essentially GUI front ends. Rsync is much loved by techies, but I’m no Unix wizard soit took a while to get things set up. This quick tutorial outlines how I’ve got backups working at Clearleft. This is by no means a definitive guide, and I’m sure there are much better ways of doing it. So if you’ve got any better ideas, please let me know.

The first step was to find something to back-up to. We thought about network attached storage (NAS), but in the end went for the simple option of a Mac Mini connected to a removable hard drive. We have two such drives and rotate them weekly to ensure we have an offsite back-up.

What we’re going to do is set the Mac Mini up so it connects to each machine on the network at a set time of day, and then run an Rsync back-up. To connect to each machine you first need to give them a distinct IP address on your network.

Go into your network preferences, select the TCP/IP menu option and in the “Configure IPv4” dropdown, select “Using DCHP with manual address”. I’m not sure what the best IP numbering convention is, but we have all our desktops starting from 192.168.1.10, so other devices like routers, printers or laptops can grab the first 10 slots automatically if they want.

OS X network preferences

Once each machine has an IP address, you need to make sure the Mac Mini can connect to it over SSH. To do this, go into the sharing preferences and check the “Remote Login” option.

OS X sharing preferences

Now, lets create the backup command on the Mac Mini. I’ve created a new folder on the Mini called back-up where I’m keeping all my configuration files. Create a new text file in this folder and call it the a sensible name like andybak.command.

First you need to set all the required flags for the rsync command. I’m not going to go into them all, but if you’re interested you can type man rsync for the full list.

rsync -a -v -r -S -x -z --delete -e

The next thing you need to do is connect to the machine and folder you wish to backup using ssh

ssh andy@192.168.1.10:/Users/andy

Now specify the target location of your backup. In our case it’s a mounted volume called “LaCie Disk”

/Volumes/LaCie\ Disk

Lastly we don’t want to back up everything, so I’m going to create an exclusions text file. Add a pointer to this text file next.

--exclude-from /Users/clearleft/backup/andy_excludes.txt

Save this file and create a new file for your excludes called andy_excludes.txt. In this file list all the folders you wish to exclude. I’ve got a lot of music on my machine so I’m going to exclude the music folder. If you have lots of movies or pictures, you may want to exclude those folders as well.

andy/Music/

Save the textfile.

Now we can run the command and see if it works. If you want to be extra cautious there is a flag you can add to your command file that will run a simulation instead of the real thing. As this will be the first time you’ve run this command, the initial backup may take a while. To run the command, simply double click the file and it should launch and run in the terminal window.

The first thing this command will do is try to connect to the computer you’re backing up using SSH. Because this is the first time you’ve connected, it will ask you if you’re sure of the authenticity of the host. Type “yes” to proceed. You’ll next be asked the password of the host machine. Type it in now and the backup will start running. Go make a cup of tea as it may take a few minutes.

Once the back-up is complete, check that a new folder has been added to the backup drive and that all the selected files have been backed up.

Now you obviously don’t want to enter the password each time you run a backup, so you need to set up a public and private key on the backup machine, and then copy the public key over to the host machine. This is where things get a little tricky as there are numerous ways of doing this, some more secure than others. Luckily I did this ages ago, so I’m not even going to attempt to explain how this is done. If you’re interested, do a search on ssh or public key authentication on OS X.

On the Mac Mini, locate your public key. In our case the file was called id_rsa.pub and it was in a folder called .ssh. Using secure copy (scp), copy this key to the authorized_keys file in the .ssh folder on the machine you’re wanting to connect to. OIf the file or folder doesn’t exist, you will need to create it.

scp /Users/clearleft/.ssh/id_rsa.pub andy@192.168.1.10:.ssh/authorized_keys

You’ll be asked for the password of the machine you’re connecting to. Once you’ve entered it, the files will copy over, and you’ll never be asked for a password again. To check the public key is working, run the backup command again and it should run without asking for a password.

We’re almost there. Just one last step in order to make the backups really useful. We need to automate their execution. To do this, you need to decide a time for each backup to run. We run ours in the evening when everybody is out of the office, to avoid the inevitable network slowdown. First, go into the energy saver preferences for the machine you’re backing up, click the “schedule” button and wake the machine up 5 minutes before you plan to run the backup.

OS X energy saver preferences

Then go back to the Mac Mini and edit your crontab file.

sudo pico /private/etc/crontab

Set the time you want the command to run in minutes and hours, and leave the day, month ect starred out, so your backup runs every day. Under the command heading, add the path to your command along with an optional path to a log file.

/Users/clearleft/backup/andybak.command >> /Users/clearleft/backup/backup.log

Do this for every machine on your network, and every night you’ll have trouble free, automated backups.

Posted at January 23, 2007 1:02 PM

Comments

Fedeirco said on January 23, 2007 1:45 PM

I’ve written something about the same topic:

http://docs.maggi.cc/BackupYourMacHOWTO/

I hope this helps.

Remy Sharp said on January 23, 2007 1:51 PM

I just wrote a short entry just yesterday on how to set up the password-less ssh communication between boxes:

http://remysharp.com/2007/01/22/ssh-without-a-password/

Thought it might come in useful.

Nick Fitzsimons said on January 23, 2007 2:20 PM

One tip is not to use 192.168.1.* - it’s the range used by just about every wireless router out there, so can lead to funny effects when those people upstairs get wireless and don’t know enough to change the defaults. Use anything between 2 and 254 (I use 207, selected purely at random) and there’s much less chance of strange conflicts occurring.

Oh, and thanks for reminding me to do a backup :-)

Dave Child said on January 23, 2007 3:20 PM

Entirely too late by far, but I’ve just found CrashPlan ( http://www.crashplan.com/ ) - which will allow you to backup over a network, including the web, encrypted, to another computer (Windows, Mac or Linux).

In your case, you could have your removable drive permanently at a remote location attached to any PC or Mac and have backups done and managed with no fuss.

That aside, great job! Easy solutions like this are exactly what small companies need.

Matt Carey said on January 23, 2007 3:55 PM

Do you have a strategy for when the external hard disk gets full? Is it going to be kept as an archive or recycled? Will rsync let you span drives?

We have been using removable hard drives in a very similar way to you. We had 2 drives, each named exactly the same. We swapped them over every day, with one going home every night with a designated person. That way if we were ever broken into we would only loose one days work.

But the hard disks got full a couple of weeks ago so we have moved to backing up to tape. A lot easier to carry home every night and we can span tapes forever.

Andy Fragen said on January 23, 2007 4:55 PM

You might want to look at rsyncbackup . It does seem to simplify setting up rsync.

Ed Eliot said on January 23, 2007 10:03 PM

Interesting read, thanks. Thinking about it though I think I’d probably want to do the backup the other way round, i.e push the files to the Mac mini from each machine rather than have the Mac mini do all the work. That way you don’t have the potential for the Mac mini to be trying to connect to computers which might not be there - if, for example, you took a laptop out of the office one night.

From a security point of view you also wouldn’t have one computer with password less access to all other computers.

patrick h. lauke said on January 24, 2007 2:16 PM

i’ve been using cwRsync on windows for a while http://itefix.no/cwrsync/ … does the job nicely on that platform as well. i didn’t actually have to assign static IPs … the machine names were enough.

Chris Blow said on January 24, 2007 3:06 PM

A good tutorial on setting up a keypair is here: href=”http://hacks.oreilly.com/pub/h/66

(An old Linux Hacks article by Rob Flickenger).

mattack said on January 26, 2007 4:31 AM

I set up backuppc at my office. It runs on linux but can back up cross-platform and uses rsync, samba, or ssh/tar. It was easy to set up too. I recommend an Ubuntu box with a large hard drive.

http://backuppc.sourceforge.net/