Mac-based Studio Storage and Backup Advice | July 21, 2006

Clearleft recently moved into a new studio, and one of the first items on our todo list was sorting out a centralised storage and back-up solution. I’ve been looking into the various options and have to admit that I’m a little paralysed by choice. This is largely because I have very little technical experience in this field, and everybody I talk to suggests a different option.

We want to set up a machine as a local dev server so we can test sites centrally. We’d probably set up SVN and develop on the local machines, but commit to the central server for testing. As well as the actual site files, we would probably add other collateral into the repository like photoshop files etc.

We’d also like to use this machine to store common files like word templates, stock images, icons etc. We’d probably want to edit these files on the server,, rather than copy them over to our local machines, update them, and then copy them back. As such, any system we use needs to be reasonably fast. I also thought about using the machine as a centralised music server, but that may be overkill.

We’d like to back-up all the info on our workstations to the server at the end of each day. We’d also like to have a redundant back-up that we can take off-site each evening. The back-up solution would need to run automatically and be easy to re-instate files if necessary.

We want to get a set-up that will last us a few years and will be easy to expand when we add more workstations. However we don’t want to spend a fortune and an XServer is probably overkill for our needs.

We’ve been thinking about getting a mid-range G5 tower and it’s been suggested that we get two SATA drives and mirror them for extra redundancy. I’m not sure how the whole mirroring thing works, although I’ve been told that this is something OS X can handle. If not, I guess some kind of SoftRaid solution would work. We would then get a firewire drive to use as our off-site back-up and take it home every night.

A few people have suggested that, rather than back-up each workstation to the server each night, we actually have all our home folders on the server. This sounds like a sensible suggestion as it means we only have one disk to back-up, so it should be faster. However I’m worried that it may slow our home folders down, as well as leave us with a single point of failure. Basically if the server dies, none of us can work.

The other option would be to simply back-up our home folders to the server each night, minus our music directories, and then back that all up to the firewire drive. However I’m concerned that this type of back-up may take a really long time and slow everything down. I know that my home computer slows to a crawl when the back-up kicks in and it has to scan all the files for changes.

If we go down this route, we need to decide what type of back-up software to use. The Mac suppliers obviously want us to pay for Retrospect workgroup, but its quite expensive, and probably overkill. Also I never hear particularly good things about it. Another option would be to use something simple like Chronosync, at least for the time being. Somebody else recommended getting OS X server and using Net Boot and Net Restore, but I don’t know enough about these apps to make an informed decision.

So I’m really interested to hear what you guys recommend as the optimal, small mac-based studio, storage and back-up solution. Something that is easy to set-up, reliable, not too expensive, but has some room for expansion if we need it.

Posted at July 21, 2006 10:44 AM

Comments

James Wheare said on July 21, 2006 11:16 AM

Hi Andy,

I cannot recommend Shirt Pocket’s SuperDuper! enough. Very Mac-like, simple, fast, no-nonsense and has a useful SmartSync feature that means only files that have changed each day get copied over.

It’s also extensible with good scripting support and if you get stuck, their support forum is pretty responsive.

Pretty sure it’ll be perfect for your needs. And dirt cheap at $27.95

Good luck!

James Wheare said on July 21, 2006 11:19 AM

…only files that have changed each day get copied over.

Sorry, I meant, “changed since the last backup.”

Scott said on July 21, 2006 12:56 PM

You should definitely be looking at doing some data redundancy using some form of Raid formatting for drives in whatever sort of hardware you decide to get. My experiences with Apple’s Xserve’s has been quite good and the software-based Raid mirroring in Disk Utility is very easy to use. You could always go the extra mile and add on an Xraid and use real hardware based Raid, but it might be overkill for what you need currently.

My recommendation would be to have no less than 3 drives available. Mirror 2 of them for general redundancy, but backup to the third. If you want to take it one step further, get a fourth drive and clone the third (backup) drive to it as one extra layer of protection. Take it offsite for even better security/protection in case of theft/fire.

In terms of backup, SuperDuper! is very good as James suggests. I use it myself for system cloning and some backup tasks. Retrospect is a pain and not much fun to setup/use, so at least initially, SuperDuper! or something similar might suit your needs and you can look to something more robust later.

Cliff Boobyer said on July 21, 2006 1:11 PM

You could always rely on a pc raid? But then again I’m not sure it comes in white and chrome ;)

Josh Delsman said on July 21, 2006 1:48 PM

The built-in Mac OS X solution for even client systems is software RAID, so if you are really hardcore about making sure your data is safe, I would recommend even buying an Xserve on eBay. It’s completely worth the money in the long run, and the feature set is amazing. I deal with them daily, so you can e-mail me if you have any questions about ‘em!

Walker Hamilton said on July 21, 2006 1:53 PM

When you get your new OS X box, put the install disk in boot off of it, and open disk utility. This will allow you to format the disks as a RAID 0 (Mirroring). Then you simply install OS X on that single volume (which is two physical drives).

You should have another set of drive next to the server, every night you take one home. Another, you leave hooked up to the server and let CarbonCopyCloner clone your server drive over to that external drive. You bring the drive with you to work the next day, swap them and take that new version home. (this works best with 3 drives - one is always at home, 1 travels to work, another backs up and travels home).

Then if your office burns down or your server dies, you plug in that recent backup and clone it back over to the server. (CarbonCopyCloner again).

Mike Rumble said on July 21, 2006 1:59 PM

Can’t help you much with the technical details of what hardware to use and how to set things up - I tend to leave that to the eggheads.

However, I can extoll the virtues of subversion - it’s great and totally changed the way we work in our office making it easy for multiple people to work on a single project.

One thing though is that I question the value of keeping Photoshop files in Subversion, we’ve generally found that the version control doesn’t work too well with binary files such as photoshop files and diff is all but useless.

John Wise said on July 21, 2006 2:47 PM

I’d suggest a NAS (network attached storage) device as your central server, rather than a dedicated computer. You can handle the workstation-to-server file sync with SuperDuper!. Backing up the NAS is as simple as pushing a button on the NAS, once you’ve configured what to backup and attached an external USB drive (sorry, no firewire here, but it’s got Gigabit Ethernet ;)

http://www.infrant.com/products_ReadyNAS_NV.htm

Jan Brasna said on July 21, 2006 4:29 PM

You already got good tips for managing the drives/backups themselves.

For the data management I advise you to add positively everything to the SVN repository (or maybe Trac to manage all the projects and belonging assets on the top of SVN…), even images, documents, proposals, client brief - then you have versioning/revisions/history/authors and you can easily back up all the project data from one source.

Dominic Mitchell said on July 22, 2006 11:29 AM

I’ll second the subversion recommendation. Have a subversion server running, check out from it to the local workstations. Everything that’s checked in again gets backed up.

If you want me to come in and spend some time on it with you I’m more than happy to do so.

David Stone said on July 22, 2006 12:33 PM

Firstly, you’ll need two servers, one on-site and one off-site - as much as you’d like to think you’ll be taking the back up home every day.. you’ll forget or go to the beach for BBQ/drinks! The off-site one copies all the data from the on-site one. The theory is if the data is on the on-site server, you’ll have remote back up incase of fire etc, now my 2p point by point:

1) We want to set up a machine as a local dev server so we can test sites centrally. We’d probably set up SVN and develop on the local machines, but commit to the central server for testing. As well as the actual site files, we would probably add other collateral into the repository like photoshop files etc.

SVN for every project and once you’ve checked in a commit it automatically exports to an area on the server so you [and clients] can preview it over HTTP (although it will be a little slow remotely if your on ADSL). The ‘automatic export’ should be to an area you can access and edit (avoid this though) - just incase there is some server side code you need to debug on server.

2) We’d also like to use this machine to store common files like word templates, stock images, icons etc. We’d probably want to edit these files on the server, rather than copy them over to our local machines, update them, and then copy them back. As such, any system we use needs to be reasonably fast. I also thought about using the machine as a centralised music server, but that may be overkill.

Use a Samba share both of these. A centraised music server I’d recommend - it’ll make life easier when your trying to find that tune that Jeremy was playing the other day. Your network should be alright, save time (I assume, never had to worry about this one) would be depending on the speed of your network (100 or 1000 Mbps?) and the speed of hard disks in the server (if worried get SATA300).

3) We’d like to back-up all the info on our workstations to the server at the end of each day. We’d also like to have a redundant back-up that we can take off-site each evening. The back-up solution would need to run automatically and be easy to re-instate files if necessary.

For backing up your workstations you can dedicated part on the on-site server for this. Assuming your workstations can boot from network set your workstations to boot, backup, and shutdown over night. Again, use a Samba share to access back ups so it’s just like browsing your sites - very easy to access, this should be Read Only BTW.

4) We want to get a set-up that will last us a few years and will be easy to expand when we add more workstations.

Use RAID and just plug in more drives (plan to have space inside your box!). The only part of this setup that would need upgrading would be hard disk space.

5) thinking about getting a mid-range G5 tower and it’s been suggested that we get two SATA drives and mirror them for extra redundancy. I’m not sure how the whole mirroring thing works, although I’ve been told that this is something OS X can handle. If not, I guess some kind of SoftRaid solution would work. We would then get a firewire drive to use as our off-site back-up and take it home every night.

I’ve not looked inside a G5, however, if it’s anything like a G4’s you’ll run out of room inside those quickly - espically if your mirroring them (think 1 drive for the space of 2). Mirroring [or RAID1] is having two hard disks with identical information on them - this means read time is twice as fast and write time is twice as slow as a stand-alone disk. As I said at the top, forget a firewire drive:

6) A few people have suggested that, rather than back-up each workstation to the server each night, we actually have all our home folders on the server. This sounds like a sensible suggestion as it means we only have one disk to back-up, so it should be faster. However I’m worried that it may slow our home folders down, as well as leave us with a single point of failure. Basically if the server dies, none of us can work.

I did this on Windows, had “My Documents” as my home on the server, performance was fine. Yes, it might leave you with a single point of failure, however, if all your client files are on the on-site server and it fails your stuck anyway. You could have access to the off-site server from your office and download what is needed in an emergency should this happen so you can carry on working. I can’t tell you how OSX will act to ‘loosing it’s home directory’ though.

7) The other option would be to simply back-up our home folders to the server each night, minus our music directories, and then back that all up to the firewire drive. However I’m concerned that this type of back-up may take a really long time and slow everything down. I know that my home computer slows to a crawl when the back-up kicks in and it has to scan all the files for changes.

As above, do it automatically at night.

8) If we go down this route, we need to decide what type of back-up software to use. The Mac suppliers obviously want us to pay for Retrospect workgroup, but its quite expensive, and probably overkill. Also I never hear particularly good things about it. Another option would be to use something simple like Chronosync, at least for the time being. Somebody else recommended getting OS X server and using Net Boot and Net Restore, but I don’t know enough about these apps to make an informed decision.

Rsync. Free. Open source. Powerful. ‘nuff said.

The next thing to do would be to have all your email stored on server so you can easily back that up, and access it from anywhere etc etc… next time we’re out for beers/BBQ grab me for a chat - I setup something almost identical for The Escape Committee.. and it’s still running a few years on.

Cheers,
Dave

Jonathan Hohle said on July 22, 2006 2:45 PM

I just set up dirvish to do full nightly backups of of all the computers in my house (too many). it uses rsync and hardlinks, so if files don’t change, they don’t get recopied, but every night’s backup repository always has a full file set.

I’m using Linux on the backup server, and the only problem with that is Linux’s rsync won’t save extended attributes. I get around that by mounting the backup directory over NFS and having the mac back itself up every night. If you had an all Mac network, the Mac backup server could easily do mac and Unix computers without needed to set up each client individually.

For backups, you only need 120-150% of the area you initially backup (depending on how often your files get modified), restoration is easy, and you can set up all the rules you want for excluding files (none of the mp3s in my iTunes directory get backed up, but all the purchased music does, for example).

It’s definitely a set it and forget it kind of thing, which I was happy about, but its entrenched in Unix-y setup.

Michael T. Smith said on July 23, 2006 7:38 PM

As for setting up home directories on a central server — OS X Server can definitely handle it (using LDAP.) Server comes with workgroup management software, that while a pain in the butt to use, helps get everything going pretty fast. With OS X Server you can also set up sharepoints so that you could put all of your non-versioned files in a central location.
I run IT for a summer camp in Pennsylvania and the only issue we have faced in our computer lab (where we have the home directories located on a G4 server) is just plain old network slowdown. I would just suggest a gigabit network infrastructure to help that problem.
Finally, have you played with Adobe VersionCue 2? I’m not a huge fan, as it does have some basic issues (speed!), but it may be the SVN replacement you’re looking for your binary files. I would suggest Version Cue or Extensis Portfolio.

Matt Carey said on July 23, 2006 10:54 PM

Very interesting thread as I am looking at this issue in my studio.

Currently we are running a solution which is held together with sticky tape and string! It worked fine when it was just me, and OK when my partner joined. It doesn’t work very well now there are 4 of us.

I’ve been trying to look into a SvN solution but I think I need an idiots guide to setting one up…

We currently work on our local machines and use chronosync to move any changed files over to the the fileserver. What we don’t have set-up is the off-site back-up.

Tony said on July 23, 2006 11:53 PM

I work at an all-Mac ad agency, and our home folders are on a server. Since I’ve been there, about once a month we lose 15 minutes of work to downtime related to the single point of failure problem you mentioned.

Matt Carey said on July 24, 2006 10:39 AM

there are many options to where files should be stored during the working day, but less options on the off site back-up question.

people have mentioned the multiple firewire drives option (carrying one home each evening on rotation).

what about tape drives - anyone still use them?

or an off-site server, but the cost and bandwidth makes me nervous!

Lou said on July 24, 2006 7:25 PM

Here’s our setup (still in a changing state).

WebDAV mounted Subversion shares for everything. XCode might work in SVN but most of my users don’t so this is the best option.

Home folders are backed up to the server every night. We’re still looking into which utility we’ll use, but we have SuperDuper for now backing up nightly. We’d like to have one that can catch up in the morning when a laptop rejoins the network if certain rules are met (more than 24 hours since last sync).

That takes some disk space. So we have an XServe set up in a mirroring raid. Write speed is 1x, read speed is 2x. Not ideal but over gigabit and 802.11g it’s ok.

The mirroring takes care of the server redundancy, so we just worry about a back up of the server. Right now that’s handled by a spare G4 with some big disks. It’s not remote but it’s in another room down the hall. Every night, after the other backups, it copies everything over. That takes a while.

I’d prefer to have a second XServe getting written to live (active mirror) but that requires a bit more hardware and setup than I have time and budget for now.

Vincent Grouls said on July 28, 2006 7:37 PM

At work we still use tape drives and a lot of the industry over here still does. Sure it’s quite slow when restoring but in the end it’s cheaper than firewire I would think.

Have you thought about setting all users up as Mobile Users? This is a feature in OS X Server specifically for laptop users as the changes get synced automagically whenever you connect to the server (using LDAP). Could be useful in combination with a program called MobileUserSync.

Phil Palmieri said on August 3, 2006 1:23 PM

Hey Andy,

We use an XServe and an XRaid box, the raid holds all the data nicely and we have it on RAID 10, Disk 0 - 1 are striped, then mirrored to disk 2-3.

Next we went out and bought Iomega’s REV drives, we bought a firewire one and a USB2 one, so that we could just plug the USB2 into a PC or any other machine from a store in an emergency.

Using RSYNC (Free) alogn with CRON (FREE) we did a 3 hour diffeential backup to disk 4-5 (RAID 0).

Then, each day at 5:00pm it ran a differential from the Disk 4-5 to the REV drive.

Finally, we also moved a copy to the other rev drive at 6:00pm - then removed that disk in the morning.

This has worked phenominally well. We had a drive failure on the RAID box and no-one knew.

— oh, on the raid box, everyone (even the windows people) had their own home directory they worked out of. - this made backups easy, you just rsync the /Users directory.

Phil

Matt Carey said on September 1, 2006 12:02 AM

Any news on what solution you went for Andy?

Andy Budd said on September 1, 2006 8:30 AM

In the end we went lo-fi and bought a mac mini and a couple of firewire drives. We’re using rsunc to back-up our machines to the mini every night, and then swapping drives over once a week for redundancy.