Hard drive backups

[Steve A.]

I do all of my drive letter manipulation through the Administrative Tools in XP. Once inside Administrative Too,s it's Computer Management -> Storage -> Disk Management. The last time I used partition magic I ended up wiping my system drive. It was my fault because I didn't know you shouldn't partition an OS partition, but because of that I now prefer to keep physical drives the same as logical drives.

Hey, thanks for the tip! I don't use the Administrative Tools as much as I should- instead I go for the buggy freeware add-ons that never work as they should.

It looks like the Win XP partition manager does not allow you to resize the partitions- but that is what always gets me into trouble! One problem with PartitionMagic is that it does not display more than 10 drives at least with the monitor that I am using. In that regard the EASEUS partition manager is better as you can scroll down to access any of the drives.

Steve Ahola

[R.G.]

As I type this, I'm running performance benchmarks on my new attic-NAS server. I just got the last of the cables plugged together. It consists of

4U rackmount case, $25, craigslist
AMD dual core Athlon II 235e CPU, $76
4G ECC memory $109
Asus 4A785???? motherboard, $80
400W power supply $39
two Seagate 1.5TB green drives @$80 each, $160
One compact flash adapter $13
One 2G compact flash card $15

for a total of $517 if I did the math right. It runs EON/ZFS from the compact flash card. EON is free.

This can be done more cheaply. You can run the same software on this setup with some cost in ultimate speed - which is not an issue over a 100Mb link:
CPU: AMD Sempron $33
Mobo: Asus M4N68T $46
Mem: 2GB :$50
Power: free, out of your last computer corpse, or trivial, from craigslist
case: free, out of your last computer corpse, or trivial, from craigslist
Boot disK: free, out of your last computer corpse, or trivial, from craigslist
Main storage disks: Up to you. Pick your poison. Disk currently sells for $0.07 per GB at the sweet spot of 1.5GB, gets down to $0.053/GB on sale at frys, and upwards of $0.10/GB at non-sweet spots. You pick whether you'll use it as a backup on its own, or mirrored for redundancy, or RAIDed some way.

A non-mirrored 1TB file can be had every day for $80.

So a minimal homemade NAS can be done for $209 for a 1TB storage size. The extra $300 I spent was for speed, redundancy, and fail-soft behavior. That's tied up in the dual core CPU, ECC memory and support on the mobo, and mirrored disks.

I chose to "waste" half the 3.0TB storage in favor of making a 1.5TB mirrored arrangement, so if I lose either disk, no data is lost if I don't mess about getting the faulty disk replaced. There are two more empty, unused SATA ports to be applied if - who are we kidding here? - when I run out of available space. By that time 2TB drives should be cheap.

The whole setup pulls a measured 61W out of the AC power line.

The setup on this has made me a little crazy over the last few days because I didn't remember what buttons to push, but I'm getting there. It's a little more involved than buying a prepackaged NAS device, but can be accomplished. I found it was significantly less complicated to do than some games.

[Steve A.]

... So a minimal homemade NAS can be done for $209 for a 1TB storage size. The extra $300 I spent was for speed, redundancy, and fail-soft behavior. That's tied up in the dual core CPU, ECC memory and support on the mobo, and mirrored disks.

I chose to "waste" half the 3.0TB storage in favor of making a 1.5TB mirrored arrangement, so if I lose either disk, no data is lost if I don't mess about getting the faulty disk replaced. There are two more empty, unused SATA ports to be applied if - who are we kidding here? - when I run out of available space. By that time 2TB drives should be cheap...

It's a little more involved than buying a prepackaged NAS device, but can be accomplished. I found it was significantly less complicated to do than some games.

R.G.

I did have one question about the fail safe operation. With RAID 1 I believe if the OS starts sending out crap it could corrupt files on both disks. Or is that not an issue with EON/ZSF? I thought that with RAID 1 you still wanted to make automated backups- full maybe once a week and incrementals on a daily basis.

The way I usually lose files is by unintentionally overwriting them. "Crap! Did I just save that file without changing the filename?!?"

Fry's is selling a Hitachi NAS adaptor this week for $30. It connects to an ethernet cable and has two USB 2.0 ports for you to connect your own USB drives. (I have two of the SansDigital boxes that support four 2TB SATA drives with a single USB cable- I wonder if the Hitachi NASty adaptor could handle that? I think I may have read that it only supports two drives up to 2TB each... darn!)

SimpleNET USB 2.0 to 8P8C (RJ45) NAS Adapter

I just checked the users manual and it does not say what capacity drives are supported (nor does the Specifactions on the webpage). It makes me think that this is more of a toy than a serious product.

Just a quick check: how much more NAS are getting from your $357 rig (sans drives) compared to the Hitachi NAS which runs on a few watts (it uses a wall wart- I have never measured the amp draw on those critters)?

Thanks!

Steve Ahola

P.S. Yikes! I just checked the user reviews at Amazon.com and the Hitachi device is extremely limited to say the least...

Amazon.com: Hitachi SimpleNET NAS Head USB 2.0 Portable Dongle SNET (Black): Electronics

If your SimpletonNET doesn't catch fire from overheating it would be good for serving small files over a network. But why not just share the local drives on your computers over your network if anyone is considering using a device as slow as the Hitachi NASty dongle? [With a decent NAS setup like yours I believe that it "takes a load" off the computers that would have otherwise been hosting the drives.]

[R.G.]

I did have one question about the fail safe operation. With RAID 1 I believe if the OS starts sending out crap it could corrupt files on both disks.

Ultimately, there is no protection against the OS sending out crap. Well, if you have recently taken a backup and moved it off line where the OS can't get at it, but that's the only thing. An OS deciding to go wandering through your data stepping on the daisies as it goes is pretty much impossible to stop. At that point you start wishing for an old, well proven operating system. Or I do. Solaris and the Unix it was derived from have been polished for a long time.

Or is that not an issue with EON/ZSF?

It's much less of an issue. ZFS is a copy on write system. Each write is done on a fresh sector, and only when it's all done and checked are the previous version's sectors released. Actually, they're potentially not even released. There is no performance penalty for taking snapshots; the data is still there, and just not released to be rewritten. ZFS also allows you to set a number of dupes of files, so that it can keep more than one copy. They designed ZFS specially to sidestep the common data corruption issues.

I thought that with RAID 1 you still wanted to make automated backups- full maybe once a week and incrementals on a daily basis.

You probably do with RAID1. ZFS uses the 'raid' terminology, but it doesn't do the same operations as other RAIDs with the same numbers. For instance, what would be "raid1" is termed a mirror, and does act much the same, but the nearest equivalent of RAID5 is raidz1, which is single-parity. But raidz1 uses variable width stripes, not fixed stripes across all the disks, and of course is copy on write. And the minimum number of disks needed to build a raidz1 is three. They're similar but slightly different in detail. But that wasn't what you asked.

Every machine that holds the working copy of the data you want to keep has to be backed up at intervals which you don't mind losing. In my case, I keep my working copies in my workstation. The server is to take backups of the workstation. Things which are more static than need to be preserved on the backup server are going to be (I'm working on it!!) spooled to DVDs checked by DVDisaster error checking codes, and this mess refreshed yearly by reading and error checking. Another possibility is the fact that I'm actually working on my second backup server; the first is bigger, more storage, and goes into the garage that's a separate building from the house. Sorry - that still wasn't what you asked.

Yes, you should back everything up at regular intervals you're willing to risk. A mirrored-disk server is no less needy of backup. But in my use, the backup is the workstation where the working copies go, for the fresh stuff, and in the second server or DVDs for the really stale stuff. There is nothing in RAID1 (that I know of!) that makes it particularly more needy of backups than other machines. In a non-mirrored disk system, the first failure kills your data. In a mirrored system, many disk failures that would have lost your data in a non-mirrored system don't cause you to lose data. I'll have to go look, but I believe because of the copy on write and checking setup, zfs mirror is less prone to loss than RAID1, but I'd have to run down the particulars.

The way I usually lose files is by unintentionally overwriting them. "Crap! Did I just save that file without changing the filename?!?"

zfs snapshot stops that from being a disaster. If you use snapshots...

Fry's is selling a Hitachi NAS adaptor this week for $30...It makes me think that this is more of a toy than a serious product.

That was my impression.

Just a quick check: how much more NAS are getting from your $357 rig (sans drives) compared to the Hitachi NAS which runs on a few watts (it uses a wall wart- I have never measured the amp draw on those critters)?

Good question! I suspect that it's like the difference between a carpenter's hammer and a nail gun. Hope so anyway. I got tired of manually moving around a USB drive from machine to machine. Yeah, I could net-share machine hard drives and funnel the backups into four drives spiderwebbed onto a USB port. That just relieves the walking around. And I simply do not trust a multiconnector USB ported whatsit to not quit in the middle of a multigigabyte backup file transfer. I found that backing up a 100GB working disk to a USB drive is slow going. If one of the dogs decides to take a nap on the trailing wires, or a cat decides to chew them, or the dog decides to KEEP the cat from chewing the wires and a chase ensues... well, you can guess.

P.S. Yikes! I just checked the user reviews at Amazon.com and the Hitachi device is extremely limited to say the least...If your SimpletonNET doesn't catch fire from overheating it would be good for serving small files over a network.

This reminds me - I recently went to a steak place in Dallas where they offered s'mores for dessert. Yep, they custom-made marshmallows (which, if you haven't had them, are a major improvement on the puffies in a bag) in 3" by 4" slabs, and brought you chopsticks/skewers, an alcohol burner for the table, and graham crackers. You toasted your own marshmallow slab over the burner, added crackers and...

But back at the NAS.

I would not expect great things from a $30 wiring harness. Neat idea, and better than nothing, but with serious limitations.

What I hope I got/get is a box I can wake up from my workstation, then move backup sets to the box and put it to sleep as needed. Or leave it running from say, noon to 9pm, or only 9PM to 8AM for the overnight backup jobs. The advantage of an independent box running an overly-capable OS that's been cut back because you don't need it all is that the fancy stuff can be added by just not removing it. As an example, I ... think... I could in theory cluster a largeish number of ATX boxes all running opensolaris into a single machine image cluster. I know Solaris 10 will run high performance/high availability clusters. I also know of Solaris installations which run drawers full of disks to get to petabytes of storage. Linux can and does do similar stuff, but I started the quest looking for fail-soft storage, that led to zfs, and Linux doesn't do zfs. Solaris is really just icing on top of zfs to me.

Plus the expansion. The box it's in right now won't handle the disks, but this motherboard has the slots to handle adapters for another 20 or so SATA disks if needed, and the power supply would handle that many if I got the staggered spin up working reliably. I don't think I can expand my storage needs faster than the hardware/software will handle it, and I think it makes me less prone to a data failure.

All that being said, there is a down side to this thing if you're not a computer geek. It takes some typing in command-line unix/solaris instructions to get this mess set up. I managed my own desktop AIX machine back in about 1991-1993, and learned enough of it to be able to look the necessary commands up in the book as I was typing. This is a huge disincentive to people who don't know the problems hidden in a "you asked for it, you got it" interface. There are folks working on versions of EON to go in packaged NAS boxes to bring to market now, so maybe we'll see them.

Meanwhile, I'm trying to convince Solaris geeks that producing a reference design would be a good idea. Publish a recipe: here, buy this, this and this. Hook these up, then type exactly this:... I think it would get a lot more NAS servers made, and I think that they would not get a bad name from crashing like Drobos and other NAS boxes inevitably will because the people who buy them don't understand what they're buying, and truth be told don't want to know.

Hmmm... I wonder if there's a market for a "computer pool boy", a guy who comes in ( perhaps virtually ) once a week and runs the backups, checks the storage remaining, looks at the system logs, and so on. I bet there is.

[Steve A.]

Hmmm... I wonder if there's a market for a "computer pool boy", a guy who comes in ( perhaps virtually ) once a week and runs the backups, checks the storage remaining, looks at the system logs, and so on. I bet there is.

I wanna be the computer cabana boy!

Actually I forgot that this thread is titled "Hard Drive Backups" so your NAS rig *IS* the backup- right? D'oh! I got sidetracked and was thinking strictly of using NAS to store data files centrally. So how many layers of backup do we need- backups of backups of backups? That would be like having guards watching the guards who are watching other guards!

And yes it has been quite awhile since I had a drive trashed because of OS and/or controller issues. Usually I forget to clean off the accumulated dust and it overheats (in which process it can screw up open files when it shuts off abruptly). So I think that the mirrored RAID setup should handle most of the hard drive problems we will encounter these days. I did have a question: if Windows starts sending out crap bytes over the network, they are just files and while they might overwrite a file you wanted to keep I don't think that they can actually trash a drive. Unless the files are Windows system files which would not be the case with your NAS setup. (If it is Windows system files that are trashed the drive will typically not be able to boot up but you should be able to use it as a slave drive and save the data files were on it.)

Your rig sounds really great for automated backups- and I just gave 3 or 4 computer carcasses to the eWaste man.

Steve Ahola

P.S. Has anybody here started using blank Blue-Ray discs for semi-permanent data backup? When the price comes down I think that would be a lot better than the ~4.4GB DVD-R's I am using now. It is not just raw capacity but I'd rather not split a particle project over multiple discs.

[Steve Conner]

The way I usually lose files is by unintentionally overwriting them. "Crap! Did I just save that file without changing the filename?!?"

This in a nutshell is the difference between "backup" and "synchronization".

Backup will protect you from this particular SNAFU: your backup system can retrieve you a copy of the file from before you saved over it.

Synchronization won't, as it merrily overwrites your backup copy of the file with the new corrupt one the next time you sync.

The line can be blurred. Synchronization will save you if you notice the mistake before you sync. And versioning file systems, as used by ZFS and Apple's Time Machine, will keep old versions of files automatically.

Another guideline is: If it's online, it's not a backup. I trust a USB hard drive locked away in a filing cabinet somewhat more than the same drive spinning in a NAS. If it's not plugged in, viruses and evil hackers can't get at it.

But then if it's not plugged in, you don't know for sure if it'll spin up next time you use it...

[R.G.]

I wanna be the computer cabana boy!

Meeee Too!

Actually I forgot that this thread is titled "Hard Drive Backups" so your NAS rig *IS* the backup- right? D'oh! I got sidetracked and was thinking strictly of using NAS to store data files centrally.

Yeah - the NAS is the backup.

So how many layers of backup do we need- backups of backups of backups? That would be like having guards watching the guards who are watching other guards!

There's actually a bit of discipline here. The technical wisdom is that backups should be three layers, perhaps a fourth. You want more than one layer of backup because you may accidentally overwrite good stuff erroneously in a backup (i.e. your "unchanged file name" case or similar) and lose the good copy. So two layers are good. Each successive layer adds another bit of distance between errors/failures and keeping the data you want. Generally, son/father/grandfather is enough for even pro/high reliability setups.

However, one layer of backup is massively better than no backup. It's amazing how reliable hard drives are today, given that most people never back stuff up at all. One layer is massively better. The next two layers put it pretty much completely out of the park, except for natural disasters. "Off site backups" prevent those. As a sidelight, it's amazing how much of a good life consists of being somewhere else when the $&*# falls.

The possible fourth is archival data that you think will never/hardly ever need accessed and probably never changed. That needs written to the medium least likely to ever suffer bit rot.

if Windows starts sending out crap bytes over the network, they are just files and while they might overwrite a file you wanted to keep I don't think that they can actually trash a drive. Unless the files are Windows system files which would not be the case with your NAS setup. (If it is Windows system files that are trashed the drive will typically not be able to boot up but you should be able to use it as a slave drive and save the data files were on it.)

Yeah. It's a Really Good Idea to never put your operating system/program files on the same physical drive as your data. Operating system and program files are just tools, and it's more or less cheap and easy to get new ones. Data files are personally valuable, and effectively irreplaceable.

And yes, you're correct - Windows going mad and sending crap bits/bytes over the LAN will probably not overwrite a file. The bytes over the wire have to be a very special sequence to be accepted as write commands by the remote system, and to go to the right place on the storage disks.

P.S. Has anybody here started using blank Blue-Ray discs for semi-permanent data backup? When the price comes down I think that would be a lot better than the ~4.4GB DVD-R's I am using now. It is not just raw capacity but I'd rather not split a particle project over multiple discs.

I haven't, but it's on the list to try. I worry about bit rot with each new generation of denser storage. Blu-ray is new, and we don't yet know how long until a writable bit begins to go soft. We have some data on writable CDs and DVDs, enough to make use of an error-checking and correcting code useful.

I think the same package I liked for CD/DVD archives (DVDisaster) supports Blu-Ray, but I'd have to check to be sure. In any case, the same process applies - one copy is not safe. More than one copy adds the ability to absorb an error without losing the data. Same principles in mirror/RAID/zfs setups - never keep only one copy. Either keep more than one copy, or parity/checking data that can regenerate what the lost bit was.

Originally Posted by Steve Connor:

Another guideline is: If it's online, it's not a backup. I trust a USB hard drive locked away in a filing cabinet somewhat more than the same drive spinning in a NAS. If it's not plugged in, viruses and evil hackers can't get at it.

But then if it's not plugged in, you don't know for sure if it'll spin up next time you use it...

Yep, the state of "backup" is not perfect today.

However, I would restate that this way:
If there is only one copy online, it's not a backup. In fact, if there's only one copy, period, it's not a backup. Fire/flood/ice age/solar flare/etc. can kill it.

But I view two distinct copies existing somewhere as a backup. To me, "backup" means "there's another copy of what I did last that I can retrieve." The fact that I have one copy in my work station and a second copy in spinning disks in the attic doesn't mean there's no backup, only that the spinning backup may be more fungible than if it was off-line. Or not. The data is tough to tease out.

I like this paper: Failure Trends in a Large Hard Drive Population. However, it fails to address what my experience and intuition tells me is the most common failure mode - mechanical failures.

This gets back to the issue of whether some mechanical device fails when it's running or when it's stopped. Hard to tell. I have had my cars suddenly start showing a problem while driving, but failing to start is more common. I personally view the possibility of acceleration-caused (i.e. oops, I dropped it!) failures and wear/breakage on connectors and cables as being more likely with a USB drive kept in a closet because I'm fumble-fingered. I view it as less likely that a hard drive kept in a chassis somewhere and powered up/down without moving the chassis as better. But that last does subject it to vibration/heat/hacking.

I attacked the vibration/heat issue with low power. Low power equals low waste heat, especially in a cabinet that's really too big for the stuff inside. And fewer disk drives, coupled with some vibration damping panels inside the cabinet walls, help keep the mechanical vibration down.

I attacked the hacking issue with firewalls. I have a hardware NAT/firewall at the entry to the house LAN. So far, this has kept the jungle at bay at the firewall, giving me a private "green zone". So far. I have no illusions that it will keep every hacker out forever. However, use of Solaris as a base provides for free (i.e. not stripped out ) a stateful IP/port filter setup built into the network drivers. It allows me to further firewall the NAS down to responding only to preselected, well-formed requests from specific IP addresses, which in my case are non-routable IP addresses on the house net. That at least makes it a little harder in general to get into the NAS even given that the house LAN is compromised.

Then there's that remote shutdown/wake up stuff I am working on. As you note, it's hard to hack it when it's in the closet turned off. My corollary to that is that it's just as hard to hack it when it's in the attic turned off most of the time.

Anyway, I make no claim to be a backup/security expert. All I know is what I read.

[Steve Conner]

This gets back to the issue of whether some mechanical device fails when it's running or when it's stopped.

All the data loss I've ever suffered consisted of hard drives that failed to spin up one morning when the computer was powered up. It's happened about 3 times in my career so far, and each time I lost a day or two worth of work.

My backup system has always been a couple of USB external hard drives that I back my computer up with every time I do a significant amount of work, and then unplug from the computer and stash somewhere secure.

The data I generate at work is the most important thing, so I keep a copy of that off-site at home. I did a trial disaster recovery over the Christmas break and was able to restore everything onto another old PC, which I now keep in a closet as another off-site backup, or if I ever want to work from home for some reason.

That's another point to bear in mind, a backup system means nothing until you've tested it, and you need to test it now and again to make sure it's still working.

[R.G.]

All the data loss I've ever suffered consisted of hard drives that failed to spin up one morning when the computer was powered up.

Me too. It appears to me that they simply fail to show up for work one day.

My backup system has always been a couple of USB external hard drives that I back my computer up with every time I do a significant amount of work, and then unplug from the computer and stash somewhere secure.

Ditto.

That's another point to bear in mind, a backup system means nothing until you've tested it, and you need to test it now and again to make sure it's still working.

Oh, yes!

My thinking on backups actually included the issue of failing while it's not powered on and on needing to test the drives periodically.

That last is one thing that led me to zfs. It has a native ability to scrub the disk contents in background, reading, computing the error-checking/correcting data, and fixing problems, while accumulating error history during the scrubs.

The "cold storage" archive on DVD does need periodic review. That's why I went to DVDisaster - it has a built-in checking mechanism, both for the bare disk (i.e. "yep, I can read all sectors without any reported errors from the on-disk ECC") and also the separately-compiled ECC files, which are kept on separate media disks (ideally) from the data disks. So once every so often, you have to fire up DVDisaster, run the disk through its checking routine, and when you find the first sector reporting errors, burn new media with the corrected data using the separate ECC files.

You could not be more right - if you can't actually back up from it, it's useless.

[Steve A.]

All the data loss I've ever suffered consisted of hard drives that failed to spin up one morning when the computer was powered up. It's happened about 3 times in my career so far, and each time I lost a day or two worth of work.

As a former HVAC service tech that was my experience with AC and furnace motors; once they were running they usually kept running until they eventually died in a shower of sparks. Hmmm... I guess that is not very reassuring.

In any case, it is the starting torque that will usually kill a motor. Once it is running it doesn't take as much torque to keep it rotating as long as the bearings do not go dry. BTW a lot of motors do not have oil ports. But if you put a lightweight oil on the motor shaft gravity can help the oil get into the bearings.

You might try that trick on the fan in your power supply- or on a bathroom exhaust fan- but you should do it the first time you hear any bearing noise.

... The data I generate at work is the most important thing, so I keep a copy of that off-site at home. I did a trial disaster recovery over the Christmas break and was able to restore everything onto another old PC, which I now keep in a closet as another off-site backup, or if I ever want to work from home for some reason.

That's another point to bear in mind, a backup system means nothing until you've tested it, and you need to test it now and again to make sure it's still working.

I remember walking into the local computer store around 1988 and the salesman said "hey watch this!" as he proceeded to format the boot drive in the computer. Wow- everything was gone! He stuck in a PC Tools 5.25" floppy and ran the unformat utility and everything was back. Whew- I was worried there for a minute.

Starting with PC Tools back in 1988 I have never found a proprietary backup format that lasted much longer than 5 years. Subsequent versions of programs never seemed to be able the old backup disks (or tapes). While the files seemed intact when it came time to producing the file I was looking for, it was always "cannot locate file".

I think that there should have been a standard format in DOS for backing up files, folders and disks to removable media. If a company wanted to use their own proprietary super duper format for backups, go for it. But make mine DOS.

In lieu of all that I have kept my backup files in an uncompressed format so if the file can be read, it is back.

Steve Ahola

[R.G.]

Starting with PC Tools back in 1988 I have never found a proprietary backup format that lasted much longer than 5 years. Subsequent versions of programs never seemed to be able the old backup disks (or tapes). While the files seemed intact when it came time to producing the file I was looking for, it was always "cannot locate file".

I think that there should have been a standard format in DOS for backing up files, folders and disks to removable media. If a company wanted to use their own proprietary super duper format for backups, go for it. But make mine DOS.

In lieu of all that I have kept my backup files in an uncompressed format so if the file can be read, it is back.

Wise words. The problem with backup media in general and backup formats in particular and especially is that the machine to read the media and the software to read it go out of production/use/support within 3-5 years. Even if the medium holds good data forever, you have to be able to read it, and in the written format.

Back at the clay tablets and hieroglyphics: fortunately, the reading mechanism is the human eye, and that's been relatively preserved over time: they still make the machine! But the "backup format" had been lost.

The Rosetta Stone was profoundly useful because it represented a translation tool between backup formats.

I have always been suspicious of compression algorithms. Compression amounts to encryption if you no longer have the algorithm. I prefer uncompressed backups, and unencumbered file formats as you note, and for the same reasons.

In most cases, the media lasts longer than the machines that read it.

One component of good data integrity is that periodically, and much more frequently than the medium-reading machines can go out of currency, data has to be read, checked for errors and corrected, then transcribed into new media.

I have an example which I found out about only yesterday. My friend the amp tech has made enough of a reputation for himself that he gets sought out by touring bands to be their on-the-road tech support shop, and handsomely paid for it. He had a very capable cell phone with which he took tour pictures, along with pics of this family, friends, etc.

You probably picked up that "had". He lost it, or it was stolen. His whole contact list and set of photos had not been downloaded. No backup. Every place you have data that's worth keeping, you have to back it up.

[bob p]

Hmmm... I wonder if there's a market for a "computer pool boy", a guy who comes in ( perhaps virtually ) once a week and runs the backups, checks the storage remaining, looks at the system logs, and so on. I bet there is.

There is a market for that kind of thing at the enterprise level, where certain essential but boring / tedious / redundant jobs get contracted out. as an example, Bank of America has cabana boys -- they bill under the name of International Business Machines. They charge a lot of money though. I dunno if anyone does this sort of thing for the little guy at a rate that would be affordable.

[bob p]

Steve, there was some talk up above about backup schemes. Most people are familiar with the Full Backup / Incremental backup paradigm, but its important to remember that the combination of a Full + several Incremental backups still only represents one layer of backups.

As RG mentioned, most real backup systems provide 3 layers of data. When it comes to 3-layer systems, there are two backup schemes that are the most commonly used. One is called "Grandfather-Father-Son" and the other is called "Tower of Hanoi" after the child's puzzle that bears the same name. A little googling on either of those topics will provide you with LOTS of reading material.

Suffice it to say that if you really want to protect your data, you need to go several layers deep, and you need to move data offsite. The best multiple-layer backup system in the world still won't help you if all of your data is on-site and the house burns down or someone steals the computers. So if you want to get really serious about this, you're talking about 3-layers on-site and 3-layers off-site. Remote off-site backups aren't that hard to do if you've got sufficient bandwidth.

If you've got a friend/family member who will let you run a server in their attic you can do all of this stuff automatically.

[bob p]

Wise words. The problem with backup media in general and backup formats in particular and especially is that the machine to read the media and the software to read it go out of production/use/support within 3-5 years. Even if the medium holds good data forever, you have to be able to read it, and in the written format.

This is a really important point -- in the past I've been burned when using proprietary backup schemes. If the company goes under, gets bought out, changes the algorithm, or the hardware fails, you can really find yourself in a tough spot. Back in the 80s/90s I used a proprietary system that burned backup data in an encrypted format on a stack of CD-R. It was state of the art at the time. I created stacks of backup CDs on spindles to protect my data.

Unfortunately, I lost the ability to access the data when one of the hardware/software links to the proprietary decoding algorithm was broken. I never expected that those encrypted CDs that I burned would never be readable when my brand-name CD-RW drive broke and I replaced it with something else. The decoding software refused to decode the CDs and perform a system restore because I was no longer using that company's CD-RW drive!

From a practical standpoint, there are really strong arguments for using as generic an approach as possible, and shadowing an exact copy of the data in its native format. If you go that route, very simple solutions like RAID and RSYNC can be really helpful. Given the plummeting cost of live media, you really have to question whether or not its better to store a copy of the raw data in its native format, or to compress the data to save space.

My personal preference is to use the simplest method possible. Optimally, you want to configure your setup so that if you're ever faced with emergency reconstruction, you won't be needing any specialized tools. Specific hardware requirements, like a spare RAID adapter, can make things difficult. So can specific software requirements, like using a specific backup system that requires lots of configuration. HW & SW requirements like those only serve to complicate the process of a system restore.

Its really handy to have a physical volume that carries a shadow copy of the data in its native format, as it can be moved into position with a minimal amount of fuss. Simply moving a drive is easy. Having live data on the network is easy too. Its really handy if all you need to do to restore a broken drive is to replace the physical medium and resync the system.

[bob p]

Meanwhile, I'm trying to convince Solaris geeks that producing a reference design would be a good idea. Publish a recipe: here, buy this, this and this. Hook these up, then type exactly this:... I think it would get a lot more NAS servers made, and I think that they would not get a bad name from crashing like Drobos and other NAS boxes inevitably will because the people who buy them don't understand what they're buying, and truth be told don't want to know.

I'm not at all familiar with the Solaris side of things, but I think that this is exactly what the BSD guys have attempted to do with FreeNAS.

I had given some thought to building a FreeNAS box, just to get my hands on a turnkey installation that uses ZFS. There are a few unfortunate side effects that I ran into when I was thinking about setting up this kind of system:

a. i'm familiar with many flavors linux, and multiple linux distributions makes life complicated enough. i wasn't too excited about having to learn my way around BSD.

b. The HCLs (hardware comparability lists) for "recipe" installs are often pretty short, so you get locked into a short list of options.

c. the turnkey installs are often limited in what they'll do for you. they work great as long as you don't think outside of the box that encompasses the original designer's objectives. if you understand what the box does, and you want to do something that goes beyond the basic recipe, then you end up having to do a complete DIY. the NAS recipes aren't that great for people who like to tinker.

i just built another file server for the home LAN, and i guess its a NAS box according to current terminology. i looked at a number of the NAS box software packages and i found that my needs weren't fully served by the turnkey options that are available. the result was that to get a fully featured system to do what i wanted it to do and was tweakable with the level of granularity that i wanted, i had to stick with configuring an OS from the ground up. i ended up building another linux box, primarily because of familiarity and i wanted to follow the path of least resistance. someday i really want to build a box that supports ZFS, just to try it out.

if you're concerned about ultimate reliability, another option is what's known as "high availability" file servers that use a distributed file system like RedHat's GFS or Gluster. essentially, these systems distribute redundant copies of your data across redundant file systems that lie across multiple file servers. the servers keep track of one anther's status using a heartbeat algorithm and mirror one another's data. if one server fails, the system automatically reroutes traffic to one of the other servers as it steps in transparently. that's how mission critical data gets handled at the enterprise level. i've thought about doing the redundant server setup for the home LAN using CentOS, but i can't even think about doing that until I tackle the consolidation problem I've got facing me... more on that in the next post.