Hard drive backups

[Steve A.]

Norton is regarded by most people who think about antivirus things as very much like Dune's Baron Von Harkonnen - bloated, mentally and physically diseased, probably smelly if you get too close, and at its center, pure evil, any good it does being an accidental byproduct of whatever evil it was trying to do. Did I mention that I don't like Norton AV?

Afterthoughts on this.

In the past month while browsing around on the shadier side of the tracks I have gotten several messages from Norton that they stopped an attack on my computer- you can click to get more details, most of which are over my head.

I never noticed any of the other AV programs I've used doing that; perhaps they did it silently without any notification.

There are a lot more hackers these days and they do not need to be computer whizzes. I was just reading an article that said that there are programs that they can use that only require them to check the boxes for the features that they want. Voila! Instant malware.

I'll have to dig up that article and post a link to it.

Steve Ahola

P.S. If anybody has been getting similar messages from the AV that they are using please let me know!

[Steve A.]

Very possibly the easiest way to backup your boot disc

My boot disc has been flaky for some time. I have not been able to defrag it since it will abort when running into problems.

So yesterday I decided to copy my boot disc to a new drive using the WD Data Lifeguard Tools, selecting the "make boot disc" option. For a 250GB drive it only took about 2 hours to make the copy of C:, E: and F:.

Rather than hassling with DVDs or using a ghost/backup program that produce one large file in a proprietary format, an exact copy of the disc has files which are easily accessible.

So if you alternate between two hard drives and create a bootable copy every week, you could replace your boot drive if your boot drive was to fail with a copy not older than a week. And you should be doing incremental backups of all newly created or edited files so you should be covered.

Just another way to skin the cat.

Steve Ahola

[tedmich]

whats wrong with a big NAS system setup 5 x 1Tb with weekly backups and 2 sets of drives that are swapped biweekly and retired when they reach 20% MTBF (all in a fireproof climate controlled outbuilding)?

they say BR disks last 200yrs stored in dark dry places, sue em if they don't!

Tape backups? ugh...

Pioneer says they have 900Gb opticals with 16 layers...some day soon...

[Steve A.]

whats wrong with a big NAS system setup 5 x 1Tb with weekly backups and 2 sets of drives that are swapped biweekly and retired when they reach 20% MTBF (all in a fireproof climate controlled outbuilding)?...

If I had a system like that I don't think that I would put it in an outhouse- oops! you said outbuilding. My bad!

Steve Ahola

[MistaT]

If you have an old PC, you can install Ubuntu on it (Its a free Open source operating system..) put a big hard drive inn it for say, $100, and then install Samba on it. (again free..) then just use this for a backup server.

This is what I do at my home, and at the radio station that I work at. Just snag an old 1.5 - 2 gHz PC that most people throw away these days, and put the biggest hard drive in it. A PC like that will only use about 1% of the cpu resources while you are beating the hard drive to death with files.

Mine are on a UPS, and I haven't rebooted them in well over a year. They just run / work.

[bob p]

RAID

Steve, as I read this thread I can't help but think we've had this conversation before... either that or I'm just suffering from a case of deja vu...


One of the major problems with RAID is that you can really hurt yourself if you don't know what you're doing. It used to be that RAID was only available on high end equipment, and since most people weren't using SCSI subsystems RAID wasn't being widely used. Only tech geeks who knew what they were doing used it, and most users didn't even know what RAID was, much less how to implement it safely.

All of this changed dramatically with the advent of "Software RAID", or "fake RAID".

Originally, "Hardware RAID" required an expensive, purpose-built, controller card. These cards had processors on them that were optimized for rapid XOR processing, and they were stuffed with lots of memory. They were expensive, and they were reliable.

"Software RAID" came along when CPUs developed enough clock speed to have disposable CPU cycles, and enough memory that data-intensive tasks could be off-loaded from dedicated hardware onto the CPU. This resulted in a proliferation of inexpensive (cheap) SW Raid cards. Lots of people bought them because they were cheap.

Its the consensus of opinion among data center hardcore types that SW RAID is inherently dangerous and unreliable because if your CPU locks up, there goes your RAID controller (and your drive integrity). For them HW RAID is the only acceptable solution as the dedicated controller is immune to CPU lockup type errors.

Its also important to understand the difference between RAID types and where to use them. Here are some basic rules that eeryone seems to agree on:

First, RAID should be used for data storage, not for the OS itself.

Second, there are RAID implementations for performance that are inherently dangerous, like RAID 0. Think Russian Roulette. It may be exciting at first, but if you play long enough the end result is always a disaster.

Third, RAID 5 does provide data integrity through parity-- so if a drive fails the array can continue to work while the drive is being replaced. Of course, the time requirement for rebuilding an array can become a practical problem now that drives are getting to be so big. There's also the theoretical problem with any RAID implementation now that drives are becoming so big.

In the big scheme of things, there are plenty of good ways to achieve a good backup solution without requiring a RAID installation. More on that later.

Here's a good article on Wikipedia if you'd like some further reading:

RAID - Wikipedia, the free encyclopedia

[bob p]

SAMBA, NAS, Backups

I have a house that's full of PCs, so I've already addressed some of those concerns about data centralization and backup. I did it on the cheap, using old hardware that I had lying around, so this type of solution may or may not appeal to you. If you don't want to build a box there are plenty of turnkey NAS solutions out there.

Back in the days when I did Linux development I built a huge compiling farm in my basement. I was publishing my own distribution for a while, which required that I build the entire OS from source code, sometimes several times a day. My basement had so many PCs running that it looked like a data center.

To centralize data for the Windows boxes, I took an approach that sounds a lot like what Steve did -- I took an old PC and stuffed it full of the biggest drives I could find, and I built a centralized file server for the desktop LAN using linux, samba (SMB) and rsync.

To aid in the data centralization paradigm, none of my windows boxes house their "My Documents" folders locally on the Windows box. The "My Documents" folder icon actually points to a directory on the SMB file server. Because SMB is a Windows protocol, the setup looks transparent to the user -- the linux SMB file server appears as a shared drive in the Windows user space. Its very easy for the user to re-map the location of My Documents to point to a Windows network drive.

By moving all of the data off of the Win boxes, any of the workstations could lose their drives (or have the OS intentionally replaced with the linux flavor of the month) without the risk of data loss -- because all of the "data" for all of the boxes is stored in a centralized location on a dedicated file server. To prevent data loss the drives on the dedicated file server get backed up automatically to a second set of hard disks that acts as a shadow copy. This is done using Rsync via a cron job.

In Windows speak, this means that all of this has been happening automatically, via batch files that are automatically executed according to a clock schedule.

This server was built using old throwaway hardware -- this SMB server runs on linux. It was built on a P2-350 with 384 MB of RAM. The only investment that I had to make was in the drives. This box has been rock solid -- its been running 365/24/7 for the past 7 years without a hiccup. I'm finally thinking about rebuilding it now just because I want to add more storage space. (As another example of how to leverage old hardware: I built an SPI-compliant firewall/router on a 486-66/DX2. It stops all of the exploits that get through my D-Link appliance firewall/router.)

The Modern buzzword: NAS

NAS is an acronym for Network Appliance Storage. Local LAN file servers and backup systems have come a long way in the past 5-10 years. The term "NAS" wasn't around in the home market 10 years ago. If you were building an NAS-type system 10 years ago, you were a trailblazer. The first NAS-type appliance that I can even remember was the "Yellow Box" that first came along in around 2005. Today everyone makes NAS boxes -- you can buy off-the-shelf appliances at Best Buy. Today DIY'ing a NAS box is very popular -- there are even NAS distributions that automate the process of DIY, so you don't have to be a linux geek to build your system.

At one end of the spectrum for home-NAS setups, you've got NAS applications like Windows Home Server. It costs money and requires modern hardware. I think its minimum requirements are a 1GHz processor and 1 GB of RAM.

At the other end of the spectrum you have the "free" solutions based on linux and BSD. They will run on just about any old box that you might have lying around. As an example, FreeNAS requires 128 MB of RAM and will run on a pentium class machine.

If you have an old box lying around and you don't want to beat your head against the wall learning to use linux, you might want to try out something like FreeNAS.

FreeNAS [FreeNAS]

FreeNAS is a packaged BSD distribution that's optimized for the installation of turnkey NAS deployments. It has a GUI setup interface, so you don't have to learn all of the linux command line syntax. Because its a BSD distribution, it supports ZFS out of the box. If you're interested, "two smart guys" did a youtube video on how to do an install. Looks pretty easy.

YouTube - Step by Step How to Make a FreeNAS Box


Remember: NAS / BACKUP is a 2-Stage Solution

If you want to DIY, bear in mind that there are two stages to building a viable NAS backup system. The first stage is to build out the hardware, so you have a networked file server that's capable of doing the data storage for you. The second stage is to find a viable backup solution (software) that runs on top of it.

Once you've got the sserver built, the software solution can be something simple. If you keep your data locally on the Windows boxes' hard drives, you can just use Windows Backup across the network. Or you can store the data on the network file server, which actually runs linux, and fool the Windows boxes into thinking they're talking to another windows box. Then you can do automated backups using RSYNC in cron jobs. If you want to get more sophisticated, there are (free) enterprise level solutions out there like AMANDA and BACULA. Some of them will require that you have a dedicated backup server, and that your run backup clients on each of your workstations. The client/server apps then work together to do the backups across the network. What you choose essentially boils down to how complicated of a system you want/need.

Someone mentioned Acronis. I routinely use True Image to create image files of my Windows boxes' hard disks, just to streamline the process of re-creating a fresh install. The software automatically stores the image file on the SAMBA file server on my LAN. That way if the PC configuration ever needs to be restored to bare metal, I just boot from the Acronis CD, pull the image down across the LAN, and restore the image to the new hard disk.

I'm sure that if you dig though your pile of CDs you'll find True Image 9.0. With a NAS box, that software will take you a long way. True Image works so well that Acronis is now marketing an enterprise-level BackupRecovery program that's based on True Image.

[Steve A.]

Steve, as I read this thread I can't help but think we've had this conversation before... either that or I'm just suffering from a case of deja vu...

Bob:

Thanks for sharing your encyclopedic knowledge with us! All of the talk about losing data and drives (and my crappy backup routines!) are the same old same old, but RAID was something entirely new to me!

One question about RAID 0 (I believe this was hardware RAID): the recovery tech said that if it was the controller that was bad you could not just replace it as you would if it was a IDE or SATA controller. Is that true or was he feeding me a line of BS?

When he explained RAID 0 to me, it sounded like you were increasing the chances of total data loss by a factor of at least 4. You get the improved throughput but unless you have a comprehensive backup solution in place you are courting danger.

I have run into a problem with backing up data automatically to an external drive with Win XP: the drive letter can change depending on what other external drives have been plugged into the USB bus; the internal drive letters can be set in stone. With the drive letters changed the backup will be written to a different drive or skipped altogether if there is no such letter.

Good hearing from you!

Steve Ahola

P.S. My main solution for data backup has been based on the concept of "be fruitful and multiply"... In the normal course of events I would end up copying my files to first one drive and then another until I had maybe 6 or 8 copies of the important files scattered among my various drives (I have run out letters in the alphabet!) Of course that is not a very efficient use of disk space plus it can be a real bear trying to locate the other copies in case I screw one up. The #1 reason that I lose a file is because I accidentally overwrite it; I'll be tired or distracted and will forget to save the edited file under a new name.

[Steve A.]

SAMBA, NAS, Backups

I have a house that's full of PCs, so I've already addressed some of those concerns about data centralization and backup. I did it on the cheap, using old hardware that I had lying around, so this type of solution may or may not appeal to you. If you don't want to build a box there are plenty of turnkey NAS solutions out there...

Bob:

I was just reading an ad for NAS hard drive enclosures and one of the selling points was that you could access the data in these drives anywhere that you had internet access. Can you point me to a few links that would explain how this works (I snoogled around a little bit and found nothing)? It seems to me that if you are accessing the NAS drives through the internet, you could just as easily access the drives connected to your computer. What am I missing here?

Thanks!

Steve Ahola

[defaced]

I have run into a problem with backing up data automatically to an external drive with Win XP: the drive letter can change depending on what other external drives have been plugged into the USB bus; the internal drive letters can be set in stone. With the drive letters changed the backup will be written to a different drive or skipped altogether if there is no such letter.

I don't have a drive with me to verify, but I seem to remember bumping the drive letter up, say to like M, so that everything under it can shuffle, but when I plug that particular portable device in, it is recognized as M. It's been a while since I've done this, so I very well may be mis-remembering.

[Steve A.]

I don't have a drive with me to verify, but I seem to remember bumping the drive letter up, say to like M, so that everything under it can shuffle, but when I plug that particular portable device in, it is recognized as M. It's been a while since I've done this, so I very well may be mis-remembering.

Maybe you were doing something I'm not because I just pulled a marathon session fixing my damned computer and was running it with various combinations of no external hard drives up to 10 (I have more but usually plug in no more than 10 at a time).

I have assigned drive letters to all of my drives with PartitionMagic and they usually stick. However if I reboot with other removable drives connected to the USB bus (like my iPod) if that other drive is recognized first it seems to take the first available letter, even if I had that assigned to one of my "real" external drives.

While I haven't seen that happening very often, I did run into an interesting situation when I was finally fixing this computer that I have been complaining about for a year or two. There was a serious problem with the second drive on my boot disc (F: ) so I deleted it. Actually I had been using a partition on my second bank of 4 SATA drives assigned to F: and everything worked just fine. But when I deleted the real drive F: all hell broke loose. My G: drive would boot up as F:, my J: drive would boot up as G: and my L: drive would boot up as J:. Further down the alphabet the rest of the drives kept their original letters (go figure!)

Actually I ran into the scenario I described in the post you quoted working on a friend's computer last summer and I had not installed (and used!) PartitionMagic so that might explain why the ext hd drive letter was unpredictable depending on whether she had her phone or MP3 player plugged into the USB bus.

Speaking of PartitionMagic, for a free alternative I strongly recommend EASEUS Partition Master 4.1.1 Home Edition. (Most of the "free" hard drive utilities are just a tease hoping that you will buy the real program- if they aren't out and out scams.) It doesn't have all of the bells and whistles of PartitionMagic but what it has it does well.


Steve Ahola

P.S. On Friday I finally was able to run CHKDSK on my boot drive (something I have not been able to do for a few years). But it took until Sunday to solve the problem installing the XP SP3 upgrade- I kept getting error messages saying that it did not have proper access and that it was unable to unlock the file that the SP3 upgrade was trying to edit.

On Sunday I tried the "shotgun approach". I was unable to turn NAV off in "Services" so I decided to uninstall it. (This was on a copy I had just made of my boot drive so I could always go back to the original disk if I screwed up.) I also uninstalled a lot of programs that I really didn't need. The final thing (possibly the item that actually made the difference) was pulling up the properties menu for the EXE file and seeing that the file might be blocked (with a button to unblock it). Whatever I did the SP3 install was finally successful. When it fails you get this charming error message from Windows (Thanks for screwing up my computer, Bill! It was your damned company that kept telling me to install the upgrade without bothering to check that it did have access to edit the registry.):

[bob p]

One question about RAID 0 (I believe this was hardware RAID): the recovery tech said that if it was the controller that was bad you could not just replace it as you would if it was a IDE or SATA controller. Is that true or was he feeding me a line of BS?

I've never heard that claim before, and I've never run into that sort of problem... but then I've only used the top tier name-brand controllers like 3ware, LSI and Adaptec. Even so, I'm inclined to believe its just BS -- think about it -- who in the world would consider putting important data on a drive array if its not possible to recover the data if the controller should fail and need to be replaced??? Nobody!

[bob p]

Bob:

I was just reading an ad for NAS hard drive enclosures and one of the selling points was that you could access the data in these drives anywhere that you had internet access. Can you point me to a few links that would explain how this works (I snoogled around a little bit and found nothing)? It seems to me that if you are accessing the NAS drives through the internet, you could just as easily access the drives connected to your computer. What am I missing here?

When it comes to the decision of the NAS Box vs DIY, think about this:

Anything that these NAS-in-a Box devices can do can be accomplished with an open-source (freeware) solution if you're willing to invest some time and effort. Actually, those NAS boxes are probably running linux or BSD internally, and they're just pre-configured to provide certain features.

Which works best in any particualr application depends on your budget, your free time, whether a NAS box will do everything that you want it to do, and your willingness to start tweaking if you need features that the NAS box doesn't offer.

If you're the kind of person who wants to buy a turnkey solution, the kind of thing where you'll have basic functionality when you open the box, then one of the NAS appliances might be for you.

OTOH ii you have additional demands that go beyond what the NAS box offers, and you're willing to learn a little bit of 'nix and do some tweaking along the way, then maybe one of the DIY solutions would be better for you. If you're willing to learn to use Linux, then you can duplicate ANY of the features that the NAS boxes offer. In fact, you can make the linux box do more than any NAS appliance is capable of doing.

Somewhere in the middle between these two extremes are the software distributions that are geared towards the DIY crowd -- like Windows Home Server and FreeNAS.

So in the big scheme of things, its all about how much ttime and effort you want to put into the project. If you're cramped for time and you need something fast, buy a NAS box. OTOH, if you've got a bunch of hardware sitting around, you've got some free time, and you'd like to learn along the way, DIY would be a great way to learn some very useful skills.

If you wanted to go the DIY route with Linux, you could start off with a small project (like building a SAMBA server), and add features to it later on (like remote access) as you have more free time and as your needs change.

[defaced]

I have assigned drive letters to all of my drives with PartitionMagic and they usually stick. However if I reboot with other removable drives connected to the USB bus (like my iPod) if that other drive is recognized first it seems to take the first available letter, even if I had that assigned to one of my "real" external drives.

I do all of my drive letter manipulation through the Administrative Tools in XP. Once inside Administrative Too,s it's Computer Management -> Storage -> Disk Management. The last time I used partition magic I ended up wiping my system drive. It was my fault because I didn't know you shouldn't partition an OS partition, but because of that I now prefer to keep physical drives the same as logical drives.

[bob p]

I was just reading an ad for NAS hard drive enclosures and one of the selling points was that you could access the data in these drives anywhere that you had internet access. Can you point me to a few links that would explain how this works (I snoogled around a little bit and found nothing)? It seems to me that if you are accessing the NAS drives through the internet, you could just as easily access the drives connected to your computer. What am I missing here?

There are a number of different server applications that provide different functionality. In the server world these applications are known ans "services". Which services you need to install depend upon what you're trying to accomplish.

SAMBA is the linux service that is used to provide an SMB server for a Windows network. Windows "drives" or "folders" that are shared on a Samba server are known as "shares." So if all that you want to do is to share files on a private network, all that you need is a SAMBA server.

If you want to access the data remotely, then the situation becomes a lot more complicated. First of all, there's the security concern -- on a private LAN security isn't too big of a deal -- you can establish security by limiting people's access to the wires. Once you hook up to the internet you have to worry about limiting people's access to your data. That's a big concern and it makes the discussion really complicated. That's a BIG problem and I'm going to sidestep it for now.

If you just want to make your data downloadable online, there are simple services that you can run to make the files accessible for download, like FTP (file transfer protocol) servers and HTTP (hypertext transfer protocol) servers. These

If you want to be able to directly access the LAN from the outside world, that can be as simple as running a program called SSH that provides a secure shell connection between PCs. On a linux box it provides a basic command line functionality from anywhere in the outside world.

Going one step further are VPNs, or Virtual Private Networks. These are more complicated setups, that require server software at home and client software on the remote machine. They allow a remote log-on to your system and they use encryption in the packets that are transmitted between the client and the server. This creates a secure "tunnel" from one point to the other that emulates a local LAN connection. Once the data is encrypted, its transmitted over the internet as if you've got your own private network running across a public network as a backbone. Its not really a private network, since you're using the internet as a backbone. Because the data is encrypted people call it a "virtual" private network.

Essentially then, all of these different "services" are all implemented the same way -- by installing a new program on one of your computers on your home LAN. These services typically run on different port numbers on your computer and transfer data across the LAN. The only difference between doing this at home and letting someone access the data via the internet involves implementing port forwarding on your router, so that the data packets can be routed to/from the outside world (WAN) instead of being confined to the local area network behind your firewall (LAN).

You may not realize it, but you've already got experience implementing this sort of thing, because you're running bittorrent on your PC. You installed the server application (bittorrent), you programmed your router to allow port forwarding to the right PC on your LAN, and you gave other people permission to access your box. Providing remote access to your NAS box would be implemented the same way.

With a good OS and the right apps, there's nothing that you can't do. As an example, my niece built a streaming audio server at home, so that she can provide streaming audio to herself while she's at work. With a music collection like yours, you could start an internet radio station.