Hard drive backups

[bob p]

I have a problem that is a lot like Steve's -- we both have data that's stored on lots of hard disks. the difference is that while lots of his discs are offline, i've got all of them online. my basement is like a data center. i've got almost a dozen PCs, acting as dedicated firewall/routers, workstations, file servers, and backup servers. the backup servers and file servers are stuffed to the gills with discs, and i have multiple file servers just because the storage density of the older media is so low. this system really needs to be revamped and consolidated. now that 2 TB drives are available, i think its possible to build a 10 TB storage box using LVM and put all of the file server eggs in one basket.

i have to admit that my storage system has grown incrementally, and that it looks like a system that's gone through a lot of add-ons rather than being designed from the ground-up for its current role. i've got several terabytes of storage scattered across several file servers using drives that are small by today's standards. this is primarily an artifact of having spent years collecting digital media on whatever size drives were available at the time, and adding new drives and a new PC to the LAN as i ran out of space. what i really need to do is to replace a lot of old, smaller drives with a handful of new, high density drives, and eliminate some of the CPUs.

right now i'm in the process of gradually consolidating to a smaller number of big drives running on fewer boxes. to do this right really requires considerations about scalability. in the linux world, this means at least using a tool like LVM, or taking it to a higher level, a distributed file system like GFS or Gluster.


one of the endpoints i'm looking toward is consolidating all of the audio file data storage so that its available to any one of a number of iTunes compatible clients via an mt-daap compliant back-end server. that way i'll be able to access the entire collection across the LAN using an iTunes type of front end. optimally, i'd like to have a wireless tablet PC (like a Slate or an iPad) running as the front end to the home stereo system, so that i can play anything in the library on-demand. well, that's the goal, anyway...

Steve, I think you've got a very extensive collection of digital audio data. How many GB or TB do you have in total? I think that you could really benefit from the type of setup I'm thinking about.

[R.G.]

...as an example, Bank of America has cabana boys -- they bill under the name of International Business Machines...

Yeah. I spent the majority of my adult life working for that pool-boy consortium.

b. The HCLs (hardware comparability lists) for "recipe" installs are often pretty short, so you get locked into a short list of options.

I found, eventually and by luck, that there are some good answers that are cheap and available. I'd be happy to offer any advice that my (thin) experience here can give. Actually, I already did. See my home server blog, where I'm putting out pithy comments about my experience. Just got my second NAS up and running, with no drama.

i just built another file server for the home LAN, and i guess its a NAS box according to current terminology. ... someday i really want to build a box that supports ZFS, just to try it out.

If you're that capable a hacker, you may want to just dive in. I can tell you at least one set of hardware that will run opensolaris/zfs with no gotchas. It's this: ASUS M4A785TD-EVO mobo, Any of the AMD "AM3" processors, DDR3 memory sticks, and one of several PCIe SATA adapter cards to fluff up the number of SATA disks you can use. The Norco RPC-470 is a rackmount case that has space, power, and cooling that supports ten 3.5" drives, three 5.25" drives, which can be another five 3.5" drives if you want, and that's inside the case. Standard ATX power fits. It has five internal fans.It could hold up to 30TB inside the case if you wanted to fully expand it that way.

On the other side, you could put together a minimal Asus/AMD system to play with out of a large number of mobos. My second server has 400G in two old 200G drives, and boots from a 2GB compact flash card. It's a neat toy.

The big brother is fully headless, both SSH and VNC remote GUI terminal with only what's in the basic install CD from Opensolaris. Anyway, send message or email if you want more details.

now that 2 TB drives are available, i think its possible to build a 10 TB storage box using LVM and put all of the file server eggs in one basket.

As I mentioned, the box I have would go to 30TB inside the box if I wanted to buy the storage. And zfs was **designed** to make managing disk storage trivial. "Volume manager" work is essentially designed out, as far as managing disks, quotas, migrating data and so on.

If you're really wanting space, the Norco RPC-4020 has 20 hot swappable bays in a rackmount case for $320. That's, what, 40TB, or 20TB mirrored?

i've got several terabytes of storage scattered across several file servers using drives that are small by today's standards. this is primarily an artifact of having spent years collecting digital media on whatever size drives were available at the time, and adding new drives and a new PC to the LAN as i ran out of space. what i really need to do is to replace a lot of old, smaller drives with a handful of new, high density drives, and eliminate some of the CPUs

I suspect you could sell the smaller servers, even at low used-computer prices, and pay for the big server. As I said, I'm happy to share whatever my experience can provide.

[bob p]

thanks for the tips, RG. as it turns out, I had been looking at the Asus M4A785-M motherboard. Its already on my short list for precisely the reasons you've mentioned on your blog. I really appreciate your feedback on it. I like the fact that it has 6 SATA headers on the board. It should be a great starting point.

My two main file servers are actually real file servers -- older Intel server boards with dual P3 and 1 GB of ECC. They're running linux, primarily to serve up SMB shares to Win/Lin clients. They hold a mix of ATA and SCSI disks. Their weakest link is the ATA drive interface. I've been thinking about upgrading them by adding PCI SATA adapter cards in them, vs. just building new machines outright. Of course, just updating the disk subsystems would be the cheapest way to go for the time being. I still haven't decided which way I'm going to go.

I have to sheepishly admit that I've actually got a few old desktop P1, P2 and P3 machines that have been pressed into service as well. To me CPU speed hasn't been the most important factor. As you know a slow CPU can still serve up data faster than the other bottlenecks in the system, like drives and the network interface.

The software side of things won't be at all difficult, as long as I stick with something that's familiar. I can set up a linux file server from bare metal with my eyes closed. The only thing that's been keeping me from testing the waters with BSD or Solaris has been the time commitment that it will take to get up to speed on a new platform. For me the path of least resistance has been to stick with/build on what I already know... at least until I have the spare time to learn something new.

I'll check out the rest of your blog and see what I can learn from your experience. Right now I'm getting called for dinner...

thx.

[R.G.]

I've been thinking about upgrading them by adding PCI SATA adapter cards in them, vs. just building new machines outright. Of course, just updating the disk subsystems would be the cheapest way to go for the time being. I still haven't decided which way I'm going to go.

I have to sheepishly admit that I've actually got a few old desktop P1, P2 and P3 machines that have been pressed into service as well. To me CPU speed hasn't been the most important factor. As you know a slow CPU can still serve up data faster than the other bottlenecks in the system, like drives and the network interface.

The first job I had inside the belly of the beast was designing power supplies. I think the first thing you should do is get on a path to a machine with new power supplies. Of the whole mess - except possibly the fans - the power supplies are under the greatest stress. A power supply over 5 years old may work fine, but it's on the far end of the bathtub curve.

I put a new 80+ certified power supply in each new server; I got lucky and got a Corsair 400 for $39, no shipping, on a sale at newegg. But even if you have to pay for it, it's worth it. The power supply is a single point of failure for your data. Let it go mad and your disks are toast.

Actually, using a bit of engineering econ, 1kwHr of electricity is $0.12 to me. By using many older systems and higher power disk drives, you're probably paying for a new server every year in electricity to feed the machines, and then with electricity to air condition the wasted heat out of the house. It's modestly stunning the first time you do a "power is money" comparison on what a new server can do.

New "ordinary" 3.5" drives pull about 6W when idle or under modest activity. Older 3.5" drives pull 8-16W. If you got one of the bad ones, you're spending 10w-Hr per power-on hour. There are 8760 hours per year, so that's 87kWhr extra per disk drive and at $0.12 per kwHr, $10.50 per year per disk drive. Could be less if you used later disk drives. If you use 100G drives, there are 10 of those in 1TB, so that's $100.50 payback in a year on just the electricity costs if you move to a 1TB file. Not to mention that the 1TB is at the near end of the bathtub curve for failures and the 10x100G are at the far end. And I just got two 1.5TB files from Frys for $80 each on sale. Baby server is getting 3TB to play with.

2.5" drives put typically 2W max for 500GB, but cost about twice as much per bit. The power/cost tradeoff is much near even for them.

The above analysis ignores the power spent to run the rest of the multiple systems. Disk drives aren't the only power user.

The dictum to not put all your eggs in one basket has a lesser-known alternative: put all your eggs in one basket, then watch that basket carefully.

The only thing that's been keeping me from testing the waters with BSD or Solaris has been the time commitment that it will take to get up to speed on a new platform. For me the path of least resistance has been to stick with/build on what I already know... at least until I have the spare time to learn something new.

I last managed a *nix system in 1994 before starting Opensolaris. You're going to have no trouble at all, other than looking up the commands for OS versus Linux.

[bob p]

Yes, power is money and I'd really like to shrink the power bills. It will make sense to start the consolidation process sooner rather than later.

Is power consumption the reason that you wanted to boot your OS from compact flash? Just to eliminate one spinning disk?

Although lots of people seem to be using USB memory and CF sticks now now, I had concerns about the lifespan of flash memory. It wears out as its used. I'd think that a CF disk or a USB flash stick would wear out pretty quickly with all of logging that typical *nix systems does in /var/log/whatever.

[R.G.]

Is power consumption the reason that you wanted to boot your OS from compact flash? Just to eliminate one spinning disk?

Yes. Well, that and to see if I could do it. That's actually booting EON, which is a version of Opensolaris which has been stripped down to make a "NAS" appliance version. You can boot the whole Opensolaris unmodified in 9G, maybe 8G, so an 8G or 16G compact flash card would work. I used a 2G.

The ZFS operation in EON is the same as the full-gallon Opensolaris, as I understand it.

In the big brother NAS I used two 2.5" 40G notebook drives. They're IDE leftovers, operated in mirror. I used them so I could use the one IDE port on the MB in that one and not have to put in another PCIe SATA card. There's a lot more details buried under here that would bore most people at this forum to tears.

Although lots of people seem to be using USB memory and CF sticks now now, I had concerns about the lifespan of flash memory. It wears out as its used. I'd think that a CF disk or a USB flash stick would wear out pretty quickly with all of logging that typical *nix systems does in /var/log/whatever.

Yes. However - you knew that was coming, right? - CF is good for between 10K and 100K writes per location, depending. And internal to the cards, they do "wear levelling" to spread things over the whole drive, not just one place all the time. So "pretty quickly" varies.

And you can move the logs from /var/log/stuffandsuch to a vdev on the zfs disks. It doesn't hurt much to make a 2G vdev which is only the OS logs.

[bob p]

so how did things go when you were working on staggered drive spin-up? my very limited knowledge of the implementation for SATA is that your hardware drive controller has to support the feature, in addition to the OS. do you know anything about the level of support provided by the Asus M4A785-M? I've looked at the manual but I couldn't find anything. I'm guessing that you can only tell by browsing the BIOS.

i also wanted to as if that board is capable of booting from a USB flash drive. it wasn't really covered well in the manual.

thx.

[R.G.]

so how did things go when you were working on staggered drive spin-up? my very limited knowledge of the implementation for SATA is that your hardware drive controller has to support the feature, in addition to the OS.

Still working that one. My "contribution" so far is that Seagate drives require pin 11 on the SATA power cable to be open to start with. If it's grounded, the drive does not do staggered spinup, period. Most SATA power connectors ground pins 10, 11, and 12 by default. Pin 12 must be grounded, it's part of hot plugging and is a bit longer than the majority. As a result, you have to commit surgery on either the drive or the connector to free up pin 11 for Seagate brand. This is different from brand to brand. WD has a jumper. Don't know about Hitachi and Samsung. Won't buy Fujitsu.

But I think that yes, the disk controller has to do SSU, and the BIOS has to not forbid it. I did a quick search and didn't find a clear answer as to whether the M4A785M (the one you're looking at) or the M4A785TD-V EVO (the one I have) support it or not.

The correct answer to staggered spinup comes in two flavors - (a) gotta have it and (b) don't care.

If your answer is (b), then it's at least as cheap in money terms just to get a bigger peak-power power supply. Staggered spinup, is useful primarily for letting you buy a smaller, and presumably cheaper and more efficient power supply. Power supplies are most efficient at nearly full load. The constant losses are spread over a bigger power out. Smaller is sometimes cheaper.

When I bought power supplies, I researched reliability and came up with several types, the Corsair CMPSU-400CX being one of them. When there was a sale getting me these for $39, I jumped. The 400 is nominally too big for my needs, because it has enough +12 to spin up more drives than I need simultaneously, about 10 if I calculate right, while still starting the motherboard. It's available for $50 at newegg now, but there's a $10 mail in on it. I ignore mail ins - they have been made hard enough to actually get that I assume they don't exist. Getting a free-shipping advertising gimmick is worth more money. In any case, the price was low enough on a good-reputation power supply, I didn't keep flogging google.

If you gotta have it, the trick devolves to finding out whether the motherboard does it natively, or to using add-in controllers which uncategorically do. You get into expensive adapters quickly. The cheapest disk controller that I found that uncategorically says it supports it is the HighPoint RocketRAID 2300, about $120.

do you know anything about the level of support provided by the Asus M4A785-M? I've looked at the manual but I couldn't find anything. I'm guessing that you can only tell by browsing the BIOS.

Maybe. My criteria is that if it supports it, it's easy to find. Probably no staggered spinup native to the mobo. My solution was "I got the power supply to handle it, it's cheaper that way than buying a high end disk controller, I'm going to lose some watts/$ this way, but (shrug) sometimes you really can't get everything for a cheap price.

In spite of what my writing looks like, I do have a practical side.

i also wanted to as if that board is capable of booting from a USB flash drive. it wasn't really covered well in the manual.

The ...TD-V EVO variant does. There are BIOS options to set it. I thought "oh, I'll just go download the manual and see." but the Asus web site is not providing me with a download at the moment.

I'd guess yes, but the mobo manual will tell. You could get the M4A785TC-V EVO. I know that one does. You pay for it in terms of five internal SATA ports instead of six, but there is an esata port that can be refunneled back inside if you're determined, or could be used to connect to a bigger external box of disks.

[bob p]

Still working that one. My "contribution" so far is that Seagate drives require pin 11 on the SATA power cable to be open to start with. If it's grounded, the drive does not do staggered spinup, period. Most SATA power connectors ground pins 10, 11, and 12 by default. Pin 12 must be grounded, it's part of hot plugging and is a bit longer than the majority. As a result, you have to commit surgery on either the drive or the connector to free up pin 11 for Seagate brand. This is different from brand to brand. WD has a jumper. Don't know about Hitachi and Samsung. Won't buy Fujitsu.

I guess that by "surgery" you mean breaking the connection for Pin 11. Sounds like it would be better to do this on the cable, perhaps by cutting a trace on the connector, rather than by cutting a trace on the drive's connector. Cables are cheap compared to drives.

I wonder if they make "server grade" cables that leave pin 11 open?

But I think that yes, the disk controller has to do SSU, and the BIOS has to not forbid it. I did a quick search and didn't find a clear answer as to whether the M4A785M (the one you're looking at) or the M4A785TD-V EVO (the one I have) support it or not.

Sorry, didn't realize that you had a different board. I was only familiar with the M and not the TD variant. Where did you find it? I can't find info on the TD model via google, or at places like the egg or tiger.

[R.G.]

I guess that by "surgery" you mean breaking the connection for Pin 11. Sounds like it would be better to do this on the cable, perhaps by cutting a trace on the connector, rather than by cutting a trace on the drive's connector. Cables are cheap compared to drives.

I wonder if they make "server grade" cables that leave pin 11 open?

Yes, that's what I meant. Don't know about "server grade" cables. I use X-Acto knives.

Sorry, didn't realize that you had a different board. I was only familiar with the M and not the TD variant. Where did you find it? I can't find info on the TD model via google, or at places like the egg or tiger.

This path: newegg.com > Computer Hardware > Motherboards > AMD Motherboards > ASUS > CPU Socket Type : AM3, and it's toward the end of the page. I checked the box, I have this one:M4A785TD-V EVO. I think that what got me on that one was two PCIe X16 slots on the theory that if I needed expansion, the PCIe X16 would be the higher performance way to add it. Onboard graphics (for bringup), ECC or non-ECC memory supported, 2x PCIe x16, and a lot of stuff I'll turn off like fancy audio and video.

The onboard graphics and onboard NIC are supported under Opensolaris and simply run on installation, which seems to be a big, if hidden issue with some motherboards.

Here is a subtle decision point. Adding disk controllers can cost more than the motherboard easily enough. It makes sense to get over 4 and 6 if you can SATA ports on the mobo. Cheap 2x and 6x SATA controllers exist, but they mostly are limited to SATA 1 and have chancy compatiblity. The big, 8-port cards are $100 and up, and other than the Supermicro PCIx eight port are quite expensive. That's the $100 one. The LSI and Rocketraid ones are $150 to sky's-the-limit for higher performance, staggered spin up, and so on.

Another subtlety: Opensolaris and zfs don't need specialized and more expensive Raid-Edition disks with limited time response on errors like RAID implementations outside zfs do to keep from marking a disk failed when it's just retrying forever.

Another subtlety: You don't need, will have difficulty using, and will need to disable if its there if your added disk controllers have RAID ability. You want Opensolaris to see individual disks and not have RAID stuff in its way when zfs is dealing with disks.

[R ski]

I use an external hard drive to back up, so much easier than DVD,s and the hard drive can run my digital work station projects

Now one of the issues with external hard drive back up, if certain folders such as photos or wave files are organized in some timely fashion, how does one keep the current new data up dated on the external hard drive with simplicity.

The program in windows XP from good ole doss days days still works. What is neat about this command line is only the changed data is updated into there appropriate folders at a click, no need to worry on what goes where.

xcopy c:\lyrics g:\basement\lyrics /d/e/c/h/k/y This example lyrics is the source on C and G drive is the external hard drive, basement is the folder that refers to the basement PC (Mine)\lyrics is the destination folder. The /d/e/c/h/k/y are the switched that take care of the updating status. Write the command in note pad, save the file in a handy place, as long as the doss file names on the directory matches xcopy will work.

[bob p]

Xcopy -- you're right -- i used xcopy A LOT back in my DOS days, and in my early windows days. Today i'm keeping everything that matters on a *nix box, as i find the "granularity" of my control over the box to be much finer. I find that RSYNC is a great tool for keeping shadow copies of data up to date.

[bob p]

This path: newegg.com > Computer Hardware > Motherboards > AMD Motherboards > ASUS > CPU Socket Type : AM3, and it's toward the end of the page. I checked the box, I have this one:M4A785TD-V EVO. I think that what got me on that one was two PCIe X16 slots on the theory that if I needed expansion, the PCIe X16 would be the higher performance way to add it. Onboard graphics (for bringup), ECC or non-ECC memory supported, 2x PCIe x16, and a lot of stuff I'll turn off like fancy audio and video.

Thanks for the path. I did an on-site search for that board but I couldn't find it, though I did find it by drilling down as you recommended. I agree completely, its worth paying a little extra for the board that has the extra high speed slots -- even if the cost involves losing 1 SATA header.

I definitely want something with ~6 SATA headers, for obvious reasons. I didn't like the fact that every board I found with 5-6 SATA headers had a rather sparse supply of expansion slots. The board that you've found is a nice one -- the added expansion slots make it a better fit than the one that I was looking at.

[bob p]

Another subtlety: Opensolaris and zfs don't need specialized and more expensive Raid-Edition disks with limited time response on errors like RAID implementations outside zfs do to keep from marking a disk failed when it's just retrying forever.

I've read about the drive manufacturers taking steps to "cripple" drives via timeouts at the firmware level, to compel customers to buy enterprise-class drives for RAID implementations. I'm not all that enthusiastic about RAID. IMO ZFS takes a better approach to data redundancy.

To me, RAID isn't appropriate for an application that doesn't require fault tolerant instantaneous availability of mission critical data. I'm not doing OLTP and I certainly don't need that kind of fault tolerance for a music server -- my audio files don't change -- they're essentially WORM (write once, read many) data. If the data doesn't change then maintaining real time backups is a waste of energy.

I'm thinking that if I have, say, 10 drives full of music data that doesn't change very often, then what's the point of having another 10 drives running 24/7 as a real-time mirror? That wastes a lot of energy.

A "greener" approach would be to just have a separate backup server that powers itself on via the BIOS clock to perform backups, and then powers itself down when its done. That would cut the power consumption of the hard disk arrays in half. That's significant compared to a 24/7/365 scenario.

From a power standpoint its cheaper still, to have a separate music server that's only switched-on when the stereo is switched-on. There's no sense in having all those drives running 24/7/365 if you're not listening to music round the clock.



It'll be interesting to see how the new B-Tree File System (BTRFS) comes along in Linux. the linux filesystem guys have decided that they want a ZFS type of file system for linux, and since there's the licensing / kernel/userspace problem for porting ZFS to linux, they've decided to build a ZFS equivalent file system. Its not ready for prime time yet, but there is a technology preview available in the new RHEL 6 Beta.

[R.G.]

A "greener" approach would be to just have a separate backup server that powers itself on via the BIOS clock to perform backups, and then powers itself down when its done. That would cut the power consumption of the hard disk arrays in half. That's significant compared to a 24/7/365 scenario.

Yep. That's really what all that "remote management" stuff is about in my blog. I really only need to power the thing up when I want to run a job. My stuff isn't so critical that I can't stand losing a day or so. But losing the entire history, or a month or two is a disaster.

From a power standpoint its cheaper still, to have a separate music server that's only switched-on when the stereo is switched-on. There's no sense in having all those drives running 24/7/365 if you're not listening to music round the clock.

+200

It'll be interesting to see how the new B-Tree File System (BTRFS) comes along in Linux. the linux filesystem guys have decided that they want a ZFS type of file system for linux, and since there's the licensing / kernel/userspace problem for porting ZFS to linux, they've decided to build a ZFS equivalent file system.

I have great hopes for that. Not read for prime time was one of the things that steered me into zfs. I don't love zfs for itself, I want what it does. I hope the BTFS guys can take a hint and make admin easy and trivial. That aspect of zfs is outstanding. The idea that more bits is just more bits, and modelling after virtual memory is great.

There are some issues with zfs' design, notably that you can't reduce a vdev or expand it very easily. But most of the rest is an amazing step in the right direction. Especially since I understand that the source code is tiny by comparison to some other things. 700 LOC comes to mind if I remember right.

I used to manage the department that maintained the file system in AIX. It took some very talented people a lot of time and effort to keep that code set in order. It wasn't under 1kLOC.

For the readers, here's my home server blog: R.G. Keen's Blog.