John, I have to admit that managing databases..as well as protecting them, gives me gas! I really worry about these things, as when they go down, it creates chaos.
We actually have our system drives mirroring to other internal drives on the same server. These are hot, and in the case of minor problems, we can drag files back in place from backups pretty easy. We also back up all of our system, php and Apache files to a removable drive that creates bit-for-bit copies of the entire volume. These are actually bootable volumes if we need to make one of them a boot drive and get up and running.
These are normal drives outside of the storage Raid. Our RAID is a level 5 mirroring to an entirely different, but identical RAID 5 array.
I have been dong this since our first file server was an 8-foot rack filled full of Seagate Barracuda 500 Mb drives....faulty ones at that. After about 9 months they started failing one after another. At one point the failures were happening fast enough that even a single parity drive was not enough. We couldn't get replacement drives fast enough. I can't remember the costs (showing my age here) but they were price enough that we didn't keep any more than we needed on hand. You're right though, to swap a 500 Mb drive and do a rebuild was a pretty simple matter...much less time than a 1.5 or 2 TB drive. Seagate finally acknowledged the issue, and paid for the replacement of all of our drives for 1.5 GB drives We thought we had died and gone to heaven!
Currently our RAID is somewhere around 80TB. Getting this in my pocket for an off site is getting more challenging. We are now backing up to hard drives as there is not as much of a cost advantage to tapes as there once was. We have used DAT, DLT, AIT AIT II AIT III, and most recently with an AIT III jukebox loading barcoded tapes. Expensive and really a pain.
Hard drives are much easier. We are looking for methods to backup entire over the web to another identical server lying in a different building...different city for that matter. Unfortunately this takes huge bandwidth, but we're getting closer to getting this done.
Thanks a bunch for your input.