Before writing the first part to this blog series, I decided to do a Bing search of “backup best practices”. Now, don’t scoff at my use of Bing as my preferred search engine — Microsoft practically pays registered users of Bing Rewards to use their search engine. But I digress…here is the result I got:
Wow. Nearly 8 million results. With all that information available about how to best implement a backup strategy, who needs another blog about what you should be doing to protect your environment? Well, if you haven’t validated your backups by performing one or more test restores, read on.
Let’s talk about my friend, who I’ll call Joe. Joe built a virtual server and backed it up with software optimized for a virtual environment. Success notifications for each backup job filled Joe’s email inbox, so he slept well at night. Days went by, then weeks, then months. Then Joe’s virtual server crashed. So, he turned to his trusty backup solution, launched a restore, and… nothing.
The restore failed! Vendor support to the rescue, right? Wrong. There was nothing they could do. Poor Joe hadn’t tested his backups. If he had, he would have known that data corruption had been introduced into his backup set. Was this the fault of the backup tool? No. The data corruption actually occurred within Windows, and that’s what led to the server crash.
Joe had been backing up corrupt data for months now.
As my ninth grade computer teacher used to say, “Garbage in, garbage out!”
Result number three of the 7,820,000 results referenced above is a Microsoft TechNet article. Here are some of the things the article suggests:
- Use external storage drives that are compatible with your server.
- Use an external hard disk with at least 1.5 times the storage capacity of the items that you want to back up.
- Use multiple external hard disks and rotate them.
- Rotate backup drives on a regular basis.
- Plan your backup strategy carefully for computers with virtual volumes.
Those are all valid recommendations. Missing from the article is what most IT admins overlook – restore testing. In part one of this guide, do an overview of automated restore procedures using a leading backup tool for virtualized environments.
Automated Restore Testing
You’re busy. You’ve got a server upgrade project that you’re managing, a new help desk ticketing system that isn’t working as designed, and an ISP cutover happening in four weeks. You’re also short staffed due to budget cuts. When are you supposed to find time to test your backups? Fortunately, there’s good news: currently available backup software can test your backups for you – automatically.
Before we move on, I have a confession to make. Prior to my coming to work for Mirazon eight years ago, I had done exactly zero restore tests to validate backups. Zero. Granted, backups weren’t my responsibility in my previous role, but I did have to swap backup tapes on occasion. But still…zero? At the risk of looking a little weird, raise your hand if you’re in the same boat.
How does one do automatic restore testing? What backup solutions offer this feature? Just for fun, I headed back over to Bing and performed various searches to see what I could find. I tried several search phrases like “automatic restore testing”, “automatic disaster recovery testing”, “automatic backup restore testing”, but I just didn’t find much. I did find this:
Well said. Enter Veeam SureBackup, a feature included with the Enterprise and Enterprise Plus editions of Veeam Backup and Replication. For the uninitiated, Veeam B & R is a tool specifically designed to backup virtual environments, namely VMware vSphere and Microsoft Hyper-V. As stated by Veeam, SureBackup can “Automatically test and verify every backup. Enterprise and Enterprise Plus editions offer automated recovery verification, including support for application test scripts and bulk VM testing.”
Here’s an example of how this can work:
- Veeam B & R performs a backup of an Active Directory domain controller.
- Veeam then spins up a copy of the virtual machine in an isolated environment from the backup it just created.
- Next, Veeam performs a series of test scripts to see if the restored VM is functioning properly. For a directory server, this would include:
- DNS test
- Domain controller test
- Global catalog test
- If any of the above scripts fail, Veeam sends out a notification to that effect so that action can be taken by the IT admin.
- If all of the above scripts succeed, a successful restore notification gets sent.
- Veeam automatically removes the restored VM after testing is completed.
Impressive, right? Got Veeam yet? Let’s take a closer look at how Veeam does this.
Within Veeam B & R’s SureBackup module, there are three key components. Taken straight from the console, they are:
- The Virtual Lab: A Virtual Lab requires a host on which to run virtual machines (VMs), and a datastore to store disk changes produced while running a VM from a backup file. An isolated virtual lab network is automatically created based on the production network selected to be mirrored in the lab. VMs in the isolated virtual lab network are accessible from the production network through a helper proxy appliance that is automatically configured and deployed to the selected host as part of Virtual Lab creation.
- The Application Group: An Application Group defines VM dependencies by specifying the required boot order of VMs supporting a given application or service. An Application Group typically includes a domain controller, a DNS server, and a DHCP server (unless static IP addresses are used).
- The SureBackup Job: To set up a SureBackup job, select the Application Group containing the core infrastructure services that the VMs to be run in the Virtual Lab are depending on, and specify which backup jobs you want to use. While the SureBackup job runs in recovery verification mode, all VMs from the selected backup jobs are started and verified one by one. When the SureBackup job is initiated by a U-AIR (universal application item recovery) request, only the required VM from the specified backup job(s) is started (in addition to the VMs in the specified Application Group).
Stay tuned for part two of this series, where we will cover each of the SureBackup components in detail.