Bob Jones is the IT director for ACME Rocket Company. He considers himself very cutting edge and is already utilizing Veeam to back up his virtual infrastructure.
He built a fairly well-designed strategy for backups and business continuity and is utilizing Veeam to back up to local disks. They utilize a fireproof/waterproof solution called I/O Safe at the ACME Headquarters. This I/O Safe protects ACME Digital Assets from a local disaster such as a fire, flood, etc. Bob built a nice script to copy backups from local disks to the I/O Safe.
ACME also uses an external drive that holds another copy of ACME’s backups. This is one more level of backup that Bob built to feel secure in the event of disaster and the I/O Safe gets destroyed. This drive is part of an offsite backup solution and this drive is rotated out once a week. Like with the I/O Safe, Bob has built a different script to copy backups to this external drive.
One day, Fred Smith – Bob’s boss – asked Bob to test backups in preparation of their annual audit. When he does, Bob finds that the backups on the external devices imported properly but test restores from either device fail with a “corrupt metadata” error message. This is puzzling because the backups from local disks always tested correctly. Veeam works perfectly in and of itself – as usual.
Troubleshooting these results, Bob finds that if he runs his script manually from local disks to either external device, he is able to import and restore successfully with Veeam as expected. Strange. With further research, Bob realizes that he may have inadvertently corrupted some data with his homegrown Backup-Copy scripts.
One of his post-job scripts calls a batch file that started both local-disk-to-other target scripts above simultaneously. Bob theorized that by running both of these scripts at the same time, something, somehow, may have gone wrong.
So, Bob went back to the drawing board to iterate an even better backup and business continuity solution using best of class processes and procedures.
Fables have a hint of truth to them. How often do you test your backups? Sometimes the best laid plans go awry and bad things happen.