IT Horror Stories: The Dead Shall Rise (Rebuilding a Server Back to Life)

rebuilding a serverWe have all heard the saying that the road to hell is paved with good intentions. This sentiment holds true in most things, including the IT business. Please allow me to regale you with one such IT horror story:

It was a dark and stormy night. (After all, don’t all horror stories start this way?) The server in this story happened to live in a local funeral home and everything was as it should have been. The server was shiny and new and the operating system had just been upgraded to the latest and greatest (Server 2012). Backups were performed every single night and the server had all its patches and updates.

Like I said, everything was as it should be…or was it? Unbeknownst to all, in this shiny new system, a dark force grew. It was ever so small at first, but it grew a little darker every day.

This evil force was disk data corruption and it remained undetected.

This server ran so well for so long that there seemed no reason to ever restart it. Several months passed with no restarts or power downs. Day in and day out, the server performed its tireless duties, never missing a beat, until, with a clap of thunder and flash of lighting, the power went out. The UPS struggled to keep the server up but eventually the server shut off from insufficient power.

The next morning was a beautiful and sunny day, but the server was unresponsive. The lights came on when the funeral home staff hit the power button and they heard the reassuring beep. However, in place of the normal blue glow of the monitor there was darkness, pierced only by the words “Cannot find system file”.

The IT director of the funeral home took action. Remembering another IT horror story and its Mirazon heroes, he called in help. The Mirazon engineers worked tirelessly for many hours but the darkness remained: “Cannot find system file.”

As you know, restoring a system can take a while, and in this case, it was impossible because all the disk repair attempts failed. The end users stood around, operations at a standstill. Finally, Mirazon consulted Microsoft and for five hours, the Microsoft engineers tried to repair the corrupted files. In the end, the only thing left to do was rebuild the server. The data was completely corrupted and there was no way to repair it.

At long last, the Mirazon team completed the rebuild and the happy blue monitor glowed on once again.

I know this is a horror story, but I like things to have a happy ending. We were able to bring the server back to life by restoring the data and the client was able to continue working without any data loss. Of course, they had to suffer significant productivity loss for the two days Mirazon spent troubleshooting, repairing, and eventually rebuilding.

While this funeral home was lucky and returned to operations as usual, this isn’t always the case. Every story should have a moral as well: always verify your backups!

Your backup is only as good as the data it contains. Regularly test your backup and make sure you can actually restore from the backup in case of emergency. This funeral home did everything else right, but they did not realize the backup was bad because they had never tested it. History has shown us: an ounce of prevention is worth a pound of cure.

Need help with your backup and disaster recovery system? You’ve come to the right place. Email us or give us a call, 502-240-0404, to learn more!