So what are we to do? The answer is defense in depth. It’s not enough to only secure your edge, or only protect your endpoints, or do extensive monitoring – you have to protect yourself at every level. We have numerous other blogs on that topic here. On top of that, you also have to prepare for that worst case scenario. What happens WHEN the enemy breaches your layers of defense? Are you prepared with multiple levels of recovery? Recovery is actually a layer to defense in depth.
Let’s walk through a recent example we got called into:
The company had a (what seemed to be) secure, modern IT environment – advanced firewalls, almost nothing published to the internet, modern AV and DNS filtering for all workstations, backups at their datacenter, and a full disaster recovery (DR) site with 100% of their environment prepped to be turned on at a moment’s notice.
There was a hole in their armor though: the password manager that their IT department used had one user setup that didn’t require MFA. An intelligent actor – not ransomware – was able to find this one user account, brute force a password, and get into the password manager. Once they were in the password manager, they were able to bypass large amounts of the security ecosystem to access the environment, and then patiently do discovery and write scripts – acting as an IT admin where their actions weren’t considered abnormal.
All of this happened in what was perceived as a secure, modern IT environment.
At 2 a.m. on a random Wednesday, they executed the script that disabled parts of AV, logged into the backup server (unique 32-character password, not on the domain), deleted backups, logged into VMware with a different, unique 32-character non-AD password and encrypted VMs, then logged into the completely separate vCenter on a separate network for DR with another unique 32-character non-AD password, and encrypted the VMs there. Within 30 minutes, everything was gone. All three copies of data on two different medias, with one being offsite – the ‘3-2-1 rule’ was followed but failed.
The ‘3-2-1 rule,’ popularized by Veeam (but originally written by a photographer), was made for an era where we were worried about hardware failures and natural disasters. That’s not the environment we live in now though – now we’re worried about deliberate malicious actions in addition to the environment. This is why Veeam has recently amended the ‘3-2-1 rule’ to be ‘3-2-1-1-0’ with three copies of data on two different medias with one being offsite, one being immutable or air gapped, and automated testing of backups showing zero errors.
If this company had simply had an immutable copy of their backups, then they would have saved themselves weeks of downtime rebuilding everything. Here is yet another newer step in keeping up in today’s definition of a modern IT environment.
Hopefully this helped to highlight the needs for additional protections above what have traditionally been recommended. Our next blog will start to tackle the final 1 in the ‘3-2-1-1-0 rule’ and start talking about how to protect a copy of backups through either air gaps or immutability.