The Modern IT Environment
PART 1
The modern IT environment is terrifying. There’s no other way to put it. It used to be that all you had to do was keep your servers online with a reliable solution, and run antivirus (AV) and you were 85-90% safe. Most businesses didn’t really have to worry much about hackers because most businesses didn’t contain data that could easily make someone money. Most attacks in the SMB realm were from disgruntled individuals or random viruses.
Today, however, zero day vulnerabilities are coming out every week and the news is filled with stories of hospital chains, school districts, and businesses being taken out by malicious actors (hackers, in common parlance). Now that data ransoms have taken off – and the hackers are getting paid from it – EVERYONE is a target. Small businesses may not be able to pay as large of a ransom, but they’re also usually easier to breech, which means even a small business with no “valuable” information can make someone money.
The world’s on fire. Security experts have responded. There are loads of amazing tools now, SIEM, EDR, fancy MFA, and all kinds of different alphabet soup names to help keep your environment safe, and these tools are GREAT. However, with the sheer number and frequency of zero day exploits and the scope of today’s modern IT environment, there’s always some hole popping up somewhere. A former co-worker regularly said “there’s nothing more terrifying in IT than a bored teenager with a laptop.” If someone is determined to get in your environment, they will.
Read the incredible story of Stuxnet if you don’t believe me.
What More Can You Do?
So what are we to do? The answer is defense in depth. It’s not enough to only secure your edge, or only protect your endpoints, or do extensive monitoring – you have to protect yourself at every level. We have numerous other blogs on that topic here. On top of that, you also have to prepare for that worst case scenario. What happens WHEN the enemy breaches your layers of defense? Are you prepared with multiple levels of recovery? Recovery is actually a layer to defense in depth.
Let’s walk through a recent example we got called into:
The company had a (what seemed to be) secure, modern IT environment – advanced firewalls, almost nothing published to the internet, modern AV and DNS filtering for all workstations, backups at their datacenter, and a full disaster recovery (DR) site with 100% of their environment prepped to be turned on at a moment’s notice.
There was a hole in their armor though: the password manager that their IT department used had one user setup that didn’t require MFA. An intelligent actor – not ransomware – was able to find this one user account, brute force a password, and get into the password manager. Once they were in the password manager, they were able to bypass large amounts of the security ecosystem to access the environment, and then patiently do discovery and write scripts – acting as an IT admin where their actions weren’t considered abnormal.
All of this happened in what was perceived as a secure, modern IT environment.
At 2 a.m. on a random Wednesday, they executed the script that disabled parts of AV, logged into the backup server (unique 32-character password, not on the domain), deleted backups, logged into VMware with a different, unique 32-character non-AD password and encrypted VMs, then logged into the completely separate vCenter on a separate network for DR with another unique 32-character non-AD password, and encrypted the VMs there. Within 30 minutes, everything was gone. All three copies of data on two different medias, with one being offsite – the ‘3-2-1 rule’ was followed but failed.
The ‘3-2-1 rule,’ popularized by Veeam (but originally written by a photographer), was made for an era where we were worried about hardware failures and natural disasters. That’s not the environment we live in now though – now we’re worried about deliberate malicious actions in addition to the environment. This is why Veeam has recently amended the ‘3-2-1 rule’ to be ‘3-2-1-1-0’ with three copies of data on two different medias with one being offsite, one being immutable or air gapped, and automated testing of backups showing zero errors.
If this company had simply had an immutable copy of their backups, then they would have saved themselves weeks of downtime rebuilding everything. Here is yet another newer step in keeping up in today’s definition of a modern IT environment.
Hopefully this helped to highlight the needs for additional protections above what have traditionally been recommended. Our next blog will start to tackle the final 1 in the ‘3-2-1-1-0 rule’ and start talking about how to protect a copy of backups through either air gaps or immutability.
