Your backup and restore process is broken–here’s how to fix it

Almost no company backs up all its critical data and–this is the important part–actually tests that those backups really work.

Backups have always been a thankless task. Backup software is incredibly complex with hundreds of options and a spotty record of actually working. Yet so little training is offered or taken advantage of that most people simply take the defaults and hope for the best.

Let’s be honest. Every time you’ve done a backup restore, even for a single file, and it worked, you breathed a sigh of relief. That’s because you know backup-and-restore events often don’t work. Many of us have had a needed restore fail. Worse, the backup software might indicate success when the job completes, but some default option set since the beginning of time made your backups worthless.

Poor backup testing is killing security

Even through we know we are supposed to test our backups, almost no one does. Those who do test their backups do so with limited restore of a single database or server. I would say that the people who do even very limited in scope testing make up  1% of security professionals.

The other 99% don’t test backups at all. We are lucky if they read backup exception reports of the stuff that the backups didn’t back up. We don’t have time to figure out why all those files and folders aren’t backing up correctly. The answer is often that those active files and programs can’t be backed up correctly. Or they could be if we had just the right backup software or had that extra expensive module that management keeps removing from the budget.

Backups and restores are a professional and logistical nightmare. Almost no one is doing it right across all critical systems. Almost no one tests backup restores in a way that assures the data can be restored in an emergency. We “wink, wink” say it’s done on every compliance survey and audit. If you say it’s done and show minimum evidence of it being done, parties on each side of the audit are glad to check that the “backup-and-restore testing” requirement is done. This is killing our industry.

Don’t let ransomware expose your bad backups

What is the evidence of bad backups? Nearly every successful ransomware attack.

Our newsfeeds are full of stories about cities, hospitals, police stations and businesses that find their data restorations lacking after their data is maliciously locked up. Consequently, they might pay hundreds of thousands of dollars in ransom or recovery work.

Ransomware-hit entities often claim that it’s cheaper to pay the ransom than to do the restores — even though their backups are good. I believe that. Restorations often take what seems like forever and getting restored data and services to work perfectly isn’t guaranteed. Our systems are overly complex. The data restoration may work, but when you start up the server or service, you still get errors when the application starts. Then you pay people to recover the recovered services.

In about 40% of cases where the victims paid the ransom because their backups didn’t work, they did not get easy or reliable access to their previously encrypted data. Not surprisingly, ransomware isn’t bug tested with high levels of customer satisfaction in mind.

Some cyber incident insurance companies make the decision on whether to pay the ransom based on the ransomware family that encrypted the data. Even though it seems cheaper to pay the ransom, the insurance companies know it doesn’t necessarily result in the encrypted data being usable again. Many companies that pay the ransom hire recovery experts, too. People who think they have good backups and refuse to pay the ransom frequently do this as well.

I don’t want to blame the backup software/service companies, even though some can be quite complex. If you follow the vendor’s recommendations and do the right testing, you can get to a state of reliable, tested backups. Almost no one does this.

8 steps to backups and restores that work

1. Make backup and restoration testing the high priority we’ve always said it was.
2. Pay someone specifically to do this as their main task. Making it one of 30 tasks they have to do means backups and restores will not be done right.
3. Test restore all critical systems in their entirety and ensure that the supported applications work as expected. Don’t let a successful ransomware attack be the first time you go through the complete process.
4. Document the restoration testing process step by step, including everything to get to the point where the test applications are proven to work perfect. The actual restoration testing should be documented, including what did and didn’t work. Most people doing complete test restores for the first time find that their test restoration processes don’t work as expected. Failure the first time should be expected. Test, learn, fix and test again. And document.
5. Perform backups in multiple timeframes (e.g., daily, monthly).
6. Encrypt your backups.
7. Store backups in multiple, separate, physically distinct locations, some of which are offline and unreachable by ransomware and hackers.
8. Get rid of the compliance checkbox mentality. Backups and test restores are more than a simple checklist question. Be prepared to show an auditor the detailed data of the restores, tests and application testing that prove it was really done the right way.

We have a big, big problem on our hands. Our backups aren’t nearly as reliable as we think or are told to believe. It’s time for the industry to acknowledge the problem, tell the right people about it, get the right focus and resources to fix it, and start doing what we’ve been saying we’re doing before malware bares our lies.