In their eagerness to perform well on IT disaster recovery exercises, many organizations make special adjustments, including ones they could not make during an actual event. Tests that are “successful” because of such tweaks can actually be functional failures—unless the organization is transparent about them and addresses the underlying problems.
Related: The Hidden Reasons Recovery Exercises Fail
Summary
MHA has been involved in hundreds of IT disaster recovery (IT/DR) exercises over the years. One thing we see a lot of is organizations doing a test before the test. Such pre-tests frequently surface issues that would delay or prevent recovery in the event of a real outage.
In many cases, after these issues are uncovered, the IT department performs remediation or implements various tweaks and adjustments. Only after a positive outcome is more or less guaranteed is the official test held.
Typically, when these exercises achieve the intended result—a smooth recovery—everyone is thrilled. The people involved breathe a sigh of relief, pat each other on the back, and happily submit a report saying they have no issues.
Then they go back to their everyday tasks, secure in the belief that they are capable of recovering their systems.
But this belief might be a delusion.
Overmanaged test outcomes give a deeply misleading impression of an organization’s recoverability.
In most cases the issues that prevented recovery during the pre-test have not truly been resolved. In many, the tweaks that enabled recovery during the official test would not be available during an outage. For example, a server might be restored during the exercise by cloning a production server, but during an outage, the production server would not be available.
Making these types of adjustments is not necessarily cheating. Sometimes there are good reasons to implement these kinds of tweaks, such as to avoid the early failure of a recovery attempt and the consequent waste of the participants’ time.
But overdoing it can cheat the organization of an honest assessment of its recovery capability and the chance to identify and close its gaps. The changes often go unrecorded. And in the collective desire to declare the test a success, they are often quickly forgotten about.
In these circumstances, conducting a “successful” IT/DR test amounts to a hollow victory.
IT departments often display great creativity and resourcefulness in making sure their systems will come back up during official IT/DR tests.
Here are some of the more common adjustments we see organizations make before or during a DR exercise:
The recovery process is practiced before the official exercise, and issues with recovery scripts, configurations, missing components, or other problems are identified and corrected.
Backups are staged, storage is prepared from snapshots, or server recovery is started before the exercise begins to avoid waiting for lengthy recovery steps.
When a server cannot be restored because of a replication or backup problem, a copy or clone of the production server is used to keep the exercise moving.
File shares or system integrations that aren’t working are bypassed by copying files to local folders or otherwise simulating the connection.
Data is added or modified to compensate for problems encountered during a restore or backup.
Special configurations are made to get applications, access, or integrations working, such as modifying a local host’s file when DNS isn’t resolving, specifying DNS servers directly, or using direct login instead of the normal Active Directory or single sign-on process.
Only a portion of a large environment is recovered. For example, an exercise might recover one application server and one database server rather than the multiple servers that would be required in a real recovery.
Security processes or controls are temporarily bypassed to allow the exercise to proceed.
IT or business personnel use alternate methods to verify data or perform business functions when the normal process isn’t available.
These kinds of adjustments can make an exercise run more smoothly. Unfortunately, they also have a tendency to mask rather than fix problems, and they often rely on measures that would not be available during an actual outage.
Overmanaging IT/DR test outcomes, and concealing or forgetting about the special measures taken, is not a harmless foible. It can negatively impact the organization in several ways.
When an IT/DR test concludes with a smooth recovery, the relief and excitement are palpable: “We aced our IT/DR exercise!” This becomes the headline of the day. The strenuous measures that were taken before and during the exercise to achieve this result are either concealed or forgotten. But many of those measures would not actually be available during a real outage. The supposed “success” of the exercise is very reassuring. It’s also functionally meaningless.
When an overengineered IT/DR exercise results in a successful recovery, management is pleased. They may never hear about the jury-rigging that was done to achieve the positive result. Believing they can now rest easy about their systems’ recoverability, they also rule out making further investments in DR maintenance or staffing. The effort to conceal or gloss over problems thus creates the conditions that make them permanent.
Small tweaks that are applied to enable system recovery during a test tend to have a lot in common with Band-Aids. They treat serious problems in a superficial way. Applying them often masks substantive, underlying problems, preventing them from being seen and addressed. Instead, these problems are allowed to persist in the background, waiting to foul up a recovery when a real outage occurs.
In some cases, an IT team knows that its defined recovery process doesn’t really work and relies instead on an undocumented “back-pocket” solution. Sometimes only one person knows about this workaround, making them a single point of failure. Again, the organization is lulled into a false sense of security. They believe they can recover. But their recovery depends on one person being around to perform an undocumented workaround.
When IT staff and the business departments overengineer their IT/DR exercises, and conceal or ignore their interventions, the organization can come away believing it is prepared for a disaster when it is actually unable to functionally recover.
The answer to this problem is not to ban pre-testing or eliminate every workaround. As previously stated, sometimes there’s a good reason for such tweaks.
What’s needed is transparency, documentation, and a better understanding of the real purposes of IT/DR testing.
DR exercises should not be taken as an occasion to judge anyone’s competence. They are opportunities to honestly assess your recoverability and identify gaps for future improvement.
DR exercises should uncover issues. If they don’t, it usually means either that the test was too easy or someone is hiding something.
The post-exercise report should identify the adjustments that were required and the problems they point to. Then the organization can address the underlying issues through better maintenance, training, technology, staffing, or other measures.
Leadership should want to know what had to be done to make the exercise work. Did you pre-test? What problems did you encounter? What workarounds did you use? What would have happened if the outage had occurred before those problems were fixed?
IT/DR exercises can be difficult to design and conduct objectively, particularly when the people running them are also responsible for the systems being tested. An experienced outside party can help create realistic exercises, identify the gaps they reveal, and make sure the organization gets the full benefit of the exercise.
The key is to judge an IT/DR exercise by what it teaches you, not by whether it was a “success.” A useful exercise leaves the organization with an honest picture of its recovery capability and a clear list of improvements to make.
Many organizations overmanage their IT/DR tests to ensure a successful recovery. By concealing the special adjustments they made, and using measures that would not be available during a real event, they create a false sense of security about their recoverability.
Pre-testing and workarounds aren’t inherently bad. But organizations need to be transparent about the tweaks they make and diligent about identifying and addressing underlying problems.
MHA Consulting has participated in hundreds of IT/DR exercises and knows the many ways an exercise can be overmanaged, as well as how to turn the problems it reveals into useful improvements. If your organization wants help designing, conducting, or getting more value from its IT/DR exercises, get in touch.
Pre-tests often uncover problems that could delay or prevent recovery. Organizations may then make tweaks and adjustments before conducting the official exercise. Organizations that make extensive adjustments to guarantee a successful outcome can undermine the purpose of the test.
The adjustments made to ensure a successful recovery may not be available during a real outage. The test can therefore create a misleading picture of the organization’s actual recovery capability.
They may stage backups, use production copies, simulate connections, modify data, make manual configurations, reduce the recovery scope, bypass security controls, or use alternate ways to perform business functions.
They can create false confidence, persuade management that no further investment is needed, hide underlying problems, and create dangerous dependence on undocumented workarounds.
It is to honestly assess recoverability, uncover problems, and identify improvements. A useful exercise is measured by what the organization learns from it, not simply by whether the recovery succeeds.