Beginner’s Guide to Recovery Exercises

Recovery exercises are an undervalued but critically important aspect of business continuity. In this beginner’s guide to recovery exercises, we’ll provide answers to some of the questions we are most frequently asked about this vital topic.

Related on MHA Consulting: Testing, Testing: Our Best Blogs on BC Testing and Mock Disaster Exercises

 

The Value and Versatility of BC Recovery Exercises

Recovery exercises are one of the most important aspects of a business continuity (BC) program. We at MHA are big believers in their value, versatility, and necessity. Exercises can tell you things about your organization’s resilience that it is essential for you to know and which you can’t find out any other way. They also provide training and practice at recovery unobtainable through any other method.

Unfortunately, many organizations neglect to develop a sound recovery exercise program. Many exercise-averse executives tell us they are confident their people will be able to figure things out on the fly or that their ordinary operations provide all the practice at recovery they need (not so; the two are as different as day and night).

Nonetheless, we do receive many thoughtful questions about recovery exercises from clients who  have the foresight to grasp their value. Below are some of the questions about exercises we are asked most often along with their answers. Our hope is that, collectively, these exchanges will make up a guide to recovery exercises that is helpful to beginners and experienced practitioners alike. 

Why perform recovery exercises?

The main reason to perform recovery exercises is to make sure you can recover your business in the event of a disruption or disaster. Absent such exercises, you have no way of knowing that the recovery plans and strategies you have put in place will actually work. All you have is the hope they will work. If this is sufficient for you, then you have no need to perform recovery exercises. However, if you would rather base the future of your organization on something stronger than hope, you should be performing such exercises. Hope is not a strategy.

What are the main benefits of performing recovery exercises?

Conducting recovery exercises brings several valuable benefits. It enables you to validate your recovery strategy and recovery processes. It helps you identify gaps in your recovery processes and strategy so you can correct them before a real event occurs. It provides a training opportunity so you can make sure everyone with a role to play in recovering the business knows how to do their part.

What are the two main recovery-exercise areas?

The two main areas for which recovery exercises are performed are:

1. IT disaster recovery (IT/DR) exercises – Technology-based exercises that focus on recovering processing functions, applications, systems, and data centers. These exercises look to see whether the organization can restore its technology and get it running again. Important, but they are not  the be-all and end-all, as many think.
2. Business continuity (BC) exercises – Center on actions taken to recover your business processes, such as manufacturing, research, finance, and accounts payable. Can include testing the organization’s ability to relocate people or processes to a new facility.

What are the four main types of recovery exercises?

The four main types of recovery exercises are:

1. Tabletop exercises and structured walk-through tests. A preliminary step in the overall testing process. Can be an effective training tool. The primary objective of this type of exercise is to ensure that critical personnel from all areas are familiar with the various plans (Business Continuity, Crisis Communication and Management, IT Recovery, Emergency Management, etc.) and that the plans accurately reflect the institution’s ability to recover from a disaster.

• Walk-through drill and simulation tests. Sometimes referred to as mock disaster exercises. Meetings where the participants walk through how the organization would respond to an emergency scenario if it were to happen in reality. These exercises look at how management would assess event impacts, communicate among the parties, and determine whether to implement recovery procedures such as relocating to an alternate site. The most commonly performed exercise.

• Functional exercises, functional drills, and parallel tests. These involve the actual mobilization of personnel to other sites in an attempt to establish communications and perform actual recovery processing as set forth in the recovery plan.

• Full-interruption and full-scale tests. The most comprehensive type of test. In a full-scale test, a real-life emergency is simulated as closely as possible. The participants carry out a recovery in the real world in real time, performing an actual failover from production locations or processing. Should only be performed once your tabletop and simulated recovery exercises have demonstrated full capability with few or minor issues.

How does an organization develop a testing program?

Start small and ramp up. Run tests based on the maturity of your program. Each of the tests is a kind of training before you move on to the next level. At each level, the stakes are higher, and the activity more closely replicates the situation of an actual disaster. Each level provides feedback and an opportunity to improve your procedures.

Start with a limited scope. You could do individual tabletop exercises with each department first. Then bring in multiple departments where dependencies exist. Make sure your processes and strategy seem sound before you go on to a simulated recovery.

Even within the different types of exercises, you’ll need to progress over time. With simulated recovery, you’ll want to start with a few applications or business units then ramp up as you become more proficient.

Very few organizations actually perform a production recovery exercise. They are risky, and most organizations have not done the necessary planning and preparation. Preparation comes from performing the other two exercises multiple times over a period of years. Even then, be sure to consider the risk to production if something goes wrong.

Are BC and IT/DR exercises performed separately?

In the beginning, probably. However, mature organizations will eventually exercise both areas together. For example, a mock disaster will be declared, and while the IT team performs the recovery of the apps and technology, the business team will be performing their functions. These integrated exercises take a lot more planning than just working on one side or the other. The teams involved will talk about dependencies, and the scope of the exercise must be clearly defined.

It’s often the case that an organization’s testing is more mature in one area than another. Typically, the IT side is ahead of the business side in terms of preparedness, processes, and documentation. In this situation, it might happen that, in a joint exercise, the DR team performs a simulated recovery while the BC side does only a tabletop exercise.

An integrated exercise is something that is worked toward over time. Once the organization has done tabletops and increased the scope to include multiple apps and environments, then it could consider an integrated test where DR and BC are brought together to run a combined exercise that leverages both plans. For a company just starting an exercise program, such a project might be two to five years down the road.

What exercises should my organization be performing?

This depends on where you are in your exercise program. You have to walk before you can run. If you’ve yet to do tabletops, that is the place to start.

When you have validated your strategies through tabletop exercises, you’ll be ready to move on to simulated recovery exercises. Then, maybe to production recovery exercises. When you have substantial experience in testing both sides of your business (BC and DR), you will be ready to think about conducting an integrated exercise as described above.

Do companies graduate beyond the need to do tabletop exercises?

No. Even companies with mature exercise programs can reap dividends from performing tabletop exercises. They are an underused resource in our exercise methodology. They’re easy to schedule and perform, take very little time, and bring ongoing benefits. Even after we learn to run, we still find it advantageous to walk a good deal of the time. These exercises are a good way to keep people thinking about DR/BC as well as to verify changes to plans, strategies, and processes when significant changes occur in the IT or business functions.

Tabletops can also be supercharged in the form of the comprehensive tabletop. This is a version where people’s feet are held to the fire in terms of making them detail step by step how they would accomplish the recovery. It’s a simple process that can uncover a lot of unexpected gaps in knowledge and preparation.

How frequently should we perform the different types of exercises?

Tabletop exercises performed within a single department can be done as time and resources allow. If you were doing a separate exercise for each department, you might do them quarterly.

Most organizations will do one to two major exercises a year. As you increase the scope of your exercises, they become more difficult to coordinate and execute.

Depending on your strategy, you might do a smaller scale exercise once a quarter with more major scoped exercises annually. Exercises demonstrating your overall recovery strategy for both BC and DR should be performed at least annually.

Depending on your recovery strategies, you may be able to perform smaller exercises much more frequently.

Thriving in Adversity

Recovery exercises are not a luxury but a fundamental necessity for ensuring business continuity. In this beginner’s guide, we’ve sought to underscore the value, versatility, and necessity of these exercises while shedding light on their pivotal role in mitigating risks and bolstering resilience.

By validating recovery strategies, honing skills and identifying gaps, exercises perform critical work in fortifying organizations against disruptions. Developing a sound recovery exercise program requires the sustained effort of many people, but having such a program allows organizations to safeguard their operations and thrive amidst adversity.

Further Reading

• Testing, Testing: Our Best Blogs on BC Testing and Mock Disaster Exercises
• Table Service: 8 Ways Tabletop Exercises Can Benefit Your Company
• Overdoing It: People Who Overplan Their Mock Disaster Exercises
• Little Things Mean a Lot: The Value of Micro Mock Disaster Exercises
• Kill the Zombies, or How to Get More From Your DR Exercises

.