IT Change Management: Don’t Leave Your Recovery Environment Behind

It is a strange irony that the changes organizations make to remain competitive frequently open them up to risk in their DR/BCM program and recovery capability.

But when it comes to business continuity, the IT change management (CM) process at most organizations is integrated in name only.

Many organizations are ambitious about making changes that will drive the business forward and are careful regarding the implementation of those changes. But the need to keep the recovery plans and environment in sync with the production environment is frequently an afterthought.

In the event of a disruption, this can have impacts on the business ranging from inconvenient to calamitous.

In today’s post, we’ll discuss some of the main issues with CM from a business continuity perspective. We’ll also share some tips on what business continuity professionals can do to make sure that routine system and process changes do not leave the organization vulnerable to major impacts.

THE ONLY CONSTANT

The Greek philosopher Heraclitus said 2,500 years ago that “The only constant is change.” This is truer than ever in today’s fast-moving corporate and technological worlds.

Most organizations have developed sophisticated change management processes to guide the implementation of the changes to their software, hardware, networks, and business processes needed for the organization to adapt and remain competitive.

However, in our experience at MHA Consulting, we have found that many IT change management processes fall short when it comes to business continuity.

Many organizations, in their (understandable) eagerness to implement changes that will help the business, put off until another day the chore of making sure their recovery plans and processes are updated to reflect the new production environment. Also impactful is delaying implementing technical changes in the DR environment or alternate site.

Unfortunately, for many companies, the day for doing this chore never comes.

If the organization is lucky, it might discover the gap during a recovery exercise. If it is unlucky, it discovers it during a real-life event.

TICKING TIME BOMBS

When alternate site technology or process lags behind a production environment, the gap is a ticking time bomb waiting to go off. In some cases, the explosion might be no more than a weak firecracker. In others, it might be strong enough to cause a recovery to fail completely.

The bigger the gap between the production environment and the recovery environment, the greater the chances that the recovery will fail.

In fact, we know of multiple cases where, if there were an outage, the organization’s current environment could literally not be recovered due to the large gap that has opened up between the production environment and the IT/Disaster Recovery (IT/DR) environment.

A BETTER WAY FOR IT CHANGE MANAGEMENT

There is a better way of managing change from the BC perspective. In fact, there are a couple of better ways.

One of them is ideal but admittedly not very feasible. The other is highly feasible as well as reasonably effective. Either one would amount to a big improvement over what we see currently at many leading organizations.

The ideal solution would be for the organization to implement changes to the recovery environment simultaneous with changing the production environment, as a planned part of the change, where change management does not allow a production change without the corresponding DR/BC items being completed. The reality, however, is that this doesn’t happen.

More realistically, organizations can adopt a functional approach to change management, from the IT/DR and BC perspective. In this approach, the organization integrates DR and business continuity considerations in their change management processes, at least to the extent of:

• Logging the changes which need to be made to the recovery environment
• Creating a plan to make those changes in a timely manner
• Monitoring and reporting on those items which have not been completed in the scheduled time

The needed changes to the recovery environment can include changes to hardware, software, data, and process dependencies. Changes would also include updates to documentation – technical recovery plans, business continuity plans, or any other documentation.

The best way to track the needed changes to the recovery environment is through a regular CM process. You can also use a spreadsheet or a ticketing system.

ASCERTAINING URGENCY

If we accept that updates to the recovery environment will generally be made some time after the changes are made to the production environment, then we encounter the issue of determining how long we can safely wait. The answer is, it depends. Theoretically, any gap between the two environments creates a risk. But some types of changes and gaps are more urgent than others.

Generally speaking, software or tool upgrades and software version changes that are implemented in the production environment need to be quickly mirrored in the recovery environment.

The same is true of hardware changes, interface changes, and changes to the overall network configuration.

Less urgent are changes to manually executed configurations, network port changes, and individual network changes.

THE ROLE OF THE BCM TEAM

Currently, BCM professionals tend to have little involvement in the IT change management process.

Ideally, BCM professionals would take on the role of advocating for the recovery environment during organizational change. They would remind the CM team of the importance of keeping the recovery environment up to date and explain the potential dangers to the organization of gaps being allowed to open between the production and recovery environments.

The BCM team should also ask questions of the CM team to ensure that they are keeping track of the changes that need to be made to the recovery environment, devising a plan to make those changes in a timely manner, and carrying out that plan.

The BCM and IT/DR teams could make collaborating with the CM department in this manner an item on the BC action item list.

The BCM team should keep in mind that IT departments, when making changes to the IT production environment, commonly say that everything is fine with regard to IT/DR. Close questioning will frequently reveal this is not the case and that IT’s focus on production has allowed significant gaps to open between the production and recovery environments, creating vulnerabilities for the organization.

By being diligent in following up in this area, BCM professionals can make a meaningful contribution to the resiliency of their organizations.

JUST DO IT

We began this post with a quote from the ancient Greek philosopher Heraclitus. We’ll close it with a more recent quotation: Nike’s trademark – “Just do it.”

That’s what IT change management comes down to from the IT/DR and business continuity perspective. It’s not hard. It’s just a chore that has to be done. And the company that lets this activity slide is adding risk rather than reducing it.

All an organization must do to keep from becoming vulnerable to change-induced impairment of its recovery capability is to keep track of the updates that must be made and to make them in good time.

The BCM team will inevitably be at a remove from this process, but by their advocacy, they can help make sure the needed updates to the recovery environment are made and that the organization is protected.