Workarounds: Don't Rely on Technology During a Crisis

Given the title of this blog, you might wonder why a consultant whose focus is technology recovery would advise you not to rely on technology to keep your business running during a crisis. The reason is, our business technology is similar to the cars we drive: Most of the time, they work well and provide great convenience, but every now and then they get a flat tire or break down. Most of us have backup plans to deal with this eventuality, such as membership in an auto club that provides roadside rescue. In the same way, we need workarounds to help us keep things going when our business technology goes down.

At MHA, one question we routinely ask when conducting a business impact analysis (BIA) is whether the client has devised manual workarounds for whatever application or automated process we are looking at. In other words, could the operation continue if the technology were to go down? Very often the answer is “no.” There are no workarounds, and if the technology went down the operation would grind to a halt.

The Case for a Manual Workaround

People’s neglect of the need for workarounds is not surprising given our reliance on technology and the efficiencies it provides. And indeed for some processes, having a manual workaround is simply not feasible due to the nature of the process. This is the case, for example, when the process requires high numbers of transactions, as in financial processing, or where the amount of data which needs to be processed is beyond human capacity, as with analytics. In these situations, if the technology is unavailable, the process must simply wait until the tech is restored.

But in most cases, it is possible to come up with some sort of manual workaround that would allow the process to be continued at least at a minimal level if the technology went down.

When Systems Fail

I previously worked as a hands-on application administrator, where I learned first-hand of how unexpected problems can strike a computer system, even when proper planning and testing has occurred. With one upgrade, I recall struggling along with my server and database administrators to get the upgrade completed even though we had previously run through it multiple times to verify the process. We encountered errors we had not seen before and which led to systems shutting down unexpectedly. Many IT professionals have had similar experiences if they care to admit it. At one point, one of us said, “Our business partners would have a heart attack if they saw what we are going through.” Ultimately, we fixed the issues, completed the upgrade, and obtained a stable environment, but there were one or two times when we were close to calling the upgrade off. I mention this not to scare anyone or to suggest that IT departments are not providing appropriate protections, but only to point out that unexpected things can happen even when all of the proper planning and testing has been done.

In many, if not most, of today’s IT shops, the time and resources needed to ensure that every contingency is addressed and that every possible situation has a redundant component are simply not available. Often, managers make a conscious decision not to implement all of the potential redundancies or protections as they are too expensive in time, money, or both.

We strongly encourage all organizations to take a hard look at their business processes and to try to identify some method by which each process could be performed during a technology outage, even if only at a minimal state. The workaround might be something as simple as taking phone calls and explaining the situation, or processing high-priority transactions or high-priority client requests manually.

Implementing Workarounds

Not long ago I went to a retail location and encountered abnormally long lines at the checkout counters. It turned out that the cash registers were down and the staff was having to hand-write sales receipts and make telephone calls to obtain credit-card authorizations. I did not stay as my purchase could wait, but I was impressed by the store’s ability to soldier on in the face of the outage. Their solution was not ideal, but at least for those customers who had to make a purchase, the function was minimally operational.

Workarounds must be developed, documented, and doable by all staff. This includes those who have grown up in the handheld device era and are not used to doing things manually. I recall a BIA discussion with a group where, when we asked if there was a workaround for a certain process, half the group nodded their heads and said, “Yes, absolutely, we would use the reference books and create the documents as we used to do.” The other half of the group, generally a bit younger, said, “What books? We have books? How would you use those?”

Hospitals face a special situation in this area. They are highly dependent on technology for clinical care. Technology is at the heart of such vital capabilities as electronic record keeping, the remote monitoring of patients, lab testing, pharmacy operations, surgical technologies, and radiology. Hospitals must have defined downtime procedures and a well-defined understanding of what can and cannot be done when the various technologies are unavailable. Such information along with well-documented workarounds should be a part of every one of their business continuity (BC) plans.

Workaround Considerations

A workaround might include the use of technologies or applications outside the control of your organization, such as Software as a Service (SaaS) solutions, a document-sharing service, or backup telephony. Something to remember is that internet access could be the weak point. As has been widely reported in the media, our national infrastructure, including power, phones, and internet, is potentially vulnerable to attack, and any device which is connected to the internet can potentially be compromised. For these reasons, you might wish to consider remote, monitored generators for backup power as a workaround. If the power goes out and you have no backup power supply, it would not matter what your backup technology was, because it would not be available. What would you do then?

The message for this week is, do not ignore workarounds in your BC plans. Spend time identifying strategies and plans that will enable you to keep your operations going in the event of a technology outage, even if these alternate solutions are not perfect. Don’t rely on a philosophy of, “We will figure it out if necessary.”