by Lou Washington

Who could forget Robert Duvall’s famous line in “Apocalypse Now,” Francis Ford Coppola’s big-screen Vietnamization of the Joseph Conrad novel “Heart of Darkness”? 

One thing you would have to say about Lt. Col. Kilgore is that he certainly knew what was going on around him.  He was seemingly quite at home in the chaos of war, one minute leading a helicopter assault, the next surfing … all with the casual intensity that only comes when one is comfortable in an uncomfortable world.

Git’er Done

While all was chaos, smoke, death and destruction, Kilgore was still all about getting on with life, getting the job done and, in a word, thriving.

In many ways, Kilgore provides a wonderful analogy for companies seeking a strategy for coping with disaster in our age.  In the morning following the “apocalypse” of a hurricane, terrorist attack, fire or virulent cyber virus, you want your company up on the beach, robust and vital, fully functional and unfazed by the bad guys.

Unfortunately, for many the morning after might just be too late.  In this flat-world, global, 24x7 economy, the game is won with business continuity.  Meaning zero downtime, no quick kick and see you in four downs, no re-assessment phase followed by a ramp-up, followed by full production late next week. 

Companies are either in business or not in business, making money or losing money, serving customers or losing customers.

No in-between, no breather, no sort of operational, no quasi-online. 

You are on or off, dead or alive, focused on creating the future or relegated to being a part of the past.

Polymers, Petrochemicals and Pollywogs

I knew a fellow who worked for a large petrochemical company.  The division he worked in produced polymers.  Personally, I wouldn’t know a polymer from a pollywog.  But, my friend asserted that shutting down their production facility for any reason was tantamount to burning millions of dollar bills in a blast furnace every hour. 

No product meant no revenue.

That lost revenue was revenue lost forever. (Ouch)

There was a time when disaster recovery meant keeping a set of tapes at an off-site facility.  You might have even had an agreement with another company across town to allow you to use their facilities and hardware in the event that your box crashed or your building burned down.

If you were really dedicated, you might even swap out the data tapes every 24 hours to ensure that you only lost a few hours’ worth of transactions.

Mirror Mirror on the Wall

Now the standard is a 24x7 hot site, online, mirroring every production transaction, remotely located and staffed by a separate crew.

When your primary data center in East Baltagonia becomes lunch for a suddenly active volcano, your back-up site in Cambrastan can automatically assume the role of production site.  This happens automatically and instantaneously without human intervention.  This is called fail-over or fallback and it is what business continuity is all about. 

The two mainframes are in constant communication with each other.  When one stops processing, the other knows that the show must continue.  Since all of the data, systems and transactions executed on the primary are also mirrored on the secondary system, this is a piece of cake for your average mainframe computer.

Of course, if you are running a large corporation with multiple divisions, locations and IT philosophies, you just might have a more challenging task.  Maybe your business-continuity strategy consists of everyone backing up their PCs to the local server every week.  Perhaps you are under the assumption that all lost laptops will be immediately reported.  And, that guy that runs the operations center down the street?  He should have some extra capacity for you if your hardware is vaporized by a lightening strike. 

Let’s say your systems all suddenly become useless because some idiot releases a virus into your operation after opening an attachment promising assorted monetary, physical and romantic improvements in the reader’s life.  Will the guy down the street really have what you need to keep your enterprise in business? 

Poobah Central

If you have forsaken the concept of centralized data systems and allowed all of the local poobahs to have their own IT operation based on what they might think is best for them, you might have a bit of a problem.

When you get the weather forecast and it’s predicting a level 5 hurricane for the following day, you’ll know what I mean.  You’ll be spending the night on the phone inventorying servers and content.  You will find yourself looking in closets and finding whole systems you didn’t even know about. After you trace network wiring up and down elevator shafts to rooms that don’t appear on your blueprints, you’ll find hardware that’s not supposed to exist.   At this point, maybe the wisdom of some level of centralization will start to make some sense.

Questions to Consider

Let’s say you have distributed everything throughout the enterprise and have dumped your glasshouse operation all together.  Consider a few of the following questions:

1. Do you know specifically what is maintained on each server in your operation?

2.  Can you quantify the effect of any particular building or location being offline and unavailable for say, 48 hours? A week? … Forever?

3. Do you know if your financial systems touch your e-mail systems, and where and how they touch?

4. Would you be able to isolate your sensitive systems from external access without implementing a project of a magnitude normally associated with moon-shots or the Manhattan Project?

5. Do you know, for sure, how many separate servers you have in operation throughout your enterprise and the content maintained on each of those servers?

The point is that there are systems that are easy to manage and easy to mirror.  There are also systems that are unmanageable because they are not planned, monitored, controlled or even documented in any consistent way. 

These systems are virtually impossible to mirror.

A friend of mine who works for a large hardware firm shared an interesting fact with me.  Following the 9/11 attacks, the only companies that were fully operational immediately following that horrible event were companies whose IT operations were mainframe-centric.  I am not saying that all mainframe-based systems escaped, but those that did were mainframe-based.

Concentrated Diverse Dispersed Systems

An article by Mary Ann Gadziala, an Associate Director of the SEC, describes how 9/11 exposed certain weaknesses in financial systems.  Even though some very smart people had designed these systems, the challenges presented on that day were simply not imagined.  Some of these included back-up facilities located too close to primary facilities - back-up sites without adequate capacity to handle the demand placed on them by the attacks.  Essentially, what was thought to be a diverse, dispersed system was in reality a concentrated system vulnerable to a single point failure.

The Federal Reserve recommends business continuity ensured by rapid recovery following widespread disruption or the loss of any one single major operations center.  Also, a program of rigorous ongoing testing of both internal and external aspects of all production and testing systems is required.

An IBM white paper dealing with this characterizes business continuity as having three aspects: high availability, continuous operations and disaster recovery.  These are not going to be present without an overt, continuous effort.  These require much more than printing a manual, signing a contract with a facilities vendor and backing up a few files.

The Requirement? Commitment.

This requires a commitment to continuous evaluation, modification and education.  It requires constant vigilance and a willingness to insist that expenses in this area are just as vital as the expenses associated with your production environment.  There will always be someone who will be willing to gamble, willing to bet against disaster and willing to leave it all to chance.  That is the battle you have to fight every day.

When you are walking about that beach on the morning after, smelling the cordite, will you know what to do?

About Lou Washington

Professional Life: I started my career in information management from the somewhat misunderstood field of Records Management. Following four years of working for the University of Missouri System's Office of Records Management, I joined Tab Products Co. in 1980. Shortly thereafter, I became interested in the software business, PCs and how those systems would shape the enterprise of the future. We were transferred to Tab's then corporate HQ in Palo Alto, CA. I was the first Product Manager for Tab's Tracker systems software products that utilized a PC-based bar-coding system to track the movements of everything from files to capital assets. I believe it was the earliest example of workflow automation available on the market. I was also peripherally involved in Tab's Laser Optics division, which brought to market one of the earliest business systems employing CD-ROM and WORM technology as an information storage media.

In 1990, I returned to Cincinnati and joined Cincom Systems where I began to learn about and work with mainframe-oriented products and systems. In those days, there was a real "split" between the mainframe forces and the desktop proponents. I always found this to be amusing since both had so many positive things to offer an enterprise. I could never understand why anyone would offer one at the exclusion of the other.

My present role at Cincom involves a number of things including product security, pricing, finance packaging and industry research.

My wife, Barbara, and I reside in Park Hills, KY. I am a member of Blessed Sacrament Church and I am active in a local car club, Cincinnati Cruisers. We are a group of PT Cruiser owners who enjoy tricking out our cruisers and driving around annoying people who have to drive boring cars. I am the Webmaster for the Cruisers and I invite everyone to visit www.cincyptcruisers.com and check out our awesome rides! Barbara and I both enjoy photography, travel and our two four-legged canine children, Chloe and Cookie.

Copyright Cincom Systems, Inc. All Rights Reserved