Risk Assessment after Hurricane Irene

Now that Hurricane Irene is behind us, and you've used your Crisis Control skills to salvage as much as possible in real-time, there's a natural urge to start Risk Assessment for the next natural disaster.

However, we cannot predict what the next surprise will be, since then it won't be a surprise. It's even harder to predict how much damage it will cause; nobody could predict the disaster and chaos caused by 9/11 - probably not even the terrorists who masterminded it - including Bin Laden himself.

Most of the time the size of the catastrophe is inversly proportional to the amount of preparation for it.

The trick is to break down the various troubles from the current disaster and group them into distinct categories. Each category then gets treated separately - and each category can be assessed for future prevention, if the risk/cost ratio is high enough. 

Here are some potential groupings - feel free to classify them differently.

• Communication
• Internet connectivity
• Access to our web site
• Access to our email
• Access to our customer support system
• Phone connectivity
• Mobile phones
• Land-lines
• Inter-office communication
• List of key people's home phones
• List of team-leaders contact info
• Escalation plan for:
• Emergencies
• Helping each other
• Moral Support
• Access to office
• Physical access
• Emergency exits
• Virtual access to email
• Access to sensitive info like passwords
• Backup and Restore
• Web sites
• emails
• Personal PCs
• Off-site backup
• Faster Restore

Each item on the list then needs to be taken care of. There are few - if any - generic solutions.

Let's work through the first item on our list: Access to our web site.

During the storm, our web site may have been inaccessible either because the electricity went out, or the internet connection died or the site was overloaded because of too many home-bound people trying to access it.

If our web site went offline during the storm, then we may want to have it hosted in more than one location. Unless it's a web site for the local population, and if the power to the web site goes down, then our entire user base cannot access it either, so there would be no need to be online.

• This highlights why the disaster control plans need to be updated periodically. Nowadays that mobile phones are everywhere, the above presumption is no longer valid. There maybe no electricity or internet within 100 miles of your user base, but you can be 100% sure that many - if not most - of your users will be online from their mobiles, hand-helds, iPads, iPods and other wireless technology.

But simply deciding on distributing your internet hosting is not enough. It needs to be implemented. If your system was not designed for being hosted at multiple sites, this could be a major project. At some stage, management will have to make a decision. To help them, you want to have figures of:

• The cost of damage to our reputation and traffic by remaining at a single site if it becomes unavailable for a day / week / month.
• There's also the cumulative cost, of people never coming back to a site that was down for more than a few moments.
• The monthly financial cost of hosting at 2 or more sites.
• The cost of maintaining more than one site for both engineering, QA and IT.
•  The cost of the design,  engineering and testing efforts required to upgrade the system. 

A final decision point to take into consideration is that the other site chosen may also be targeted by the next disaster.

When the Eyjafjallajokul volcano erupted, it caused airports all over Europe to be closed. Had your emergency flight plan called for racing across the continent from Belgium to Italy to catch a flight, it would have been useless as a backup plan.

Hurricane Irene proved that the 500 miles between NC and CT are not enough when doing hurricane Risk Assessment. An elementary understanding of weather patterns, the national electricity grids and the internet backbone would be required to decide on the safest location for hosting the mirror site.

If the decision is taken to implement the dual-location of the web site, then the implementation has to be tested to ensure that if either site goes down, then the other site will manage to run solo. It has to be able to handle the load; not only double the regular load, but maybe double of that, since during emergencies the site tends to get more traffic, either because it has useful information, or entertains those who are home-bound due to the circumstances, or maybe because the competition was not prepared to implement multi-site hosting.

Side benefit: Once we have multi-site hosting for out web site, maybe we can use it for other items on the list:

• Email access
• Lists of passwords and phone numbers
• Escalation plan
• Customer support
• Off-site backups

But don't assume that any of these solutions are trivial; you have to take into account issues such as security and access control when sensitive information is made available online.

In conclusion, identifying the trouble spots is only the first task. At some level it's the easiest one. Implementing solutions is harder. In future posts maybe we'll analyse what needs to be done for other items on the list.

Risk Assessment and Crisis Control are full time jobs; and the more time you spend on them, the more you'll be prepared for the next surprise that nature or mankind will spring at you.

- Danny Schoemann

Crisis Control - introduction

Crisis, as defined by Answers.com:

A highly volatile dangerous situation requiring immediate remedial action.

This can be as trivial as running out of coffee, or as serious as your biggest international customer's database crashing.

Whereas Risk Assessment is the art of preventing a crisis, when a crisis happens, a Project Manager has to know how to take control and remedy the situation in the fastest possible way.

To highlight the difference between Risk Assessment and Crisis Control, let us compare 2 typical scenarios:

• Risk Assessment:
• Ensuring the building complies with fire codes:
• Bright EXIT signs at appropriate places
• A working sprinkler system
• Employee awareness of where the fire extinguishers are and how to use them
• Periodic visits by the fire department to test the systems
• Company lectures on fire safety
• Crisis Control:
• Somebody faints. You don't call a meeting, you don't do a web search, you don't start checking your cellular for a Doctor's number. You immediately shout for help:
• "Call an ambulance"
• "Who is a paramedic?"
• Do elementary first aid

In a development environment, you would typically take the following steps:

• Inform all those who need to be informed
• Call an immediate meeting of all related people
• Discover the scope of the crisis
• Define it's urgency
• Shrink the meeting to a minimum of people
• Create a Tiger Team who will be in charge of the crisis
• The Tiger Team would drop all other work and concentrate on solving the crisis
• The Tiger Team has the authority to use any resources they need

The Project Manager's task should be to walk around from one team member to the next, ensuring they are concentrating on the crisis, escalating issues that are not easily solved, and disseminating frequent status reports.

In a future post we will discuss what to do after a crisis, besides for informing everybody that the crisis is solved, and that everybody should go back to their regular tasks.

- Danny

Introduction to Risk Assessment

Risk Assessment is typically done in the planning and design stages of a project.

Risk Assessment is somewhat of a guessing game that pessimists excel at, and consists of making a list of: What could go wrong?

Let's create an imaginary list of potential risks for a project:

1. A key engineer takes vacation
2. A new component is faulty and cannot be integrated
3. An earthquake destroys the building

Once there's a list of potential things that can go wrong, the risks have to be analyzed, prioritized and tracked.


Analyzing risks.


Typically, each risk is rated on a scale of 1 to 5 for probability and severity using the following rules:

Probability:

• 1= low chance of this happening
• 3=there's a 50/50 chance of this happening
• 5=almost sure this will happen

Severity:

• 1=project will not really be affected if this happens
• 3=project will be seriously impacted of this happens
• 5=project will fail if this happens

Taking our 3 sample risks and scoring them, by multiplying Probability x Severity gives us:

 Risk Description   | Probability | Severity | Score
 1. Vacation        |      3      |     4    |   12  
 2. Component fault |      4      |     4    |   16  
 3. Earthquake      |      1      |     5    |    5 


Prioritizing the Risks

In the above sample, we would first want to deal with Risk #2 as it has the highest score:

A new component is faulty and cannot be integrated

Typically the involved parties would brainstorm how to minimize the risk, and a game plan will be drawn up that will be tracked periodically; every few days if needed.

In this particular case we may decide to start integration and testing of the component ASAP - even before the design is ready, to make sure we have a head start.
In parallel, we may want to do research for alternate solutions as well as a web search to see what issues other people have encountered with this component.
We may also want to have a contingency plan in case we have to deliver without this component; this may involve warning management about this risk and its implications.


We would then tackle Risk #1 as it has the next highest score:

A key engineer takes vacation

Here we we have a high score that is easily solved. Call in all key engineers and ask them about their vacation plans for the duration of the project, and set down clear rules for future vacations, e.g. you need to give us 10 days warning before going on vacation.
In addition, we may want to insist that all engineering work is done in pairs so that no single engineer becomes indispensable. Or we could insist on periodic peer reviews, so that more than one engineer is familiar with each aspect of the project.

 We then look at Risk # 3 - An earthquake destroys the building - and decide that with its low score of 5 we do not have to deal with it. If it should occur we would handle it using Crisis Control, something we will discuss in a future post.


Tracking our risks.

Tracking - as we mentioned - would be done periodically in a quick meeting. The agenda would be:

•  Risk #2: Key component:
• Progress update on integration
• Progress update on search for alternatives
• Status of on-line help
• Risk #1: Key personal
• Present peer-review plan
• Approve vacation schedule
• Remove this risk from next week's agenda
• Did anybody identify new risks?

This meeting would continue until we are confident we have control of the key component that needs to be integrated.

However, a good Project Manager will be continuously be doing a Risk Assessment of the project, until it is safely delivered. This is  done almost subconsciously.


As my good friend and former CTO of Answers.com - Jeff Schneiderman - was wont to say:

Managers hate surprises.

A manager wants to find out that a project may not be delivered on time, as soon as he can, in order to create a contingency plan  and inform relevant parties.


Typically a Project Manager will raise a Red Flag when he suspects the schedule is in jeopardy. Worst case scenario he over-reacted and his suspicions are wrong. As the saying goes:

It's better to be a pessimist; you are continuously either being proven right or pleasantly surprised.

Risk Assessment minimizes the surprise factor in a project - and allows one to use the wisdom of hindsight in advance.

If you ever said "why didn't I think of that" after your project failed, then you need to improve your Risk Assessment skills.

In a future post we will discuss who to brainstorm - a crucial component of coming up with potential risks.

- Danny Schoemann, Jerusalem