Reporting

What does a readiness score tell a board — and what doesn’t it?

· 5 min read · PlanProof

Short answer

A readiness score tells a board, in one number, how well the team performed against a specific scenario on a specific day, where the weaknesses clustered, and — once there is a second exercise — whether things are improving. It does not tell the board the organization is safe, it does not cover risks that were not exercised, and it is not a certification. Used with those limits understood, it is the most useful single line in a continuity report.

Boards like a number. Risk committees like a number. The trouble with business continuity is that for years the only number anyone could offer was “we have a plan” (one) or “we don’t” (zero). A readiness score from a tabletop exercise is a better number than that, and it is worth being clear about exactly what it is better at, because a score that is over-read is more dangerous than no score at all.

Where the score comes from

Neither ISO 22301 nor NCEMA 7000:2021 provides a scoring method. Both ask for clear objectives and a post-exercise report, NCEMA 7000:2021 (8.6.1) adds criteria for measuring success, and both leave the measurement to you. Our score is our own method, built on an approach that emergency-management programmes have used for years: rate each capability that the exercise tested on a short scale, weight the capabilities by how much they matter, and combine them.

In our reports, the exercise is assessed across seven capability areas: activation and command, decisions, internal communication, external communication, regulatory notification, recovery and manual workarounds, and plan usability. Before the exercise, each area is given a few good-practice criteria, some drawn from the organization’s own plan (activation within 30 minutes, say) and some from the two standards, and every criterion is tested by something in the scenario. Each criterion is then rated at one of four levels from the recorded evidence — performed, performed with some challenges, performed with major challenges, or not performed — worth 100%, 75%, 50% and 0%. The areas are weighted (activation carries the most, because everything else depends on it), the weights are fixed before the exercise and kept for the next one, and the result is a percentage. Seventy and above shows green, forty to sixty-nine amber, below forty red.

One rule in that method matters more than the arithmetic: an area the scenario did not test is excluded. It does not score zero and it does not score full marks. It is simply absent from the calculation, and the report says so. A score that silently credits untested areas is a guess dressed up as a measurement.

What the score can tell a board

How the team performed against this scenario. A 72% on a cyber scenario means the team activated, decided, communicated and recovered reasonably well when faced with an IT outage on that afternoon. That is a real finding about a real capability.

Where the weakness is. The single number is the headline, but the seven areas underneath it are the story. A 72% made up of strong command and weak regulatory notification tells a board exactly what to ask about. In practice boards get more from the lowest-scoring area than from the total.

Whether things are improving. This is where the score earns its place. A second exercise a year later, on the same scenario family, scored the same way, turns a snapshot into a trend. “Fifty-four to seventy-one, with regulatory notification moved from major challenges to performed” is the kind of sentence a risk committee can act on and an auditor can file. It is also the sentence that justifies the budget for the next exercise, and it gives the management review that NCEMA 7000:2021 expects at least annually (9.3.1) something concrete to examine.

Whether the actions worked. If last year’s report said “notification order unclear” and this year’s score for that area went up, the corrective action did its job. If it did not, the board knows the action was closed on paper only.

What the score cannot tell a board

That the organization is safe. A score is one scenario, one afternoon, one room of people. A 78% on a flood scenario says nothing about a ransomware attack, and it says nothing about whether the flood plan would hold if the crisis manager had been on leave. Boards should hear the number and the scenario together, every time.

Anything about risks that were not exercised. This sounds obvious, but a green result in a board pack has a way of being remembered without its footnotes. If the exercise programme has covered two of six critical activities, the honest statement is that two have a score and four do not.

That a system will recover in time. A tabletop exercise tests decisions and communication. It does not restore a database. The recovery and manual workarounds area is rated on whether the team knew what to do and whether the plan supported it; proving that the backup actually restores in four hours is a separate test, and the report should say whether that test has ever been run.

Compliance. A score is evidence for an audit; it is not the audit. The standards want to see the programme, the objectives, the report and the closed actions. A high score with no actions closed is weaker evidence than a modest score with every action closed.

Precision. The levels are judgements, made by a facilitator and observers against written criteria. Two assessors could differ by a level on one area. A score of 72% and a score of 78% are the same result; a score of 48% and a score of 78% are not. Boards should read the band — red, amber, green — and the direction, not the second digit.

Five questions a board should ask about any score

  1. What scenario was this, and what was deliberately not covered?
  2. Which area scored lowest, and what is being done about it, by whom, by when?
  3. How does this compare with the last exercise, scored the same way?
  4. Which of the plan’s recovery promises have actually been tested, as opposed to discussed?
  5. When is the next exercise, and on which risk?

A management team that can answer all five has a continuity programme. A score that comes without those answers is a number on its own, and boards should be wary of it, including when the number is a good one.

Why we still think the score is worth having

With all those caveats, we would rather report a score than not. It forces the exercise to have objectives, because you cannot rate what you did not define. It forces the report to be specific, because every area needs a judgement with a reason. It gives the board a trend instead of a reassurance. And it changes the conversation at the next budget cycle from “do we really need to do this again?” to “how do we get regulatory notification into the green?”

That conversation is the reason to keep score.

Sources