Back to Insights
Van Wyk: Don't Just Write Your IR Plans, Test Them
August 28, 2017 | Incident Response Planning
By Ken Van Wyk, IANS Faculty
You’ve done your best to plan for security incidents and your plan takes into account every conceivable detail. Yet, during an actual emergency, your success or failure will be determined by people largely beyond your scope of control. Does that keep you up at night? It should.
Responding to information security crises is far more than a technical task. It involves close coordination and communication with people who do not have a technical background. This includes individuals in corporate communications, general counsel, human resources and so on. Every well-thought-out incident response (IR) plan should involve and include these folks.
Let’s say your incident response plan (IRP) already involves these people. You’ve even included swim lane diagrams to describe and visualize how the inter-team cooperation and coordination will take place. And those folks have even signed off on the IRP. That’s awesome. Now, how much confidence do you have that they’ll actually do what they’re supposed to do during a crisis event?
That’s what I’m driving at. Your IRP is only as good as the actions these people take during that stressful period of time.
From personal experience, I can tell you that while these individuals may have read your IRP and signed off on it, it was likely then filed away and all but forgotten until after — yes, after — the next crisis. I say “after” because that will be when the finger-pointing happens.
In the military, they say no war plan ever withstood its first encounter with the enemy. Much of the same can be said for IRPs. Unless all the participants know what is expected of them, they’re going to revert to their immediate intuition and judgment during a crisis, completely forgetting all the steps you laid out so carefully in your IRP.
Now, the good news is that there’s a strong chance that intuition and judgment will actually do a pretty good job getting people through the crisis. It may even carry the day and succeed, after all. But, it’s ad hoc. The participants are likely to get a lot of things right, but they’re just as likely to make some critical mistakes.
For example, it’s required to maintain a chain of custody when it comes to handling evidence. Mistakes related to something like this can have large repercussions if or when the incident under investigation winds up in a courtroom. That’s just one example of the kinds of mistakes untrained people may make during a crisis.
'Drilling' Instead of 'Training'
So, how can you best train these folks? Should you set up a day (or so) of incident response training and invite in the general counsel, human resources and all other participants? No. Please don’t. Even if you succeed at getting a few of them to attend, it’s likely to be the very cure for insomnia. Incident response training simply isn’t likely to accomplish what you need.
You need the incident response stakeholders across the enterprise to understand and internalize their roles and responsibilities, as well as the basic processes they need to follow during a crisis. They need to know how things get coordinated and communicated during a crisis. And, rest assured, this isn’t textbook learning. To get participants to understand, they really need to feel it.
To me, that means tabletop drilling. In fact, that’s a win-win if you structure it right. Invite IRP participants to “incident response training” and it’s the kiss of death. But, invite them to “emergency preparedness drilling” and you may well get them interested and motivated to attend.
This drilling should “train” them by running through some incidents and seeing how they’d respond during an actual incident. It should emphasize decision making, communication and other non-technical aspects of the incidents, but with adequate technical details to be feasible and plausible in your IT environment. And yes, the IT team should also participate, of course.
I’ve found that a half day of drilling is the sweet spot, with each incident scenario running for roughly an hour, followed up by a “hot wash” recap of the team’s successes and failures.
Be sure to make the drilling as realistic as possible. For example, communication among participants should be explicitly stated, or else it won’t exist (in the scenario). Assumptions should be tested, poked and prodded.
Look for single points of failure, like that phone bridge that everyone is using to coordinate their actions. Tell participants halfway through the scenario that the phone bridge just crashed and is no longer functioning. Do they have a Plan B in place? Does everyone on the team know what that Plan B is?
Much of incident response operations isn’t technical, after all. Almost all of it involves handling failures of various types: communication failures, systems crashing, key people being unavailable at a time of crisis, etc. Simulate those things and observe how your team responds, then be prepared to make constructive criticism.
If you do this and do it well, your team will emerge with a far deeper understanding of your actual procedures than they’ll ever learn by reading a boring document. Sorry, but it is boring. Get over it. Let them learn by doing. Oh, and be sure to loop back and make improvements in your own processes and procedures after the fact.
Tailor Your Incident Response Training to the Role
Van Wyk: Doing Your IR Planning the Right Way
Best Practices for Dynamic Business Unit Isolation