Incidents are inevitable. We will eventually face a crisis as long as we divide our resources based on imperfect information. While our regular planning hopes to reduce the probability of an incident, our preparation for an incident helps us operate under uncertainty.

Why should we prepare for an incident?

I thought we want to prevent incidents?

Consider this. The healthiest security organization has a fundamental, catastrophic flaw: Security teams make predictions.

Predictions are a foundational flaw in any strategy. Predictions force us into partial solutions.

Predictions divide resources across many possible outcomes due to a lack of information about where an adversary might succeed. Eventually, this division of resources will cause a breakdown. If we were omniscient: We would entirely and cheaply avoid every undesirable outcome.

No security team can identify the right risks with any guarantee. The team cannot correctly allocate the right amount of effort or resources or precisely solve mitigations without taking away from others. We will eventually fail.

We have to understand incident response. Let’s discuss how an incident feels.

Incidents sometimes start with an unexplainable behavior.

A small group will troubleshoot what would resemble an outage. They will slowly investigate and gather evidence that poorly suggests any particular cause.

A team member will cautiously suggest that a breach explains the behavior. A silent, uncomfortable urgency will take over while a single teammate considers escalating.

Incidents sometimes start with an external notification.

A security company cold calls you with a “disclosure.” Or, the FBI might reach out and ask for a meeting or a phone call. A journalist might reach out with a ridiculous-sounding claim and leave your communications team scratching their heads.

Incidents attract attention and requests for help very quickly.

The group involved with the incident will expand rapidly. The incident responders find themselves repeatedly onboarding new people into a nuanced timeline, still in progress.

Incidents send some employees home for the day.

Employees that are not involved with the incident often stop working while production or corporate infrastructure is being investigated.

“When will the development servers be back online?”

You won’t have an answer. Answering will be the least of your worries.

Incidents operate on incomplete information.

The size of the response team will become unwieldy. Access requests (Can you give me access to the investigation? The logs? The tooling?) will become an area of frustration. Questions like “Are we sure this was all the adversary did?” or “Are we sure this is safe to bring back online?” is a prefix to nearly every decision.

Incidents eventually become public.

A pressure of external scrutiny will creep in as a regulatory disclosure, blog post, customer notification, or lawsuit looms.

An incident response plan.

Security teams should prepare for an incident in addition to preventing one. The time is spent on process building and establishing roles and relationships.

An incident complicates a work environment. Decisions are higher stakes. Leadership roles are more pronounced. Information flows quicker than usual. Close coordination is desirable but challenging to maintain.

A straightforward incident response plan eases the pain of coordinating a complex incident. Keeping the plan simple allows you to practice.

An incident response plan expedites common friction points:

  • A rolodex of contacts: Law enforcement, Forensic, Legal, Leadership, and PR contacts who can come online quickly and contribute to a response.
  • A template for collaboration: Incidents are often about managing uncertainty. What are questions we still have? Who is accountable for answering them? What are the short term emergency actions, and what are the long term learnings?
  • Approval Points: Who approves an emergency blog post, an email to customers, or calls law enforcement? Pre-approvals for expected steps will avoid delays and meetings.
  • Internal Communications: Who communicates issues with employees?

An organization would usually spread project milestones over weeks, months, or years.

However, incidents move fast. This project is happening now and looking to resolve asap.

Work is viewed in hourly timeframes during crisis management. The direction of the whole effort can reverse with every status meeting. An agenda for meetings is important. You’ll have them regularly, likely hourly. The expanding group will need to know how to contribute and communicate during frequent check-ins.

Here is an example agenda:

Agenda:

  • Add or remove items from the investigative timeline.
  • Announce any new forensic or investigation findings.
  • Answer any open questions and ask new investigative questions.
  • Decide on short term actions (Containment, mitigation, etc.)
  • Make notes for long term actions and learnings.

Incident readiness

The most valuable contributions to an incident meeting will include leverage from investigative systems. Those, in particular, are asset and configuration management, but mostly logs. Logs and the systems and processes that rely on logs are a core investment area for incident response readiness in maturing organizations.

Application or product development environments often take logs for granted. Critical applications should produce logs that are useful for incidents.

A security team should work with your application or product teams to create logs that are useful for an incident. Additionally, they’ll be motivated to centralize these logs, so they’re accessible and searchable in a crisis.

Exercising your incident response

A security team can practice incident response. Options are available to test your incident readiness.

Simple and Easy: A tabletop exercise. Prompt your team with What if? scenarios and over a series of lunches and downtime.

A bit more complicated: An active tabletop. Take steps to investigate What if? prompts with the tooling that currently exists. Invent some ghosts and investigate them.

Very complicated: Bring on a red team to simulate an incident. Work with the red team and your resources to discover the gaps in your ability to investigate their actions.

Prompts can include incidents the organization has already seen, incidents competitors have seen, or incidents the broader industry has seen.