How do you run an investigation with a group? 🕵️
Incident response involves many decisions around containment, disclosure, remediation, and retrospectives. All of these concepts rely on a quality investigation. So, let’s focus on investigations for a moment.
Here’s some needs we have when we kick off an investigation:
Urgency, people problems, and communication breakdowns will complicate the process. Our recommendations and decisions will be better informed if we anticipate those risks in an investigation.
An investigation can be modeled as a sequence of meetings and tasks until questions are answered, and decisions can be made from the resulting information.
The following topics are important to an investigation. We tackle them in regular meetings until the investigation moves on from crisis handling.
The agenda for an investigative meeting is:
Investigative meetings may involve active incident responders. Be sure to set a scheduled cadence and push the agenda so responders can get back to focus. Meetings can happen as frequently as hourly after kick-off and go daily or weekly as a conclusion draws near.
The meeting starts by updating the summary and investigative timeline as a group.
An early investigative pain point is onboarding new people into the investigation. Time is spent answering, “What happened?” and “Why are we here?” over and over again.
Make the timeline prominent. It gives an investigation a backbone and reduces the number of investigators operating on old information.
The primary documents we collaborate with should have two prominent sections:
We waste no time on old or incomplete information. Call investigators to update the timeline first. The summaries are adjusted as a result. This has the additional benefit of soft onboarding any newcomers to the investigation straight away. The summary should reflect the most recent knowledge to bind the investigation tightly. This supports the group to self-direct themselves based on a very quickly changing narrative of what happened in the incident.
Make sure that items added to the timeline have an owner who can answer for it.
This section of the meeting has some simple, probing questions to ask to warm up contributors.
Questions to ask:
Discuss all new evidence or indicators of compromise since the last meeting. The introduction of new evidence in an investigation generates work as an outcome.
The simple act of finding a new hash, IP address, domain, or email address can kick off work in many directions. Active hunting, threat intel, forensic and administrative issues will follow up. Make sure evidence added to documentation has an owner who can answer for it.
Don’t forget non-technical evidence, like notes from interviews or calls with vendors.
Elicit the group for a quick update on what is new, what needs to be added, and to share comments. Here are some probing questions.
A timeline helps keep the investigation narrative in order.
The Q&A helps us unravel the missing parts of the story.
The first meeting elicits very broad opening questions from the team. This bootstraps the investigation. They can be as broad as “What the hell is going on?” just to get it going. Focused questions can elaborate from there.
Future meetings start the Q&A by getting everyone up to speed with recently answered questions. This is especially important when major discoveries unfold in later meetings. Simply ask, “What’s new? What have we answered?”
Here’s an example question from the list, with an answer we could have added during the meeting.
Q: How did Bob’s password get exposed to the attacker?
So, what’s new?
A: It turns out local malware grabbed the password.
Great! We have answered the question, but it leads to new questions. Elicit the group for those questions and answer them if possible.
Q: What did the malware do?
Great! This question gets assigned to someone.
The investigative Q&A involves all the open-ended curiosities that need to be answered. These conjectures are a creative process and quickly turn into an inquiry with logical reasoning. For example, you may start with:
Q: Why does this security company say our customer data is on the dark web?
Now we can start with logical reasoning. We start to close in on the best explanation for the incident by narrowing questions about what we’re curious about. This is knowledge work. Investigations are not a deterministic process - smarter people ask better questions, and every investigation is different.
Some investigations do have investigative patterns that we can follow, just like how chess has standard openings. However, every investigation varies on the technology, actors, victims, and circumstances involved.
The conjectures we make come from decomposition of our questions. Deduction, induction, and abduction help us form hypotheses and develop evidence for the investigation:
Q: Was our customer data lifted from the only bucket it is written to?
That hypothesis is from deduction. If the data is assumed to exist in one place, it must have been stolen from there.
Q: Did an attacker find leaked credentials?
This example is based on induction, as leaked credentials are a frequent scenario and suggest it could be true here as well.
Q: Did an attacker access a misconfigured CI/CD environment that was leaking bucket credentials?
This narrows down even further using abduction, a plausible cause. A deep dive to see if the CI/CD environment is misconfigured or exposed may reveal something.
We can now get into familiar forensic tasks to study these starting conjectures. When evidence comes back, we start answering questions or making new ones. That evidence will reduce uncertainty and improve decisions afterward.
Some playbooks can be built for common incidents (Ransomware, BTC mining, etc.) with questions formulated ahead of time. However, the investigation process is what’s actually happening without playbooks involved.
The list of questions will be long at first. Give each question an owner. The owner is not accountable for the answer. Instead, they should own the status for actions that should get us to an answer. Upon the next meeting, we’ll look to them for an update on how we’re doing with an answer and what may be blocking.
Every meeting should iterate on this Q&A. Answer questions with updated information and elicit new questions from the group. We repeat this until we’ve reduced uncertainty from the incident enough to conclude it.
Questions to ask:
This essay narrowly discusses the investigation. Investigations are a part of incident response, there’s a lot of other things to do in periodic response meetings.
You can make decisions around containment and eradication once the investigation gives stakeholders the confidence to do so.
The conclusion of an investigation is just another decision. There’s no investigative standards to tell you exactly where the finish line is. My rule of thumbs:
Most importantly:
It’s problematic to end an investigation too early, but this judgment can’t be avoided without knowing what you don’t know.
That’s just a hard thing about bad situations.
Good luck!