Security teams are managed similarly to other engineering teams. However, differences exist when considering the risk based goals a security team pursues.
- Hiring: Who makes sense to hire when the time is right.
- Leadership: Qualities that might be desirable in a security leader.
- Budget: Reasoning through budget discussions with a security team.
- Planning: Encouraging risk based knowledge workers to suggest direction.
- Performance: Framing the performance problem of risk based knowledge work.
This section helps an organization familiarize itself with the knots of managing a security team.
“When do we hire a security person?” is becoming more difficult to answer.
A different perspective: There are effective time frames to land the first hire.
An early hire will have a vast horizontal influence in an organization. Wherever risk appears, security work appears. A security team is involved in nearly every aspect of an organization. The earliest hires will find risks throughout a company and will begin producing work.
Here are some concepts to consider when deciding on the first security hire. You may notice that some of these are relevant for future hires, too.
Task management is a substantial part of the workplace culture.
Security teams often work horizontally across organizations. A new hire will have an easier time collaborating on risk findings if there are smooth expectations in prioritizing, assigning, or collaborating on work with others in separate organizations. A new hire will desire some level of conversation and prioritization setting on their work compared to work generated by the business. Having this centralized through existing task management norms will reduce the friction around these decisions.
Early security teams often find more issues than can immediately be fixed. This backlog makes discipline around task tracking of a bit more importance. Issues that are not an immediate focus do not disappear; they just become a part of a backlog that needs to revisit, especially as resources grow or become free.
There are clear partners across the company.
Security teams bring together many horizontal organizational concerns and are notoriously stretched thin across them as a result. Different organizational concepts (HR, IT, Infrastructure, Facilities) come with their unique risks. Security may have to develop foundational technology in these organizations just to accomplish familiar risks.
For instance, a security team will likely want to patch vulnerabilities or request configuration to employee laptops. Centralized laptop management is often the responsibility of an IT organization that security depends on. If no such organization exists to depend on, security may end up owning these organizations to pursue their goals.
A partnership with an existing team is more efficient but not required. Some security teams do end up owning functions commonly owned by other team models.
Budget is available to support a security employee.
Security teams have budget needs that are unlikely to be covered by an existing engineering budget. There is no right answer to how much budget to expect a security team to have. Instead, a budget discussion is a great topic to discuss in an interview with a founding team member. A discussion about budget helps set expectations between a candidate and a business about what sort of support will be available. A smaller security team is more limited without external resources like vendors, software licenses, or consulting. A separate, potentially more risky approach is to bring on a team member full time to explore what opportunities exist for investment. This approach assumes that a speculative amount of budget will be available to mitigate their findings. Discuss where to share the burden of a security budget with other organizations with candidates.
Self-organization has begun around security responsibilities.
There should already be ad-hoc security work in flight. Small companies make security progress before filling a dedicated security headcount. Earlier teams make increasingly greater progress on security as infrastructure platforms innovate. For example, centralized authentication, logging, and secrets/credential storage often have modest progress before a security team comes along. These developments are recent progress. A candidate can sense a shared responsibility around security if these workstreams have already begun. Shared responsibility avoids a situation where a company expects new talent to shoulder security. Security is not a vertical set of tasks. Rather, security is best shared horizontally across an organization.
Recruiting and belonging
Organizations want candidates to say “yes” as often as possible. The following concepts increase the lure of joining your organization.
An organization should try to make clear (and compete on) the following needs a candidate will be looking to satisfy.
Money: At the time of this writing, the “security” specialty often earns a higher wage than other IT or engineering careers. However, “salary bands” have been an area of difficulty for tech companies and security engineers. Confusion is common in determining whether software engineers are working as security engineers or if some security roles require less technical skill sets that non-engineering talent can fill. Be clear about salary bands when recruiting top engineering talent.
Recognition: The security community rewards its members for contributions in the space. A healthy conference circuit and community ecosystem give many opportunities to share. Recognition is particularly valuable for some members of the community. The freedom to travel or speak publicly is often a requirement for some candidates. There should be benefits to most organizations by supporting community involvement. However, be sure to weed out boondogglers. Set expectations with candidates.
Curiosity: Candidates are in a state of constant learning. New platforms, technologies, or products are very attractive for candidates looking to keep their skills fresh and sharp. In addition to their desire to learn new technology, young companies have an additional offering to expose business units more transparently to candidates. Growing companies are often more interesting to candidates hoping to learn how a business works by watching it grow and evolve.
Challenges: Knowledge workers are attracted to difficult challenges. Security candidates are no exception. Larger customer bases, high-risk data, well-resourced adversaries, complex organizations, or a high expectation of incidents are all things that should make an employer more interesting to a security candidate.
Altruism: A desire to protect others has been a core part of the security community from its earliest beginnings. This desire is steadily growing as the role of security becomes more prominent in organizations and greater society. A clear narrative around an organization’s positive impact is always important, regardless of security candidacy. A security candidate will look to mitigate the risks an organization may generate and impose on society.
There comes a time when leadership skill sets become useful on a security team. Small teams may be hungry for a voice of prioritization that is familiar with their expected work. Larger teams and organizations will need leadership with a wider range of experience.
Before hiring a leader
As discussed earlier: A premature leadership hire for a small team may take a headcount away from much-needed individual contribution. Rather than hiring senior leadership, some amount of work direction can come from best practices, checklists, and maturity models in the early stages.
When to hire a clear leader
A security team can do well without subject matter expertise in a leadership role in the earliest phases. Though, unexpected needs will eventually arise. Some examples:
- Leaders in other organizations become resistant to the work involved with mitigation.
- Prioritization of risks and mitigations become more contentious.
- “Team of Teams” appears on the horizon with a complex budgeting process.
- Re-organizations fragment relationships between organizations by locations, products, or customers.
- Longer-term, multi-person technical efforts begin.
It may be time to hire a leader. Leaders come with many titles: a title may be a manager, director, or “Chief Security Officer” or nothing at all.
Hiring the CSO
The Chief Security Officer is often the go-to title for security leadership. These roles are rarely in the “C-Suite” and are often far removed from it. Manager, director, or vice-president are options for titles for early leadership.
Titles are often something that comes with an organization. This writing won’t discuss titling very much. Rather, the appealing traits of a leader in a security team are more important.
They’re a fit.
An organization normally has some way of determining “fit” for incoming candidates. Security employees may violate typical hiring guidelines under some circumstances.
First, security candidates can often spin very convincing horror stories about how bad your security can be, how evil and scary threats can be, and how good they are at breaking into systems or companies, and so on. The security industry describes this behavior as FUD: Fear, Uncertainty, and Doubt. Don’t be fooled: badass-hacker candidates are safe to avoid.
Second, you may be in an urgent sprint to find someone due to an ongoing breach or a customer need. Be very careful to maintain your organization’s hiring standards in a crisis.
- Will the candidate, be able to work in the environment?
- What are the candidate’s views on leadership?
- Does a candidate’s management expectations differ from company philosophy?
- What are their motivations to secure your company?
- Why pursue a security career?
They confront skeletons.
Bad candidates will be very excited to start building before they know what’s going on at your organization. Proper talent will want to become familiar with three things:
- Risks to the organizations
- Deficits of trusts to the customer
- Governance requirements
An adequate understanding of these topics may take some onboarding time.
- What is your first week going to look like?
- What information you need to have access to or know?
- What risks will you tackle first?
- How would you approach these “x” risks?
They can recruit talent.
Hiring leadership is also a commitment to maintaining an organization over the long term.
Performance management (terminations) or growth will require leadership to be involved with recruiting. Security has notable differences from other organizations, mostly in terms of mission.
First, security industry professionals are often motivated by aspects of the security industry. Their employer is a regular paycheck while they develop professionally. A proper leader will connect the security team’s interests and mission to the larger organization.
Second, security teams are hugely dependent on reliable relationships with other teams within an organization. There may be efficiencies in building out other organizations before security and working to partner with them.
Lastly, top tier security engineering is currently hiring drought. Massive tech companies in the Bay Area compete for seemingly all talent. The drought may change with the recent COVID-19 shakeup in progress as of this writing. A strong recruiter background can be a valuable trait a leader can bring to the table.
- What’s your plan for team building?
- What attracts you to our mission, and how would you explain our mission?
- What areas of our org are critical to your success?
- How do you partner and get along with those organizations?
They can build solutions with fragmented problems.
Security professionals often have a mental mapping of
Risk -> Mitigation they can access pretty quickly. In discussion with a candidate, they may prove to suggest mitigations for any type of risk you suggest. While impressive, this is not necessarily an indication they can implement these mitigations in your environment.
Security often runs into trouble when implementing company-wide mitigations. In particular, anything that changes an employee’s workflow.
Candidates that can bring experience with the project management side of implementations are valuable. It’s often a leadership role to advise rollout plans for mitigations that impact a work environment.
- Can you describe how your previous workplaces complicated a typical security practice?
- What mitigations that should have been simple ended up being headaches?
- What risks do you feel are the hardest to mitigate properly, and for what reason?
- What areas do common solutions lack?
They can take risks.
Risk-taking may seem counter-intuitive in a profession focused on reducing risk. Consider the statement:
We can avoid all risks by taking no risks at all.
Security teams often have this impression of a gatekeeper. Modern teams are trying to shake this image. Gatekeeping can come from veto power or from having imposed undue friction on development.
Discuss how a candidate develops and exerts influence in the organization they work in.
Healthy attitudes are welcoming of innovation, ideas, and approaches. They’re able to view the increase in work from the risks as an exciting part of the business they can assist with.
- In what cases have you vetoed a project?
- What causes you to vehemently side against another team?
- What sort of conflicts existed between your team and others?
- To what extent should an insecure product, system, etc., be supported by security?
They can lead an incident response.
Security is fundamentally about risk. Risk only exists if bad outcomes are possible. All organizations immediately draw upon leadership when incidents arrive. A security team eventually needs to supply this leadership.
Additionally, leadership should be well connected with an industry rolodex. Connections include hard contacts for a contract (legal counsel, forensics), law enforcement, or soft contacts with other experienced incident responders and researchers that collaborate on known threat areas.
Candidates often claim to have been involved in security incidents in the past. It’s important to gauge what sort of involvement they had in the activities of the response.
- How do you support a team during a response?
- How have you prepared a team for a breach, emotionally?
- Have you ever run a Red Team exercise?
- What methods of intrusion have you experienced?
- What role did you fill during a response?
They can balance work.
Leadership roles are often the sources of prioritization for work. As we have often discussed, we framed security work into several different models:
For instance, our foundation of risk, trust, and governance:
- Risk: Prioritizing work to mitigate the consequences of an undesirable event.
- Trust: Prioritizing work that develops trust among our customers or peers.
- Governance: Prioritizing work that meets certain compliance objectives.
Leaders can appear fascinated with only a subset of these. For instance, they are working on public-facing blog posts and speaking engagements over risk mitigation. Or allowing for rough compliance audit experiences with a narrow focus on some certain risk area.
Additionally, leadership should understand how workflows in and out of a security organization. The previously discussed work model helps us identify rough patches:
- Business Projects: How does security help with new business efforts?
- Operations: What ongoing tasks is the security team committed to accomplishing?
- Engineering: How does security invest in greater capabilities?
- Incidents and Unplanned: What takes attention from planned work?
Some leadership will drown themselves in operational work. Others cannot escape last-minute business changes and chase them endlessly. Most are unable to get their heads above water.
The ability to prioritize the needs of a team is the concern. The needs of a team are different than risks. Sometimes, the team’s capabilities balance with the risks that need mitigation.
- When would you prioritize compliance objectives over risk mitigation?
- Have customers ever celebrated the work you’ve accomplished?
- How have you reduced operational toil in the past?
They can build technology.
The security industry changes rapidly. A time-lag exists between the development of new threats and the availability of vendors to mitigate them well. As a result, organizations may need to develop their security mitigations from scratch or tie vendors together.
Additionally, business projects may require specific technical knowledge or engineering skillsets to participate in mitigation work.
In-house mitigation work may require leadership to be familiar with software engineering environments. If so, it’s best to maintain the engineering bar with this candidate you would have in existing engineering organizations. The desire to lower the bar will be strong. Avoiding frustrating relationships due to a skills gap is worth it. Having a security organization create more toil than they manage will make relationships deteriorate.
- What have you built to reduce risk when no product was available to help you?
- Have you ever managed an engineering organization?
- What is your engineering background like, as far as building defenses or software?
Security Budget Planning
The following themes are common touchpoints in discussions about what a security team needs and how it will grow to accomplish its goals.
Resource estimation must consider the growth of an organization. The size of an organization influences how smoothly a security team can grow.
Early startups have very choppy growth patterns as they manage their runway with fundraising and revenue, along with rapidly developing initiatives or products. Late-stage businesses try to achieve predictability through regular planning cycles and steady growth of existing efforts.
Security teams in early or late phases still require leadership decision making about budget and headcount. A security team wants to have healthy influence over this decision-making process to represent their needs.
Regular Growth: Tracking the business
A simple rule of thumb is that an increase in risks should result in some increase in risk mitigations. A growing organization will inherently face greater risks, and therefore, should steadily increase investments in mitigations.
Security teams often track a set of indicators related to a companies growth. These indicators are selected to tell a story about what risks an organization expects. Take the following risk, for example:
Malware steals data from an employee laptop after being executed from a spearphish.
A risk might associate with indicator metrics. Indicators inform a security team’s budget or headcount to help mitigate that risk. For instance, the following metrics may inform us about the previous risk scenario:
- Number of employees
- Number of devices
- Number of “suspicious email” escalations to IT
Likewise, a single indicator may be an informative predictor for multiple risks. An increased amount of employees may also increase the risks of an insider threat, in addition to seeing an employee compromised by a spearphish.
Organizations differ on whether certain indicators are agreeably tied to risk and elevate them differently. These indicators are negotiated between security teams and their leadership and chosen to represent the current state of beliefs.
The trends of indicators develop a baseline for financial forecasting. However, this approach can never fully represent the growth of risks. There are always additional discussions. A business’s activity warrants growth outside of regular tracking with the business and simple indicators are not enough.
Additional Growth: Managing outlier risks
Security teams are great at spotting outlier sources of risk. Indicators are only partially representative of a whole landscape of risk. Gaps become obvious to security professionals when indicators are focused on too closely. Here is an example indicator to demonstrate this gap using our previous method:
We have 27 new vendors, from 48 vendors to 75 vendors this year. 56.25%
Let’s assume a member of your security team flags one of these new vendors as especially problematic.
One of these vendors has access to all of our customer data! They only need about a third of that data to do their job. We have no way of containing this without a major refactor of how our customer data is stored. None of our other vendors are this dangerous!
Indicators may track broad growth trends, but the team will be obligated to surface outliers (like the vendor in this example) within this data.
The team can ask for a resource with an approach that resembles a business pitch to investors. Rather than a strict focus on ROI, security teams bring in several points of value.
First, a security team demonstrates the problem:
Risks: Our vendor unknowingly exposes our data. A threat actor accesses it. Customers and regulators take civil actions against us. We have a consequential media cycle admitting to the failure and have elevated support issues with customers for weeks or months. We lose an unknown amount of customers.
Next, the team describes the mitigation:
Mitigations: Most of the data we pass to the vendor is unnecessary. We would refactor the upstream pipeline to reduce exposure and require they encrypt it at rest. We plan on adding detections for any pipeline regressions. If an engineer sends excess data to the vendor again, we can fix it again. Finally, new vendors shouldn’t suddenly appear in our customer transaction pipeline. We need to onboard them before we send any data to them.
Lastly, the resources required:
- We need two “person-weeks” of engineering time.
- Our cloud storage costs will go up after the refactor.
- We need one new headcount, going forward, to work with new vendors.
This style of pitch is seen outside of planning cycles to manage large risk discoveries. Regularize planning but offer flexibility when unknown events occur. Regularity is the aim of resource planning. Suprise requests for resources outside of it should is unplanned work. Unplanned work is not necessarily bad, but the goal is to reduce it.
Negotiation and Debate
A security team will often propose a plan that will require some amount of resources. The team, leadership, and partner teams get into the debate about these plans and budgets. Debates will follow some security-specific touchpoints: risks, mitigations, and costs.
Here is an example to make these debates clear:
Risk: An adversary accesses our HR application with stolen employee credentials.
The team has identified a reasonable request. There is still room for discussion. You may ask to scope the risk wider by eliminating the “HR application” condition and speak about all SaaS applications your employees use. Or focus on a higher risk application or a whole other area of risk altogether.
Mitigations:: We require multi-factor access to the HR application.
While this mitigation is reasonable, other options may exist. Does this mean we have to manage multi-factor for every application?
Cost: We are retaining a defense contractor firm to click the “enforce MFA” button in the HR application’s configuration.
We agree with the risk and the mitigation, but this cost is unreasonable. Leadership shouldn’t toss out the whole plan. Pushback is reasonable on this factor alone. While this example is a bit absurd, pushback may cause the risk and mitigation to re-scope to allow more efficient risk mitigation.
Partnerships within the organization
Security teams do not work in isolation. Other organizations hold fundamental resources for a security team’s strategy. Resource conversations may often find budget or headcount applied to non-security organizations.
As an example, infrastructure budgets often have engineering time to leverage cloud-based security tooling or headcount to manage remote authentication for development.
Another example would be IT budgets. IT is often responsible for the on/offboarding of employees and employee access to IT services. Central to this are single sign-on products, multi-factor products, and VPN. Non-security organizations plan for purchasing what we’d normally call “security” products.
Security teams should serve as advocates for other organizations in resource planning.
A security team will set a roadmap for themselves periodically. Work will be prioritized in this roadmap with the time available to complete it.
First, tasks originate from the interpretation of a mission and purpose, as discussed in foundations. These tasks are then balanced with practical management topics to deal with the complexity of working with a group. A single leadership role could technically write a roadmap of work for a team—however, this misunderstands the role of knowledge workers. Model risks through collaboration and a summation of disparate knowledge held by trusted peers.
A collaborative planning process allows for the maximum available expertise about an organization’s risks and opportunities to mitigate these risks for consideration. Knowledge workers are often motivated by this ability to be creative and define their own problem spaces. Risk, being a subjective topic, benefits from diverse viewpoints and collaboration between knowledge workers.
- All security team members are expected to think critically about risks.
- Team leadership should set broad direction and strategy.
- A collaborative process captures this knowledge and formulates a directed roadmap.
The Objective and Key Result (OKR) is a work planning model that plays well with leading knowledge work, but not without some weaknesses. Leadership can use risk-based directions to encourage teams to form OKRs in those directions. OKR approaches assume that an organization hires the best and brightest talent and trusts their expertise.
The OKR is structured as a broad objective, along with 3 to 5 measurable key results that describe achievable outcomes from your efforts.
Let’s assume that leadership has suggested that risks to intellectual property are a priority. They seek to influence (and reduce) the following risks to intellectual property in the company:
- The probability of IP leaking from employee inboxes.
- The probability that malware will beacon from an employee system and exfiltrate IP.
- The probability that breached credentials, data, or systems owned by an employee will also result in leaked IP.
A talented group of security engineers can now look across these statements and consider their influence on these outcomes. With effort, they can reduce these probabilities.
You may begin to spot a problem. These are not directly measurable, in an objective sense.
In the mainstream OKR belief system, the key results focus on measurements of desired outcomes.
The best possible writing for a security key result would be measurable reductions in risk probabilities.
Other engineering disciplines can directly measure their key results (perf, customer growth, revenue, etc.). However, risk has roots in probability. Our teams cannot objectively measure “ Probabilistic” outcomes. They can only forecast them.
Forecasting would measure the before and after impact on these key results. We’re unlikely to pursue strict forecasting methods with an early team. That’s ok. We’re still ok. Most organizations don’t use OKRs this strictly. We can still use the desired outcomes (reducing well-specified risks) as part of our leadership strategy to lead the team.
Security teams often focus their key results on obtainable, actionable goals around a risk-based objective. Part of the objective discussion can include specific risk targets that discuss scenarios. Here are the previous key results, except they’re used to further describe the objective.
- Reduce risk to intellectual property.
- The probability of IP leaking from employee inboxes.
- The probability that malware will beacon from an employee system and exfiltrate IP.
- The probability that breached credentials, data, or systems owned by an employee will also result in leaked IP.
- All employees enrolled with multi-factor keys.
- All cloud applications solely accessible with Single-Sign-On
- Migrate employees to new app access workflow.
OKRs seem to serve better at a team level scope. The team can assess whether their efforts were impactful on those key results. If successful, they’ll move onto other risks. If unsuccessful, they’ll need to consider more depth in their mitigation.
Objectives should have defensible roots in risk, trust, or compliance goals.
OKRs set by security teams are unique in this regard. The team is trusted to set both the target of effort and the means of getting to it. Participants require maximum trust when setting OKRs. In addition to being trusted with suggesting risks, prioritizing them, and selecting mitigations, there is a significant dependence on internal partnerships within an organization to meet these goals. Security teams work horizontally in an organization to mitigate risk.
With such wide allowances to suggest their goal-setting, security teams must suggest objectives that are achievable. Security teams have total accountability for meeting the objectives that they are trusted to build.
Lastly, let’s further discuss some weaknesses with OKRs. Common beliefs about OKRs include two features that are worth discussing. These features make discussions with a security team about OKRs surprisingly complicated. Risk-based OKRs expose weaknesses in OKRs. OKRs only work well with reasonable leadership and with forgiveness in the areas that OKRs are weak.
The first is that OKRs should be cascading. OKRs could link from leadership organizations down to the contributing teams in the reporting chain. Teams hold the cascading quality of OKRs less firmly these days. Security teams have good examples of where a crisp cascading relationship with goals may appear to break down on the surface.
An example breakdown of cascading: An organizational key result may include a product or customer growth. However, an anti-spam organization may delete malicious customer accounts and slow that growth. These are, of course, competing. The security objectives may not cascade very cleanly with the quality of certain metrics, especially from metrics above all else.
Second is that OKRs are not meant to be a clear proxy for performance management, especially when operating at a team level (as suggested). Performance assessments are difficult, with the risk measurement friction dragging us down. Let’s discuss this further.
It’s not easy tying an individual’s performance to a pure, risk-based key result. The risk based key result was already evasive for other purposes, and performance measurement is also tricky.
OKRs are not suited towards individual performance measurement, anyhow.
OKRs are beneficial because they encourage a team we must trust in defining their objectives and the path towards it. However, OKRs leave us with a more challenging performance problem.
First, expertise must come with trust. Dedicated security employees should be able to define their objectives and the path toward them without significant hand holding - this is a large part of the job. Junior members may struggle with this.
Second, the team should not constantly fluctuate Key results with conveniently available pseudo-justifications they should have foreseen. Key results can fluctuate under normal circumstances, but not at every sign of trouble. OKR philosophy preaches a 60-70% hit rate for key result completion.
Lastly, experts should be comfortable in a feedback process with their peers. Knowledge work environments push strategy and planning downward. An expert’s peers will likely have the best available context for providing feedback. “Performance summary cycles” are a common management pattern in the Bay Area (for better or worse), with a plethora of literature available. Peer feedback environments are more pronounced where opportunities for direct, objective measurement lack.
When risk measurement and performance couple tightly, it creates a circle of absurdity: Hire risk experts to tell us good news and reward them increasingly when doing so. Risk-based OKRs must focus more on achievable work and trust the best possible talent we can hire to set and accomplish them. Performance assessment is limited to more qualitative approaches as a result. Self-management, reliable estimation, and feedback loops make up a core part of this approach.