Security teams are managed similarly to other engineering teams. However, differences exist when considering the risk based goals a security team pursues.

  • Hiring: Who the right will be, and when it makes sense to hire them.
  • Leadership: Qualities that might be desirable in a security leader.
  • Budget: Reasoning through budget discussions with a security team.
  • Planning: Encouraging risk based knowledge workers to offer direction.
  • Performance: Framing the performance problem of risk based knowledge work.

This section is designed to provide context to someone who is already familiar with basic org management concepts. I’m aware that security teams are often founded with brand new managers. If you’re brand new to any kind of leadership, you might want to supplement your reading further.

Hiring

“When do we hire a security person?” is becoming more difficult to answer.

Let’s discuss optimal time frames rather than setting a precise due date for that first hire.

Hire a team, not a person.

The right hiring attitude matters. Security functions should grow with risks. Risks grow at least in proportion to the business. A plan to hire a single individual will not keep up with the demands of that growth over time, and is shortsighted.

A lot of progress towards the [fundamentals] can be made without a security team. A security team is necessary to go beyond the fundamentals, and a single individual will never be enough.

Otherwise, reconsider if you’re able to continue mitigating the [fundamentals] them without hiring a security team to flag areas that require exceptional mitigations.

More on this here.

Task management is a reliable value across the workplaces cultures.

Security teams often work horizontally across organizations. A new hire will have an easier time collaborating on risk findings if there are smooth expectations in prioritizing, assigning, or collaborating on work with others in separate organizations. A new hire will desire some level of conversation and prioritization setting on their work compared to work generated by the business. Having this centralized through existing task management norms will reduce the friction around these decisions.

Early security teams often find more issues than can immediately be fixed. This backlog makes discipline around task tracking of a bit more importance. Issues that are not an immediate focus do not disappear; they just become a part of a backlog that needs to revisit, especially as resources grow or become free.

There are clear partners across the company.

Security teams bring together many horizontal organizational concerns and are notoriously stretched thin across them as a result. Different organizational concepts (HR, IT, Infrastructure, Facilities) come with their unique risks. Security may have to develop foundational technology in these organizations.

For instance, a security team will likely want to patch vulnerabilities or request specific configuration to employee laptops. Centralized laptop management is often the responsibility of an IT organization that security depends on. If no such organization exists to depend on, security may end up owning these organizations to pursue their goals. The question is… should it?

A partnership with an existing team is more efficient but not required. Some security teams do end up owning functions commonly owned by other team models.

Budget is available to support a security employee.

Security teams have budget needs that are unlikely to be covered by an existing engineering budget. There is no right answer to how much budget to expect a security team to have.

Instead, a budget discussion is a great topic to discuss in an interview with a founding team member. A discussion about budget helps set expectations between a candidate and a business about what sort of support will be available. A smaller security team is more limited without external resources like vendors, software licenses, or consulting.

A separate, potentially more risky approach is to bring on a team member full time to explore and advise on what opportunities exist for investment. This approach assumes that a speculative amount of budget will be available to pursue their strategy.

Lastly, discuss where to share the burden of a security budget with other organizations with a founding candidate.

Self-organization has begun around security responsibilities.

There should already be ad-hoc security work in flight. Small companies make security progress before filling a dedicated security headcount. Earlier teams are making increasingly greater progress on security as infrastructure platforms innovate.

For example, centralized authentication, logging, and secrets/credential storage often have modest-to-impressive progress before a security team comes along. A candidate can sense a shared responsibility around security if these workstreams have already begun. Shared responsibility avoids a situation where a company expects new talent to shoulder security. Security is not a vertical set of tasks. Rather, security is best shared horizontally across an organization.

Recruiting and belonging

Organizations want candidates to say “yes” as often as possible. The following concepts increase the lure to candidates considering your organization.

An organization should try to make clear (and compete on) the following needs a candidate will be looking to satisfy.

Money: At the time of this writing, the “security” specialty often earns a higher wage than other IT or engineering careers. However, “salary bands” have been an area of difficulty for tech companies and security engineers. Confusion is common in determining whether software engineers are working as security engineers or if some security roles can be filled by non-engineering talent. Be clear about salary bands when recruiting top engineering talent, and be well informed on the market you compete with.

Recognition: The security community rewards its members for contributions in the space. A healthy conference circuit and community ecosystem give many opportunities to share. Recognition is particularly valuable for some members of the community. The freedom to travel or speak publicly is often a requirement for some candidates. Thankfully, there are benefits to most organizations by supporting community involvement. Just set expectations with candidates on travel and expenses to effectively weed out boondogglers.

Curiosity: Candidates are in a state of constant learning. New platforms or products are very attractive for candidates looking to keep their skills fresh and sharp. Young companies have an additional advantage; To expose business units more transparently to candidates. Growing companies are often more interesting to candidates hoping to learn how a business works by watching it grow and evolve from the early days.

Challenges: Knowledge workers are attracted to difficult challenges. Security candidates are no exception. Larger customer bases, high-risk data, well-resourced adversaries, complex organizations, or a high expectation of incidents are all things that should make an employer more interesting to a security candidate. Though, the candidate must be supported.

Altruism: A desire to protect others has been a core part of the security community from its earliest beginnings. A clear narrative around an organization’s positive impacts are always important. A security candidate will be lured by mitigating the risks an organization may generate and impose on society.

Leadership

There comes a time when leadership skillsets become useful on a security team. Small teams may be hungry for a voice of prioritization that is familiar with their expected work. Larger organizations will need leadership that has experience forecasting headcount and budget, recruiting, setting direction and communicating that direction.

Before hiring a leader

As discussed earlier: A premature leadership hire for a small team may take a headcount away from much-needed individual contribution. Rather than hiring senior leadership, some amount of work direction can come from best practices, checklists, and maturity models in the early stages.

When to hire a clear leader

A security team can do well without subject matter expertise in a leadership role in the earliest phases. Though, unexpected needs will eventually arise. Some examples:

  • Leaders in other organizations have become resistant to the work involved with mitigations.
  • Prioritization of risks and mitigations have become more contentious.
  • “Team of Teams” appear on the horizon with a more complex budgeting process.
  • Re-organizations have fragmented relationships between organizations by locations, products, or customers.
  • Longer-term, multi-person technical mitigation efforts begin.

These may indicate the time to hire leadership. Leaders come with many titles: a title may be a manager, director, or “Chief Security Officer” or nothing at all.

Hiring the CSO

The Chief Security Officer is often the go-to title for security leadership. These roles are rarely in the “C-Suite”. They may even be far removed from them. Manager, director, or vice-president are additional options for titles for early leadership.

This writing won’t discuss titling very much. Titles are often something that comes with an organization. Rather, the appealing traits of a leader in a security team are more important.

They’re a fit.

An organization normally has some way of determining “fit” for incoming candidates. Security employees may violate typical hiring guidelines under some circumstances, which we’ll discuss.

First, security candidates can often spin very convincing horror stories about how bad your security can be, how evil and scary threats can be, and how good they are at breaking into systems or companies, and so on. The security industry describes this behavior as FUD: Fear, Uncertainty, and Doubt. Don’t be fooled: those badass-hacker candidates are safe to avoid.

Second, you may someday find yourself in an urgent sprint to hire someone due to an ongoing breach or a customer need. Be very careful to maintain your organization’s hiring standards in a crisis.

Questions:

  • Will the candidate, be able to work in the environment?
  • What are the candidate’s views on leadership?
  • Does a candidate’s management expectations differ from company philosophy?
  • What are their motivations to secure your company?
  • Why pursue a security career?

They confront skeletons.

Bad candidates will be very excited to start building before they know what’s going on at your organization. Proper talent will want to become familiar with three things:

  • Risks to the organizations
  • Deficits of trusts to the customer
  • Governance requirements

An adequate understanding of these topics may take some onboarding time.

Questions:

  • What is your first week going to look like?
  • What information will you need, first?
  • What risks do you expect you’ll find first?
  • How would you approach these “x” risks?

They can recruit talent.

Hiring leadership is a commitment to maintaining an organization over the long term.

Performance management and growth will require leadership to be involved with recruiting. Security has notable differences from other organizations, mostly in terms of mission.

First, security industry professionals are often motivated by aspects of the security industry, and less motivated by their employer. Their employer is a reliable paycheck and a host to develop themselves professionally. A leader will connect the security team’s interests and mission to the larger organization to better recruit fully engaged talent.

Second, security teams are hugely dependent on reliable relationships with other teams within an organization. There may be efficiencies in building out other organizations before hiring into security, and working to develop partnerships.

Lastly, top tier security engineering is competitive. Massive tech companies in the Bay Area compete for seemingly all talent. This may change with the recent COVID-19 shakeup in progress as of this writing. Does a candidate rely on traditional sourcing, or do they have more creative approches?

Questions:

  • What’s your plan for team building?
  • What attracts you to our mission, and how would you explain our mission?
  • What areas of our org are critical to your success?
  • How do you partner and get along with those organizations?

They can build solutions with fragmented problems.

Security professionals often have a mental mapping of Risk -> Mitigation they can access pretty quickly. In discussion with a candidate, they may effectively suggest mitigations for any type of risk you suggest. While impressive, this is not necessarily an indication they can implement these mitigations in your environment.

Security often runs into trouble when implementing company-wide mitigations. In particular, anything that changes an engineers’s preferred workflow.

Candidates that can bring experience with complex implementations are valuable. Leadership skills will shine while advising complex rollout plans for mitigations that impact a work environment.

Questions:

  • Can you describe how your previous workplaces complicated a common security practice?
  • What mitigations that should have been simple… ended up being headaches?
  • What risks do you feel are the hardest to mitigate properly, and for what reason?
  • What areas do common solutions lack?

They can take risks.

Risk-taking may seem counter-intuitive in a profession focused on reducing risk. Consider the statement:

We can avoid all risks by taking no risks at all.

Security teams often have this impression of a gatekeeper. Modern teams are trying to shake this image. Gatekeeping can come from veto power or by imposing undue friction on development.

Discuss how a candidate develops and exerts influence in the organization they work in.

Healthy attitudes are welcoming of innovation, ideas, and approaches. They’re able to view the increase in work from the risks as an exciting part of the business they can assist with.

Questions:

  • In what cases have you vetoed a project?
  • What causes you to vehemently take sides?
  • What sort of conflicts have existed between your team and others?
  • To what extent should an insecure product, system, etc., be supported by security?

They can lead an incident response.

Security is fundamentally about risk. Risk only exists if bad outcomes are possible. All organizations immediately draw upon their leadership when incidents arrive. A security team eventually needs to supply this leadership.

Leadership should be well connected with an industry rolodex. Connections include hard contacts for a contract (legal counsel, forensics), law enforcement, or soft contacts with other experienced incident responders and researchers that collaborate on known threat areas.

Candidates often claim to have been involved in security incidents in the past. It’s important to gauge what sort of involvement they had in the activities of the response.

Questions:

  • How do you support a team that is responding to an incident?
  • How have you ever prepared a team for a breach, emotionally?
  • Have you ever run a Red Team to simulate an incident?
  • What types of incident were difficult to respond to?
  • What role did you fill during a response?

They can balance work.

Leadership roles are often the sources of prioritization for work. As we have often discussed, we framed security work into several different models:

For instance, our foundation of risk, trust, and governance:

  • Risk: Prioritizing work to mitigate the consequences of an undesirable event.
  • Trust: Prioritizing work that develops trust among our customers or peers.
  • Governance: Prioritizing work that meets certain compliance objectives.

Leaders may appear fascinated with a subset of these. For instance, they are working on public-facing blog posts and speaking engagements over risk mitigation. Or, they push for rough compliance and audit experiences with a narrow focus on some certain risk area. Try to surface these imbalances.

Additionally, leadership should understand how work flows in and out of a security organization. The previously discussed work model helps us identify rough patches:

  • Business Projects: How does security help with new business efforts?
  • Operations: What ongoing tasks is the security team committed to accomplishing?
  • Engineering: How does security self-invest in better capabilities?
  • Incidents and Unplanned: What takes attention from planned work?

Some leadership will drown themselves in operational work. Others cannot escape last-minute business changes, chasing them endlessly. Most are unable to escape the work they’ve buried themselves under.

A candidate who is comfortable making these prioritization decisions should be very aware of how much work their commitments create. Additionally, making investments to handle more responsibility in the future.

Questions:

  • When would you prioritize compliance objectives over risk mitigation?
  • Have customers ever celebrated the work you’ve accomplished?
  • How have you reduced operational toil in the past?

They can build technology.

The security industry changes rapidly. A time-lag exists between the development of new threats and the availability of vendors to mitigate them well. As a result, organizations may need to develop their security mitigations from scratch or tie vendors together.

Additionally, business projects may require specific technical knowledge or engineering skillsets to participate in mitigation work.

In-house mitigation work may require leadership to be familiar with software engineering environments. If so, it’s best to maintain the engineering bar with this candidate you would have in existing engineering organizations. The desire to lower the bar will be strong. Avoid having a security organization create more toil than they manage. This will make organizational relationships deteriorate quickly.

Questions:

  • What have you built to reduce risk when no product was available to help you?
  • How do you approach “build vs buy” decision making?
  • What engineering patterns will you cultivate?

Security Budget Planning

The following themes are common touchpoints in discussions about what a security team needs and how it will grow to accomplish its goals.

Resource estimation must consider the growth of an organization. The size of an organization influences how smoothly a security team can grow. How large will everything else be in 6mo?

Early startups have very choppy growth patterns as they manage their runway with fundraising and revenue, along with rapidly developing initiatives or products. Here, the question is often “when do we start” security.

Late-stage businesses try to achieve predictability through regular planning cycles and steady growth of existing efforts. Now, the question is “how large is security compared to the growth of everything else”?

Security teams in either phase still require leadership decision making about budget and headcount. A security team wants to have healthy influence over this decision-making process to represent their needs.

Regular Growth: Tracking the business

A simple rule of thumb is that an increase in risks should result in some increase in risk mitigations. A growing organization will inherently face greater risks, and therefore, should steadily increase investments in mitigations proportionate to that growth.

Security teams often track a set of indicators related to a companies growth. These indicators are selected to tell a story about what risks an organization expects. Take the following risk, for example:

Malware steals data from an employee laptop after being executed from a spearphish.

This example risk might associate with indicator metrics. Indicators inform a security team’s budget or headcount to help mitigate that risk. For instance, the following metrics may inform us about the previous risk scenario:

  • Number of employees
  • Number of devices
  • Number of “suspicious email” escalations to IT

Likewise, a single indicator may be an informative predictor for multiple risks. An increased amount of employees may also increase the risks of an insider threat, a stolen laptop, (etc), in addition to seeing an employee compromised by a spearphish.

Organizations differ on whether certain indicators are agreeably tied to risk and elevate them differently. Leadership is chosen to represent the current state of beliefs.

The trends of indicators develop a baseline for headcount and budget forecasting. However, this approach can never fully represent the growth of risks. There are always additional discussions. A business’s activity warrants growth outside of regular tracking with the business and simple indicators are not enough.

Additional Growth: Managing outlier risks

Security teams are great at spotting outlier sources of risk. Regular indicators are only partially representative of a whole landscape of risk. Here is an example indicator to demonstrate this gap using our previous method:

We have 27 new vendors, from 48 vendors to 75 vendors this year. 56.25%

Let’s assume a member of your security team has flagged one of these new vendors as especially problematic.

One of these vendors has access to all of our customer data! They only need about a third of that data to do their job. We have no way of containing this without a major refactor of how our customer data is stored. None of our other vendors are this dangerous!

Indicators may track broad growth trends, but the team will be obligated to surface outliers (like the vendor in this example) within this data.

The team can ask for a resources with approach that resembles a business pitch to investors. Rather than a strict focus on ROI, security teams bring in several points of value.

First, a security team demonstrates the problem:

Risks: Our vendor unknowingly exposes our data. A threat actor accesses it. Customers and regulators take civil actions against us. We have a consequential media cycle admitting to the failure and have elevated support issues with customers for weeks or months. We lose an unknown amount of customers.

Next, the team describes the mitigation:

Mitigations: Most of the data we pass to the vendor is unnecessary. We would refactor the upstream pipeline to reduce exposure and require they encrypt it at rest. We plan on adding detections for any pipeline regressions. If an engineer sends excess data to the vendor again, we can fix it again. Finally, new vendors shouldn’t suddenly appear in our customer transaction pipeline. We need to onboard them before we send any data to them.

Lastly, the resources required:

Resources:

  • We need two “person-weeks” of engineering time.
  • Our cloud storage costs will go up after the refactor.
  • We need one new headcount, going forward, to work with new vendors.

This style of pitch is seen outside of planning cycles to manage large risk discoveries. Regularize budget planning but plan for flexibility when unknown events occur. Regularity is the aim of resource planning. Suprise requests for resources outside of it is unplanned work. Unplanned work is not necessarily bad, but constant reduction of unplanned work keeps us above water.

Negotiation and Debate

Indeed, a security team will often propose a plan that will require some amount of resources. The team, leadership, and partner teams will get into the debate about these plans and budgets. Debates will follow some security-specific touchpoints: risks, mitigations, and costs.

Here is an example to make these debates clear:

Risk: An adversary accesses our HR application with stolen employee credentials.

The team has identified a reasonable request. There is still room for discussion. You may ask to scope the risk wider by eliminating the “HR application” condition and speak about all SaaS applications your employees use. Or focus on a higher risk application or a whole other area of risk altogether. Lots of wiggle room.

Mitigations:: We require multi-factor access to the HR application.

While this mitigation is reasonable, other options may exist. Does this mean we have to manage multi-factor for every application? Should we use the application native MFA, or should we bring in a vendor?

Cost: We are retaining a defense contractor firm who will click the “enforce MFA” button in the HR application’s configuration.

OK, while we agree with the risk and the mitigation… the cost is unreasonable. Leadership shouldn’t toss out the whole plan when risk and mitigation are sound. Pushback is reasonable on the cost factor alone. While this example is a bit absurd, pushback may cause the risk and mitigation to re-scope to allow more efficient risk mitigation. Make sure agreement can happen on individual aspects to let a debate evolve.

Partnerships within the organization

Security teams do not work in isolation. Other organizations hold fundamental resources for a security team’s strategy. Resource conversations may often find budget or headcount applied to non-security organizations.

As an example, infrastructure budgets often have engineering resource to purchase and implement cloud-based security tooling or to manage remote authentication.

Another example would be IT budgets. IT is often responsible for the on/offboarding of employees and employee access to IT services. Central to this are single sign-on products, multi-factor products, and VPN. Non-security organizations plan for purchasing what we’d normally call “security” products.

Security teams should serve as advocates for other organizations in resource planning.

Planning

A security team will set a roadmap for themselves periodically. Work will be prioritized in this roadmap with the time available to complete it.

First, tasks originate from the interpretation of a mission and purpose, as discussed in foundations. These tasks are then balanced with practical management topics to deal with the complexity of working with a group. A single leadership role could technically write a roadmap of work for a team—however, this misunderstands the role of knowledge workers. Model risks through collaboration and a summation of disparate knowledge held by trusted talent.

A collaborative planning process allows for the maximum available expertise about an organization’s risks and opportunities to mitigate these risks for consideration. Knowledge workers are often motivated by this ability to be creative and define their own problem spaces. Risk, being a subjective topic, benefits from diverse viewpoints and collaboration between knowledge workers.

  • All security team members are expected to think critically about risks.
  • Team leadership should set broad direction and strategy.
  • A collaborative process captures this knowledge and formulates a directed roadmap.

The Objective and Key Result (OKR) is a work planning model that plays well with leading knowledge work, but not without some weaknesses. Leadership can use risk-based directions to encourage teams to form OKRs in those directions. OKR approaches assume that an organization hires the best and brightest talent and trusts their expertise.

The OKR is structured as a broad objective, along with 3 to 5 measurable key results that describe achievable outcomes from your efforts.

Let’s assume that leadership suggests that risks to intellectual property are a priority. They seek to influence (and reduce) the following risks to intellectual property in the company:

  • The probability of IP leaking from employee inboxes.
  • The probability that malware will beacon from an employee system and exfiltrate IP.
  • The probability that breached credentials, data, or systems owned by an employee will also result in leaked IP.

A talented group of security engineers can now look across these statements and consider their influence on these outcomes. With effort, they can reduce these probabilities.

You may begin to spot a problem. These are not directly measurable, in an objective sense.

In the mainstream OKR belief system, the key results focus on measurements of desired outcomes.

The best possible writing for a security key result would be objective, measurable reductions in risk probabilities.

Other engineering disciplines can directly measure their key results (perf, customer growth, revenue, etc.). However, risk has roots in probability. Our teams cannot objectively measure “ Probabilistic” outcomes. They can only forecast them, which is subjective.

Forecasting would measure the before and after impact on these key results. We’re unlikely to pursue strict forecasting methods with an early team. That’s ok. We’re still ok. Most organizations don’t use OKRs this strictly. We can still use the desired outcomes (reducing well-specified risks) as part of our leadership strategy to lead the team.

Security teams often focus their key results on obtainable, actionable goals around a risk-based objective. Part of the objective discussion can include specific risk targets that discuss scenarios. Here are the previous key results, except they’re used to further describe the objective.

Objective:

  • Reduce risk to intellectual property.
    • The probability of IP leaking from employee inboxes.
    • The probability that malware will beacon from an employee system and exfiltrate IP.
    • The probability that breached credentials, data, or systems owned by an employee will also result in leaked IP.

Key Results:

  1. All employees enrolled with multi-factor keys.
  2. All cloud applications solely accessible with Single-Sign-On
  3. Migrate employees to new app access workflow.

OKRs seem to serve better at a team level scope. The team can assess whether their efforts were impactful on those key results. If successful, they’ll move onto other risks. If unsuccessful, they’ll need to consider more depth in their mitigation.

Objectives should have defensible roots in risk, trust, or compliance goals.

OKRs set by security teams are unique in this regard. The team is trusted to set both the target of effort and the means of getting to it. Participants require maximum trust when setting OKRs. In addition to being trusted with suggesting risks, prioritizing them, and selecting mitigations, there is a significant dependence on internal partnerships within an organization to meet these goals. Security teams work horizontally in an organization to mitigate risk.

With such wide allowances to suggest their goal-setting, security teams must suggest objectives that are achievable. Security teams have total accountability for meeting the objectives that they are trusted to build.

Lastly, let’s further discuss some weaknesses with OKRs. Common beliefs about OKRs include two features that are worth discussing. These features make discussions with a security team about OKRs surprisingly complicated. Risk-based OKRs expose weaknesses in OKRs. OKRs only work well with reasonable leadership and with forgiveness in the areas that OKRs are weak.

The first is that OKRs should be cascading. OKRs could link from leadership organizations down to the contributing teams in the reporting chain. Teams hold the cascading quality of OKRs less firmly these days. Security teams have good examples of where a crisp cascading relationship with goals may appear to break down on the surface.

An example breakdown of cascading: An organizational key result may include a product or customer growth. However, an anti-spam organization may delete malicious customer accounts and slow that growth. These are, of course, competing. The security objectives may not cascade very cleanly with the quality of certain metrics, especially from metrics above all else.

Second is that OKRs are not meant to be a clear proxy for performance management, especially when operating at a team level (as suggested). Performance assessments are difficult, with the risk measurement friction dragging us down. Let’s discuss this further.

Performance

It’s not easy tying an individual’s performance to a pure, risk-based key result. A risk based key result is evasive due to its subjectivity. Performance measurement on top of this foundation is tricky, and OKRs are not suited towards individual performance measurement as a result.

OKRs are beneficial because they encourage a team we trust to define their objectives as well as the path towards it.

First, expertise must come with trust. Dedicated security employees should be able to define their objectives and the path toward them without significant hand holding - this is a large part of the job. Junior members may struggle with this and need direction. High performing employees can identify achievable OKRs and make regular progress towards them.

Second, the team should not constantly fluctuate Key results with conveniently available pseudo-justifications they should have foreseen. Key results can fluctuate under normal circumstances, but not at every sign of trouble. Written OKR philosophy preaches a 60-70% hit rate for key result completion.

Lastly, experts should be comfortable in a feedback process with their peers. Knowledge work environments push strategy and planning downward. An expert’s peers will likely have the best available context for providing feedback. “Performance summary cycles” are a common management pattern in the Bay Area (for better or worse), with a plethora of literature available. Peer feedback environments are more pronounced where opportunities for direct, objective measurement lack.

When risk measurement and performance coupled too tightly, it creates a circle of absurdity: Hire risk experts to tell us good news and reward them increasingly when doing so. Risk-based OKRs must focus more on achievable work, rather than risk measurement. We must trust the best possible talent we can hire to set and accomplish them. Performance assessment is limited to more qualitative approaches as a result. Self-management, reliable estimation, and peer feedback loops make up a core part of this approach. There is simply no constant “quota” for knowledge work, unless a knowledge worker has set one for themselves in the time being.