Security Department Performance Indicators

1. You are here:
2. Handbook
3. Security at GitLab
4. Security Department Performance Indicators

On this page

• Executive Summary
• Key Performance Indicators
  • Age of current open application and container vulnerabilities by severity
  • Security Engineer On-Call Page Volume
  • Security Control Health by System
  • Security Impact on Net ARR
  • Estimated Cost of Abuse
  • Security Team Member Retention
  • Security Average Age of Open Positions
• Regular Performance Indicators
  • Security Budget Plan vs Actuals
  • Security Handbook MR Rate
  • Security Department Discretionary Bonus Rate
  • Security Incidents by Category
  • Operational Security Risk Management (Tier 2 Risks)
  • Security Observations (Tier 3 Risks)
  • Third Party Risk Management
  • Security Automation Iteration Velocity Average
  • Security Department Promotion Rate
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Executive Summary

KPI	Health	Status
Age of current open application and container vulnerabilities by severity	Confidential	We continue to see sustained numbers below SLO targets. However for the month of January and February, S1 average days open is over the threshold due expected circumstances and not reflective of a negative trend. There has also been an slight increase in the average number of days that S2s, S3s, and S4s are staying open, however all are still below their respective targets. This KPI is currently considered healthy.
Security Engineer On-Call Page Volume	Confidential	A 7% increase in on-call pages was experienced over the last 3 months as compared to the last 12 months. Notably, this shows recent stability in the on-call page volume. The number of S1 paged incidents is zero over the last 3 months, compared to 1 S1 page per month over the last 12 months average. These volumes are well within the acceptable threshold and we attribute this stability to security improvements made through rapid action groups and working groups. Short term fluctuations are to be expected. Long term trends should be identified and actions should be taken to correct negative trends and to continue promoting positive trends.
Security Control Health by System	Confidential	Security Compliance performs regular testing of controls by system and uses the System Health Rating methodology to determine the overall health of each system in scope of GitLab’s compliance and regulatory programs. A System health rating of 1 means all controls evaluated for that system are operating fully effectively and a system health rating of 5 means none of the controls are operating effectively. 57% (or 13 out of 23) systems assessed met the target. There are 118 observations open for these 23 systems.
Security Impact on Net ARR	Confidential	As anticipated, Field Security observed a significant increase in customer assurance requests in January 2023, the last month of the fiscal year, resulting in the most amount of requests processed since inception of this KPI.
Estimated Cost of Abuse	Confidential	Confidential metric - See notes in Key Review agenda
Security Team Member Retention	Confidential	Action = Security leadership have created OKRs focused on addressing team member concerns contributing to attrition rates.
Security Average Age of Open Positions	Problem	High demand for Security professionals has lead to an extremely competitive hiring market. Security leadership are actively involved in the recruiting processes to reduce burden on recruiters and shorten time to hire.

Key Performance Indicators

Age of current open application and container vulnerabilities by severity

The age of current open vulnerabilities gives us an at the moment snapshot in time of how fast we are scheduling and fixing the vulnerabilities found post-Production deploy. The age is measured in days, and the targets for each severity are defined in the Security Handbook. For Security purposes, please view this chart directly in Sisense.

Target: Time to remediate

Health: Confidential

• We continue to see sustained numbers below SLO targets. However for the month of January and February, S1 average days open is over the threshold due expected circumstances and not reflective of a negative trend. There has also been an slight increase in the average number of days that S2s, S3s, and S4s are staying open, however all are still below their respective targets. This KPI is currently considered healthy.

URL(s)

• https://app.periscopedata.com/app/gitlab/758795/Appsec-Embedded-Dashboard

Security Engineer On-Call Page Volume

This metric is focused around the volume and severity of paged incidents to the Security Engineer On-Call. This data can be used to track and identify trends associated with disruption work, which if and when possible, should be minimized. For Security purposes, please view this chart directly in Sisense.

Target: Number of pages/month does not exceed +50% of monthly average of the last 12 months for 3 consecutive months

Health: Confidential

• A 7% increase in on-call pages was experienced over the last 3 months as compared to the last 12 months. Notably, this shows recent stability in the on-call page volume. The number of S1 paged incidents is zero over the last 3 months, compared to 1 S1 page per month over the last 12 months average. These volumes are well within the acceptable threshold and we attribute this stability to security improvements made through rapid action groups and working groups.
• Short term fluctuations are to be expected. Long term trends should be identified and actions should be taken to correct negative trends and to continue promoting positive trends.

URL(s)

• https://app.periscopedata.com/app/gitlab/592612/Security-KPIs?widget=9217413&udv=0

Security Control Health by System

Security Compliance performs regular testing of controls for in scope systems and uses the System Health Rating methodology to determine the system health, ie risk, rating of each. A System health rating of 1 means all controls evaluated for that system are fully effectively (very low risk) and a system health rating of 5 means none of the controls are operating effectively (high risk).

Target: Confidential

Health: Confidential

• Security Compliance performs regular testing of controls by system and uses the System Health Rating methodology to determine the overall health of each system in scope of GitLab’s compliance and regulatory programs. A System health rating of 1 means all controls evaluated for that system are operating fully effectively and a system health rating of 5 means none of the controls are operating effectively. 57% (or 13 out of 23) systems assessed met the target. There are 118 observations open for these 23 systems.

URL(s)

• https://app.periscopedata.com/app/gitlab/977899/Security-Impact?widget=14446739&udv=0

Security Impact on Net ARR

The Field Security organization functions as a sales and customer enablement team therefore a clear indicator of success is directly reflected in the engagement of their assessment services by Legal, Sales, TAMs and customers themselves. Assessment services include completing security questionnaires, participating in customer calls, creating and providing security documentation, and facilitating customer audits. The dashboard is calculated as (assessments completed by customer + total contract value by customer = monthly dollar security impact).

Target: Confidential

Health: Confidential

• As anticipated, Field Security observed a significant increase in customer assurance requests in January 2023, the last month of the fiscal year, resulting in the most amount of requests processed since inception of this KPI.

URL(s)

• https://app.periscopedata.com/app/gitlab/977899/Security-Impact?widget=13710204&udv=0

Estimated Cost of Abuse

This metric tracks the estimated cost of abuse in terms of CI Compute Cost & Storage costs from blocked accounts. It also includes aggregated Networking Cost data when it is over the baseline spend for known skus that only trend up during periods of elevated abuse, although this is not tracked at the user level. It does not include reputation damage costs or labor costs of having to manually prevent certain types of abuse.

Target: less than $10K/Month

This KPI cannot be public.

Health: Confidential

• Confidential metric - See notes in Key Review agenda

URL(s)

• https://app.periscopedata.com/app/gitlab/780726/WIP:-Blocked-User-Usage-Data?widget=12404680

Security Team Member Retention

We need to be able to retain talented team members. Retention measures our ability to keep them sticking around at GitLab. Team Member Retention = (1-(Number of Team Members leaving GitLab/Average of the 12 month Total Team Member Headcount)) x 100. GitLab measures team member retention over a rolling 12 month period.

Target: at or above 84%

This KPI cannot be public.

Health: Confidential

• Action = Security leadership have created OKRs focused on addressing team member concerns contributing to attrition rates.

URL(s)

• https://app.periscopedata.com/app/gitlab/862338/Security-Department-Retention

Security Average Age of Open Positions

Measures the average time job openings take from open to close. This metric includes sourcing time of candidates compared to Time to Hire or Time to Offer Accept which only measures the time from when a candidate applies to when they accept.

Target: at or below 50 days

Health: Problem

• High demand for Security professionals has lead to an extremely competitive hiring market. Security leadership are actively involved in the recruiting processes to reduce burden on recruiters and shorten time to hire.

Chart (Sisense↗)

Regular Performance Indicators

Security Budget Plan vs Actuals

We need to spend our investors' money wisely. We also need to run a responsible business to be successful. For Security purposes, please view this chart directly in Sisense. Latest data is in Adaptive, data team importing to Sisense in FY22Q2

Target: See Sisense for target

Health: Okay

• This chart was recently updated and now reflective of current state. Security are within budget expectations.

URL(s)

• https://app.periscopedata.com/app/gitlab/633239/Security-Non-Headcount-BvAs

Security Handbook MR Rate

The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/security` over time.

Target: 1

Health: Attention

• Security was highly focused on operations and hiring in FY23. Action = Security expect to see a marked increase in handbook updates once project work picks back up.

Chart (Sisense↗)

Security Department Discretionary Bonus Rate

The number of discretionary bonuses given divided by the total number of team members, in a given period as defined. This metric definition is taken from the People Success Discretionary Bonuses KPI.

Target: at or above 10%

Health: Attention

• Security had observed a downward trend for bonus nominations recently. The previous action for Managers to actively encourage team members to submit nominations where appropriate was effective at bringing the metric back on target.

Chart (Sisense↗)

Security Incidents by Category

The metric groups security incidents by incident category and provides visibility into possible trends in attack types or targeted systems. Tracking of this metrics allows GitLab to adjust security control strategies, identify opportunities for improvements, and address security controls needing attention. For Security purposes, please view this chart directly in Sisense.

Target: Number of security incidents in any category does not exceed +50% of the individual category's 3-month average.

Health: Okay

• While the total number of security incidents has increased by 16% over the last 3 months, all incident categories are within expected and acceptable thresholds.
• Category labels are applied in accordance to the SIRT handbook page. In the event that "NotApplicable" keeps growing, we'll be adding additional categories.

URL(s)

• https://app.periscopedata.com/app/gitlab/592612/Security-KPIs?widget=9277541&udv=0

Operational Security Risk Management (Tier 2 Risks)

Operational risk management enables organizations to proactively identify and mitigate operational security risks that may impact Organizational Output, Brand Reputation, Business Continuity, Customers & Stakeholders, Legal & Regulatory and/or Financials. This heatmap has been generated from ZenGRC. Numbers within each box indicate the total number of potential risks. Red boxes indicate the risk level is HIGH. Orange boxes indicate the risk level is MODERATE. Green boxes indicate the risk level is LOW. The heatmap shows risks that are currently open and accepted, in remediation or planned for remediation.

Target: TBD, this will be determined upon Sisense integration for detailed dashboarding

Health: Okay

• #<Middleman::Util::EnhancedHash There are currently 6 high risks on the Security Operational Risk Register. Security’s Annual Risk Assessment was completed in Q2. As part of the ARA and standard operational risk activities (such as risk treatment or trends in the IT environment), risk scores displayed in the heatmap are subject to change over time. Risks identified during the ARA have been associated with Risk Treatment plans (8/8 or 100%). Each of these plans have been validated with risk owners. Action="We are targeting a refresh of 100% of Tier 2 Risk Treatment Plans (38 of 38) by the end of February 2023.">

Chart (Sisense↗)

Security Observations (Tier 3 Risks)

This is a subset of an existing KPI. Please see the definition for the parent KPI.

An indicator of information system and process risk, there are multiple inputs that lead to identification of Observations to include Security Compliance continuous control testing, third party (vendor) assessments, external audits and customer assessments.

Target: Confidential

Health: Okay

• 61% (or 621 of 1017) of all observations opened to date have been resolved, 102 of the 621 were high-risk observations. Of observations identified this fiscal year, ~2% (or 1 of 54) have been resolved. 11 high-risk observations have been identified.

Chart (Sisense↗)

Third Party Risk Management

An indicator of third party risk, third party risk assessments proactively identify potential vendor security risks as part of onboarding or contracting, enabling business owners to make risk based decisions throughout the vendor lifecycle.

Target: TBD, this will be determined upon Sisense integration for detailed dashboarding

Health: Okay

• 50 vendor intake requests have hit our queue in FY24Q1 (16% YoY increase). These requests have resulted in 12 new third party full assessments which is a 15% decrease year over year.

Chart (Sisense↗)

Security Automation Iteration Velocity Average

We attempt to complete 7 issues or more every two weeks. The measurement indicates how well the Security Automation team is scoping milestones over the last 10 iterations and provides a view of the average team velocity.

Target: 7

Health: Okay

• This PI is considered healthy. The Security Automation team has seen a slight decrease in average issues per iteration due to planned and unplanned team member time off. The target average will be re-evaluated on a quarterly basis as the team grows and becomes more efficient through process updates, additional automation deployments, and technology integrations.

Chart (Sisense↗)

Security Department Promotion Rate

The total number of promotions over a rolling 12 month period divided by the month end headcount. The target promotion rate is 12% of the population. This metric definition is taken from the People Success Team Member Promotion Rate PI.

Target: 12%

Health: Attention

• Security have recently announced multiple promotions. Action = Security leadership are now executing Individual Development Plans as part of one on ones with a direct focus on career development.

Chart (Sisense↗)

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
  • Incubation Engineering Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
-1	Confidential	Metric & metric health are confidential
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.