| KPI | Health | Status |
|---|---|---|
| Security Hiring Actual vs Plan | Okay |
|
| Security Non-Headcount Plan vs Actuals | Unknown |
|
| Security Average Location Factor | Attention |
|
| Security Recruiting Average Top-of-Funnel Location Factor | Unknown |
|
| Security Handbook Update Frequency | Unknown |
|
| Security Department MR Rate | Attention |
|
| MTTM (Mean-Time-To-Mitigation) for S1-S2-S3 security vulnerabilities | Okay |
|
| Average age of currently open bug bounty vulnerabilities by severity | Attention |
|
| Net vulnerability count for each month | Okay |
|
| HackerOne budget vs actual (with forecast of 'unknowns') | Attention |
|
| Security Risk Heatmap | Attention |
|
| SIRT (former Security Operations) Page Volume | Attention |
|
| Security Compliance Observations | Okay |
|
| GCF Security Control Effectiveness | Attention |
|
This is a subset of an existing KPI. Please see the definition for the parent KPI.
Employees are in the division "Engineering" and department is "Security".
URL(s)
Chart (Sisense↗)
Status: Okay
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
URL(s)
Status: Unknown
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas. Data comes from BambooHR and is the average location factor of all team members in the Security department.
URL(s)
Chart (Sisense↗)
Status: Attention
We need to be proactive in measuring our location factor, starting with candidates who are at the top of the recruiting funnel.
Target: 0.58
URL(s)
Chart (Sisense↗)
Status: Unknown
This is a subset of an existing KPI. Please see the definition for the parent KPI.
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/security` over time.
Target: 100
Chart (Sisense↗)
Status: Unknown
Security Department MR Rate is a key indicator showing how productive our team members are based on the average MR merged per team member. We currently count all members of the Security Department (Director, EMs, ICs) in the denominator because this is a team effort. The projects included contributes to the overall product development efforts.
Target: 5 MRs per Month
URL(s)
Chart (Sisense↗)
Status: Attention
The MTTM metric is an indicator of our efficiency in mitigating security vulnerabilities, whether they are reported through HackerOne bug bounty program (or other external means, such as security@gitlab.com emails) or internally-reported. The average days to close issues in the GitLab CE project (project_id = '278964') that are have the label `security` and S1, S2, or S3; this excludes issues with variation of the security label (e.g. `security review`) and S4 issues. Issues that are not yet closed are excluded from this analysis. This means historical data can change as older issues are closed and are introduced to the analysis. The average age in days threshold is set at the daily level.
URL(s)
Status: Okay
The average age of currently open bug bounty vulnerabilities gives a health snapshot of how fast we are fixing the vulnerabilities that currently exist. This is important because it can help indicate what our future MTTM will look like and whether we are meeting our defined SLAs. The query is built by using the `ultimate_parent_id` of `9970` and is only for `open` `state` issues labelled with `security` and `hackerone`. The average age is measured in days, and the targets for each severity are defined in https://about.gitlab.com/handbook/engineering/security/#severity-and-priority-labels-on-security-issues.
Target: https://about.gitlab.com/handbook/engineering/security/#severity-and-priority-labels-on-security-issues
URL(s)
Status: Attention
The net vulnerability count for each month allows us to see whether we are adding or eliminating from the security backlog. This is the open count - close count for each month. It's important to note that these are only for bug bounty reported vulnerabilites. This chart is intended to be used with other data to try and illustrate a more detailed story.
Target: 0 or less
URL(s)
Status: Okay
We currently run a public bug bounty program through HackerOne, and this program has been largely successful - we get a lot of hacker engagement, and since the program went public, we have been able to resolve nearly 100 reported security vulnerabilities. The bounty spend is however, a budgeting forecast concern because of the unpredictability factor from month to month.
Status: Attention
The purpose of the Security Operational Risk Management (“STORM”) program at GitLab is to identify, track, and treat security operational risks. Ultimately, the management of operational risks helps support the broader organization wide objectives by ensuring that security risks that may impact GitLab's ability to achieve its customer commitments and operational objectives are effectively identified and treated in order to sustain and maintain the operational stability of GitLab's offerings.
URL(s)
Status: Attention
This metric is focused around the volume and severity of paged incidents to the Security Incident Response Team.
Status: Attention
Observations are a result of GCF security control testing when that testing indicates that the control is not designed effectively or is not operating effectively. These observations indicate a gap that requires remediation in order for the security control to be operating in a state that can pass an internal or external audit.
URL(s)
Status: Okay
Each GCF security control is assigned a control effectiveness rating after it has been tested. It's possible for a single control to have multiple effectiveness ratings if it has been tested against multiple different systems. A control effectiveness rating of 1 indicates that the control has no open observations and an effectiveness rating of 5 indicates that there are 1 or more high-risk observations open.
URL(s)
Status: Attention
The Trust & Safety team is responsible for tracking, automating, and reporting on blocked abusive accounts on GitLab.com, and tracking reduction in cloud spend where the activities are due to abuse. This metric is the volume of accounts mitigated for abusive activity. We are in the process of developing a KPI around the efficacy of our abuse report handling.
URL(s)
Status: Unknown
The true mark of a successful public bug bounty program is partially tied to hacker outreach and engagement. We’ve already implemented quite a bit of security automation to engage hackers. For example, when findings are submitted by hackers, an automated dynamic message is generated by our automation scripts and they receive a response with expected ETA on a reply (this ETA is dependent on how many findings are in our ‘to be triaged’ bucket at that time, so the ETA changes). This ensures the hacker knows 1) we received the report, 2) a human will be reading the report within a certain ETA, and 3) a human will be replying within an ETA. Our program went public in mid-December 2018, and we are now focusing on outreach and incentive programs to retain the top hackers to our program - about 90%+ of the most impactful reports are currently submitted by our Top 5-10 hackers, so it is imperative we retain those individuals.
Status: Attention
Although the Security Department is always working towards minimizing friction to day-to-day company operations, it is never possible to be 100% frictionless when adding an additional step into an existing process. As cumbersome as this may feel (despite our best efforts), this is usually a necessary step to maturing our security posture and meeting customer requirements (especially large enterprise customers). We plan to measure Security response times to new processes such as access request approval issues and security reviews for 3rd party vendor requests to highlight where bottlenecks are to the current process, and use a data-driven approach to iterate on better processes.
Status: Unknown
Discretionary bonuses offer a highly motivating way to reward individual GitLab team members who really shine as they live our values. Our goal is to award discretionary bonuses to 10% of GitLab team members in the Security department every month.
Target: 10%
Status: Unknown
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| 0 | Unknown | Unknown |
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | We are correctly capturing and measuring data for GitLab.com and self-managed instances. |
| 2 | Attention | Missing one of: GitLab.com or self-managed data. |
| 1 | Problem | Missing both: GitLab.com and self-managed data. |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionmaturity table, make changes to the Performance Indicators Maturities ERB fileThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicator_target(name) helper function returns the target value from the target property for a specific PI from the Performance Indicators data file.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from the Performance Indicators data file and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this is the Performance Indicators data file which is in YAML file. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department) |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI. |
urls |
Optional | list of urls associated with the (K)PI |
sisense_data |
Optional | contains elements/properties related to Sisense, including chart (numeric Sisense widget ID), dashboard (numeric Sisense dashboard ID), shared_dashboard (Sisense shared dashboard ID), embed (v2) |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
instrumentation |
Optional | has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI. |
shared_dashboard, chart , and the dashboard key-value pairs to the Performance Indicators data file under the sisense_data property: in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :urls: should be an array (indented lines starting with dashes) even if you only have one urlmaturity.level and health.level display a value between 0 and 3 (inclusive). The health value is a manual input while the maturity is an automated value based on other discrete (K)PI data properties.