| KPI | Health | Status |
|---|---|---|
| Quality Hiring Actual vs Plan | Okay |
|
| Quality Non-Headcount Plan vs Actuals | Unknown |
|
| Quality Average Location Factor | Attention |
|
| Quality Recruiting Average Top-of-Funnel Location Factor | Unknown |
|
| Quality Handbook Update Frequency | Problem |
|
| Review App deployment success rate for GitLab | Problem |
|
| GitLab project master pipeline success rate | Okay |
|
| Average merge request pipeline duration for GitLab | Attention |
|
| Quality Department Member MR Rate | Problem |
|
| Average cost per merge request pipeline for GitLab | Attention |
|
| New issue first triage SLO | Unknown |
|
| P1/P2 open bugs past target SLO | Attention |
|
This is a subset of an existing KPI. Please see the definition for the parent KPI.
Employees are in the division "Engineering" and department is "Quality".
Target: 24 by November 1, 2019
URL(s)
Chart (Sisense↗)
Status: Okay
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
URL(s)
Status: Unknown
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.
Target: 0.58
URL(s)
Chart (Sisense↗)
Status: Attention
We need to be proactive in measuring our location factor, starting with candidates who are at the top of the recruiting funnel.
Target: 0.58
URL(s)
Chart (Sisense↗)
Status: Unknown
This is a subset of an existing KPI. Please see the definition for the parent KPI.
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching /source/handbook/engineering/quality/**over time.
Target: 35
URL(s)
Chart (Sisense↗)
Status: Problem
Measures the stability of our test tooling to enable engineering efficiency.
Target: 99%
URL(s)
Chart (Sisense↗)
Status: Problem
Measures the stability of the GitLab project master pipelines to accelerate cycle time of merge requests, continuous deployments and GitLab EE releases.
Target: 95%
URL(s)
Chart (Sisense↗)
Status: Okay
Measures the average duration of our merge request pipelines to accelerate our development cycle time, and continuous deployments. See epics 1853 and 25 for individual work items.
Target: 30 minutes
URL(s)
Chart (Sisense↗)
Status: Attention
Quality Department MR Rate is a key indicator showing how productive our team members are based on the average MR merged per team member. We currently count all members of the Quality Department (Director, EMs, ICs) in the denominator because this is a team effort. The projects included contributes to the overall product development efforts.
Target: 10 MRs per Month
URL(s)
Chart (Sisense↗)
Status: Problem
Measure the cost per pipeline to measure engineering efficiency. This is calculated by taking the total cost of all merge request pipelines divided by the number of pipelines.
Target: $1.25
URL(s)
Chart (Sisense↗)
Status: Attention
Measure our speed to triage new issues. We currently have ~400 new issues every week in CE/EE. We need to go through all of them and identify valid issues and high severity bugs.
Target: 5 days
URL(s)
Status: Unknown
Measure the number of bugs past the priority SLO timeline.
Target: 0 P1, 10 P2 for customer labelled bugs.
URL(s)
Status: Attention
Discretionary bonuses offer a highly motivating way to reward individual GitLab team members who really shine as they live our values. Our goal is to award discretionary bonuses to 10% of GitLab team members in the Quality department every month.
Target: 10%
URL(s)
Status: Unknown
Measures the average duration of our full QA/end-to-end test suite in the master branch to accelerate cycle time of merge requests, and continuous deployments.
URL(s)
Status: Unknown
Measures the stability and effectiveness of our QA/end-to-end tests running in the master branch.
URL(s)
Status: Unknown
Tells us the hit rate of defects per each stage group on a monthly basis
URL(s)
Status: Problem
Tells us the monthly creation and resolution of high severity S1 and S2 bugs aggregated from each product group.
URL(s)
Status: Unknown
Tells us the monthly average time to resolve high severity defects.
URL(s)
Status: Problem
Measure our ability to differentiate high severity defects from the pool so we can prioritize fixing them above trivial bugs.
URL(s)
Status: Attention
Measure amount of throw away work vs merged.
URL(s)
Status: Attention
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| 0 | Unknown | Unknown |
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | We are correctly capturing and measuring data for GitLab.com and self-managed instances. |
| 2 | Attention | Missing one of: GitLab.com or self-managed data. |
| 1 | Problem | Missing both: GitLab.com and self-managed data. |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionmaturity table, make changes to the Performance Indicators Maturities ERB fileThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicator_target(name) helper function returns the target value from the target property for a specific PI from the Performance Indicators data file.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from the Performance Indicators data file and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this is the Performance Indicators data file which is in YAML file. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department) |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI. |
urls |
Optional | list of urls associated with the (K)PI |
sisense_data |
Optional | contains elements/properties related to Sisense, including chart (numeric Sisense widget ID), dashboard (numeric Sisense dashboard ID), shared_dashboard (Sisense shared dashboard ID), embed (v2) |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
instrumentation |
Optional | has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI. |
shared_dashboard, chart , and the dashboard key-value pairs to the Performance Indicators data file under the sisense_data property: in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :urls: should be an array (indented lines starting with dashes) even if you only have one urlmaturity.level and health.level display a value between 0 and 3 (inclusive). The health value is a manual input while the maturity is an automated value based on other discrete (K)PI data properties.