| KPI | Health | Status |
|---|---|---|
| MRARR | Attention |
|
| MTTC S1 bugs | Okay |
|
| MTTC S2 bugs | Attention |
|
| GitLab project average successful merge request pipeline duration | Attention |
|
| GitLab project master pipeline success rate | Attention |
|
| Review App deployment success rate for GitLab | Attention |
|
| GitLab project merge request pipeline average time to failure | Attention |
|
| Quality Non-Headcount Plan vs Actuals | Okay |
|
| Software Engineer in Test Gearing Ratio | Problem |
|
MRARR is the measurement of Merge Requests from customers, multiplied with the revenue of the account (ARR) from that customer. This measures how active our biggest customers are contributing to GitLab. We believe the higher this number the better we'll retain these customers and improve product fit for large enterprises. The unit of MRARR is MR Dollars (MR$). MR Dollars is different than the normal Dollars which is used for ARR.
Target: Greater than 20M MR Dollars per month
URL(s)
Chart (Sisense↗)
Health: Attention
Mean time to close of severity 1 bugs. Measured from time severity label is applied to bug closure.
Target: Below 30 days
URL(s)
Chart (Sisense↗)
Health: Okay
Mean time to close of severity 2 bugs. Measured from time severity label is applied to bug closure.
Target: Below 60 days
URL(s)
Chart (Sisense↗)
Health: Attention
Measures the average successful duration the GitLab project merge request pipelines. Key building block to improve our cycle time, and effiency.
Target: Below 45 minutes
URL(s)
Chart (Sisense↗)
Health: Attention
Measures the stability of the GitLab project master branch success rate. A key indicator to the stability of our releases.
Target: Above 95%
URL(s)
Chart (Sisense↗)
Health: Attention
Measures the stability of our test tooling to enable engineering efficiency.
Target: Above 99%
URL(s)
Chart (Sisense↗)
Health: Attention
Measures the feedback loop efficiency to team members when there is a failure that needs action.
Target: Below 30 minutes
URL(s)
Chart (Sisense↗)
Health: Attention
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
URL(s)
Health: Okay
Amount of Software Engineers in Test against the targeted gearing ratio
Target: At 46 Software Engineers in Test
Chart (Sisense↗)
Health: Problem
Measures the percentage of closed bugs under SLO time.
Target: TBD
URL(s)
Chart (Sisense↗)
Health: Attention
Identifies the cumulative number of open S1 and S2 bugs per month.
Chart (Sisense↗)
Health: Problem
Average age calculates the average time a bug has been open as of the reported day. Number of bugs is shown to provide context to Average Age. This performance indicator is scoped to S1 open bug issues for all is_part_of_product projects on a daily aggregation for the past 1 year.
Chart (Sisense↗)
Health: Unknown
Average age calculates the average time a bug has been open as of the reported day. Number of bugs is shown to provide context to Average Age. This performance indicator is scoped to S2 open bug issues for all is_part_of_product projects on a daily aggregation for the past 1 year.
Chart (Sisense↗)
Health: Unknown
Tells us the monthly create and resolve of high severity bugs aggregated from each product group.
URL(s)
Health: Unknown
Percent known to unknown severity on open bugs.
Chart (Sisense↗)
Health: Unknown
Average days from creation to merge for Community contribution MRs by month
Target: Below 10 days
Chart (Sisense↗)
Health: Attention
Distribution of unique authors for all merged Community contribution MRs by month.
Target: Above 150 contributors per month
Chart (Sisense↗)
Health: Problem
The number of MR Coaches defined by team.yml role
Target: Above 50 coaches per month
URL(s)
Chart (Sisense↗)
Health: Attention
Percentage of merged `~"Community contribution"` MRs that are labelled `~feature`
Target: Above 30%
Chart (Sisense↗)
Health: Okay
This is a subset of an existing KPI. Please see the definition for the parent KPI.
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching /source/handbook/engineering/quality/**over time.
Target: Above 1 MR per person per month
Chart (Sisense↗)
Health: Attention
Quality Department MR Rate is a key indicator showing how productive our team members are based on the average MR merged per team member. We currently count all members of the Quality Department (Director, EMs, ICs) in the denominator because this is a team effort. The projects that are part of the product contributes to the overall product development efforts.
Target: Above 10 MRs per Month
URL(s)
Chart (Sisense↗)
Health: Attention
Measure the cost per pipeline to measure engineering efficiency. This is calculated by taking the total cost of all merge request pipelines divided by the number of pipelines.
Target: Below "$1.75"
URL(s)
Chart (Sisense↗)
Health: Attention
Measure the total pipeline cost per merge request to measure engineering efficiency. This is calculated by taking the total cost of all pipelines divided by the number of merge requests.
Target: TBD
Chart (Sisense↗)
Health: Attention
Discretionary bonuses offer a highly motivating way to reward individual GitLab team members who really shine as they live our values. Our goal is to award discretionary bonuses to 10% of GitLab team members in the Quality department every month.
Target: At 10%
Chart (Sisense↗)
Health: Okay
This is a subset of an existing KPI. Please see the definition for the parent KPI.
Employees are in the division "Engineering" and department is "Quality".
Target: 24 by November 1, 2019
URL(s)
Chart (Sisense↗)
Health: Okay
This is a subset of an existing KPI. Please see the definition for the parent KPI.
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor for team members hired within the past 3 months so hiring managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in more efficient location factor areas with another hire. The historical average location factor represents the average location factor for only new hires in the last three months, excluding internal hires and promotions. The calculation for the three-month rolling average location factor is the location factor of all new hires in the last three months divided by the number of new hires in the last three months for a given hire month. The data source is BambooHR data.
Target: Below 0.58
Chart (Sisense↗)
Health: Okay
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.
Target: Below 0.58
URL(s)
Chart (Sisense↗)
Health: Attention
Amount of Engineering Productivity Engineers against the targeted gearing ratio
Target: At 16 Engineering Productivity Engineers
Chart (Sisense↗)
Health: Problem
Amount of Quality Engineering Managers against the targeted gearing ratio
Target: At 7 Quality Engineering Managers
Chart (Sisense↗)
Health: Attention
Measures the average duration of our full QA/end-to-end test suite in the master branch to accelerate cycle time of merge requests, and continuous deployments.
Target: TBD
URL(s)
Health: Unknown
Measures the stability and effectiveness of our QA/end-to-end tests running in the master branch.
Target: TBD
URL(s)
Health: Unknown
Measure our speed to triage new issues. We currently have ~400 new issues every week in CE/EE. We need to go through all of them and identify valid issues and high severity bugs.
Target: Below 5 days
URL(s)
Health: Unknown
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name} |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section) |
section |
Optional | the product section (Ex: dev) as defined in sections.yml |
stage |
Optional | the product stage (Ex: release) as defined in stages.yml |
group |
Optional | the product group (Ex: progressive_delivery) as defined in stages.yml |
category |
Optional | the product group (Ex: feature_flags) as defined in categories.yml |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI. |
health.level |
Optional | indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI. |
health.reasons |
Optional | indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
urls |
Optional | list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url |
funnel |
Optional | indicates there is a handbook link for a description of the funnel for this PI. Should be a URL |
sisense_data |
Optional | allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes. |
sisense_data.chart |
Optional | indicates the numeric Sisense chart/widget ID. For example: 9090628 |
sisense_data.dashboard |
Optional | indicates the numeric Sisense dashboard ID. For example: 634200 |
sisense_data.shared_dashboard |
Optional | indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b |
sisense_data.embed |
Optional | indicates the Sisense embed version. For example: v2 |
sisense_data_secondary |
Optional | allows a second Sisense dashboard to be embeded. Same as sisense data |
sisense_data_secondary.chart |
Optional | Same as sisense_data.chart |
sisense_data_secondary.dashboard |
Optional | Same as sisense_data.dashboard |
sisense_data_secondary.shared_dashboard |
Optional | Same as sisense_data.shared_dashboard |
sisense_data_secondary.embed |
Optional | Same as sisense_data.embed |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
pi_type |
Optional | indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI) |
product_analytics_type |
Optional | indicates if the metric is available on SaaS, SM (self-managed), or Both. |
is_primary |
Optional | boolean flag that indicates if this is the Primary PI for the Product Group. |
implementation |
Optional | indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI. |
implementation.status |
Optional | indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI. |
implementation.reasons |
Optional | indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
lessons |
Optional | indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI. |
lessons.learned |
Optional | learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned |
monthly_focus |
Optional | indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI. |
monthly_focus.goals |
Optional | indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal |
metric_name |
Optional | indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name. |
Above ...Below ...At ...At or above ...At or below ...shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property: in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :