Engineering Function Performance Indicators

1. You are here:
2. Engineering
3. Engineering Function Performance Indicators

On this page

• Executive Summary
• Key Performance Indicators
  • Engineering Non-Headcount Plan vs Actuals
  • Engineering Handbook Update Frequency
  • MRARR
  • R&D Overall MR Rate
  • R&D Wider MR Rate
  • Engineering Division New Hire Average Location Factor
  • Volume of Community MRs
• Regular Performance Indicators
  • Engineering Hiring Actual vs Plan
  • Engineering Average Location Factor
  • Diversity
  • Team Member Retention
  • Engineering Discretionary Bonus Rate
  • Engineering Division Narrow MR Rate
  • Merge Requests Types
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Executive Summary

KPI	Health	Status
Engineering Non-Headcount Plan vs Actuals	Okay	We are below our budget significantly
Engineering Handbook Update Frequency	Okay	Above target and trending up
MRARR	Attention	Amount is far under the perceived amount, efforts underway to align contributors to organizations. We are working on getting our larger customers to contribute regularly on a monthly cadence.
R&D Overall MR Rate	Okay	Above target
R&D Wider MR Rate	Attention	Set target to 2 to be ambitions and confirm our goals as we work to drive more initiatives to improve Need to verify data
Engineering Division New Hire Average Location Factor	Okay	Now on target
Volume of Community MRs	Attention	This metric is new, but moderately below target

Key Performance Indicators

Engineering Non-Headcount Plan vs Actuals

We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.

Target: Unknown until FY21 planning process

URL(s)

• https://app.periscopedata.com/app/gitlab/633290/Engineering-Non-Headcount-BvAs

Health: Okay

• We are below our budget significantly

Engineering Handbook Update Frequency

The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/**` or `/source/handbook/support/**` over time.

Target: Unknown until we reach maturity level 2

Chart (Sisense↗)

Health: Okay

• Above target and trending up

MRARR

MRARR is the measurement of Merge Requests from customers, multiplied with the revenue of the account (ARR) from that customer. This measures the cumulative value of paid customers' contribution to GitLab. The unit of MRARR is MR Dollars (MR$). MR Dollars is different than the normal Dollars which is used for ARR.

Target: Greater than 100M MR Dollars per month

URL(s)

• https://app.periscopedata.com/app/gitlab/765640

Chart (Sisense↗)

Health: Attention

• Amount is far under the perceived amount, efforts underway to align contributors to organizations.
• We are working on getting our larger customers to contribute regularly on a monthly cadence.

R&D Overall MR Rate

This is the total number of MRs that go into the product on a monthly basis (regardless if the author is a GitLab team member or an open source contributor) divided by the total headcount of the GitLab Inc R&D (the Product Management and Engineering divisions, but not the Support department). See Merge Request Rate for more detail on the MR Rate taxonomy. This metric measures how efficiently we are able to convert our investor's capital and the energy of our community to develop product and deliver value to users. The largest negative driver of this metric have been periods of intense recruiting and explosive headcount growth (100% in 2018, 130% in 2019). And the largest positive drivers are our iteration value (work breakdown) and open source contributions.

Target: Greater than 6 MRs per month

Chart (Sisense↗)

Health: Okay

• Above target

R&D Wider MR Rate

This is the total number of MRs authored by non-GitLab team members that go into the product on a monthly basis divided by the total headcount of the GitLab Inc. R&D (the Product Management and Engineering divisions, but not the Support department). The reason we sometimes track the Wider rate is that MRs from our community take effort from one of our coaches to merge, and we want to incentivize that behavior.

Target: Greater than 2 MRs per month

Chart (Sisense↗)

Health: Attention

• Set target to 2 to be ambitions and confirm our goals as we work to drive more initiatives to improve
• Need to verify data

Engineering Division New Hire Average Location Factor

We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor for team members hired within the past 3 months so hiring managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in more efficient location factor areas with another hire. The historical average location factor represents the average location factor for only new hires in the last three months, excluding internal hires and promotions. The calculation for the three-month rolling average location factor is the location factor of all new hires in the last three months divided by the number of new hires in the last three months for a given hire month. The data source is BambooHR data.

Target: 0.58

Chart (Sisense↗)

Health: Okay

• Now on target

Volume of Community MRs

Frequency of Community contribution merged MRs from projects that are part of the product over the last 24 months.

Target: Above 300 per month

Chart (Sisense↗)

Health: Attention

• This metric is new, but moderately below target

Regular Performance Indicators

Engineering Hiring Actual vs Plan

This is a subset of an existing KPI. Please see the definition for the parent KPI.

Employees are in the division `Engineering`.

Target: 525 engineering members by Jan 31, 2019

Chart (Sisense↗)

Health: Okay

• Engineering is on plan

Engineering Average Location Factor

We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.

Target: .58 average location factor

Chart (Sisense↗)

Health: Attention

• We aren't recruiting much, so we can't influence this positively
• It is above target, and there is a slow, steady updward trend

Diversity

Diversity, Inclusion & Belonging is one of our core values, and a general challenge for the tech industry. GitLab is in a privileged position to positively impact diversity in tech because our remote lifestyle should be more friendly to people who may have left the tech industry, or studied a technical field but never entered industry. This means we can add to the diversity of our industry, and not just play a zero-sum recruiting game with our competitors.

Target: It's against company policy to set diversity quotas (but we may add benchmarks to compare ourselves against).

URL(s)

• https://docs.google.com/presentation/d/1Rj6brdP2hKU1zgj0ll1poywBw0aR9jIAdT4GsLZfIIU/edit#slide=id.g4bb9f44e0b_6_160
• https://docs.google.com/spreadsheets/d/1Jef9oopd8AeZRYM2P9B8ye2TxfdzxI0FY6ixhlyzW_c/edit#gid=1035217495

Health: Attention

• Engineering is now at the tech benchmark for gender diversity (~16%), but our potential is greater and we can do better. 20% should be our floor in technical roles. Other types of diversity are unknown.
• Get data wired up and visualized in periscope.
• Clear data sharing with legal.
• Get better data about racial and country diversity.

Team Member Retention

People are a priority and attrition comes at a great human cost to the individual and team. Additionally, recruiting (backfilling attrition) is a ludicrously expensive process, so we prefer to keep the people we have :)

Target: 90%

URL(s)

• https://docs.google.com/spreadsheets/d/1Jef9oopd8AeZRYM2P9B8ye2TxfdzxI0FY6ixhlyzW_c/edit#gid=1405344200

Health: Okay

• I seem to recall our attrition is now below 10% which is great compared to the tech benchmark of 22% and the remote benchmark for 16%, but the fact that I can’t just look at a simple graph makes me nervous…
• Get into a system (BambooHR?) and visualized in periscope
• Break out voluntary and involuntary

Engineering Discretionary Bonus Rate

Discretionary bonuses offer a highly motivating way to reward individual GitLab team members who really shine as they live our values. Our goal is to award discretionary bonuses to 10% of GitLab team members in the Engineering function every month.

Target: 10%

URL(s)

Health: Unknown

• We currently track bonuses at the department level, but not at the Engineering-function level.

Engineering Division Narrow MR Rate

Engineering MR Rate is a key indicator showing how productive our team members are based on the average MR merged per team member. We currently count all members of the Engineering Division (Director, EMs, ICs) in the denominator because this is a team effort and we want to be cognizant of managerial overhead.

Target: Greater than 6 MRs per Month

Chart (Sisense↗)

Health: Problem

• We have found some bugs in the data, so we need to confirm it's accurate before taking any action
• We need to pick a reasonable target

Merge Requests Types

Distribution of monthly Merge Requests that are categorized into types of feature work. Includes work on; new features, enhancements, maintenance and bug fixes projects that are part of the product.

Chart (Sisense↗)

Health: Unknown

• Metric has no target and is being monitored

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Threat Management Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• Data Team KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
lessons.learned	Optional	indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a Red stepped-line for timeseries target.
  • Directional targets will be used:
    • Above ...
    • Below ...
    • At ...
    • At or above ...
    • At or below ...
  • Optional: Use a single Green bar for the current month (in progress). Preferred for metrics which accumulate through a month but optional due to the performance impact to render the chart.
  • Optional: Use a Black line for rolling average.
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
• Multi-series performance indicators should consider the following guidelines:
  • If series are mutually exclusive, use stacked bars for each series with a monthly time series
  • If series are not mutually exclusive, use grouped bars for each series with a monthly time series
  • Do not graph any targets in the chart.
  • Current month styling guidelines will not apply
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :