This page acts as the raw materials for the Engineering Key meeting, where we review Key Performance Indicators (KPIs) and Objectives & Key Results (OKRs) with stakeholders.
The Google document for note taking is confidential within the company because customers may be referenced. Please see the calendar invite for the URL.
This meeting covers keys for the entire Engineering Division as well as the Development, Quality, Security, and UX departments. the Infrastructure and Support departments within Engineering have their own key meetings.
The following performance indicators are limited to just those defined as keys. They are sorted by ascending health (so problems are at the top). And they are grouped by the org to which they roll up.
MRARR is the measurement of Merge Requests from customers, multiplied with the revenue of the account (ARR) from that customer. This measures how active our biggest customers are contributing to GitLab. We believe the higher this number the better we'll retain these customers and improve product fit for large enterprises. The unit of MRARR is MR Dollars (MR$). MR Dollars is different than the normal Dollars which is used for ARR.
Target: Greater than 20M MR Dollars per month
Chart (Sisense↗)
Status: Attention
This is the total number of MRs authored by non-GitLab team members that go into the product on a monthly basis divided by the total headcount of the GitLab Inc. R&D (the Product Management and Engineering divisions, but not the Support department). The reason we sometimes track the Wider rate is that MRs from our community take effort from one of our coaches to merge, and we want to incentivize that behavior.
Target: Greater than 2 MRs per month
Chart (Sisense↗)
Status: Attention
Set target to 2 to be ambitions and confirm our goals as we work to drive more initiatives to improve
Need to verify data
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/**` or `/source/handbook/support/**` over time.
Target: Greater than 300 per month
Chart (Sisense↗)
Status: Okay
This is the total number of MRs that go into the product on a monthly basis (regardless if the author is a GitLab team member or an open source contributor) divided by the total headcount of the GitLab Inc R&D (the Product Management and Engineering divisions, but not the Support department). See Merge Request Rate for more detail on the MR Rate taxonomy. This metric measures how efficiently we are able to convert our investor's capital and the energy of our community to develop product and deliver value to users. The largest negative driver of this metric have been periods of intense recruiting and explosive headcount growth (100% in 2018, 130% in 2019). And the largest positive drivers are our iteration value (work breakdown) and open source contributions.
Target: Greater than 6 MRs per month
Chart (Sisense↗)
Status: Okay
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor for team members hired within the past 3 months so hiring managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in more efficient location factor areas with another hire. The historical average location factor represents the average location factor for only new hires in the last three months, excluding internal hires and promotions. The calculation for the three-month rolling average location factor is the location factor of all new hires in the last three months divided by the number of new hires in the last three months for a given hire month. The data source is BambooHR data.
Target: Below 0.58
Chart (Sisense↗)
Status: Okay
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/development/**` over time. The calculation for the monthly overall handbook update frequency rate is the number of handbook updates divided by the number of team members in the Development Department for a given month.
Target: At or above 0.5
Chart (Sisense↗)
Status: Attention
To be aligned with review time from Development. Monthly Product MRs review to merge time (RTMT), it tells us on average how long it takes from requesting the first MR review to being merged. This metric includes only MRs authored only by team members in the Development Department. No community contributions are included. The VP of Development is the DRI on what projects are included.
Target: At or below 4.5 days
Chart (Sisense↗)
Status: Attention
We are focused on increasing the maintainer pool to have additional reviewers in place.
December is up to 5.89. This is likely due to Friends and Family holiday and increased PTO in December. We recently increased the target to 5 for December to account for an increase in PTO during the holidays.
Largest Contentful Paint (LCP) is an important, user-centric metric for measuring the largest load speed visible on the web page. To provide a good user experience on GitLab.com, we strive to have the LCP occur within the first few seconds of the page starting to load. This LCP metric is reporting on our Projects Home Page. LCP data comes from the Graphite database. We reference this webpage to compare performance information from various hosted software development services.
Target: Below 2.5s at the 90th percentile
Chart (Sisense↗)
Status: Attention
This is a new KPI.
LCP p90 data is captured every 4 hours and we report on the latest value each day.
Can we improve the sales renewal process to meet a 90% satisfaction rating from internal sales teams?
Target: Above 90%
Chart (Sisense↗)
Status: Okay
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
Status: Okay
We continue to spend less than planned
Based on new financial model provided by finance, we'll make adjustments
Development Department Narrow MR Rate is a performance indicator showing how many changes the Development Department implements directly in the GitLab product. The projects that are part of the product contributes to the overall product development efforts. This is the ratio of product MRs authored by team members in Development Department to the number of team members in the Development Department. It's important because it shows us how productivity within the Development Department has changed over time.
Target: Above 10 MRs per month
Chart (Sisense↗)
Status: Okay
In the past 6 months, the Development Department Narrow MR rate has stayed between 6 and 11.5, with smaller fluctuations month to month. From a department standpoint, our team members are performing just as effectively as they were a year ago.
In December, the narrow MR rate decreased by ~1 MR per Development Team Member. This is most likely due to Friends and Family company holiday and a huge increase in PTO days around the holidays.
The MTTR metric is an indicator of our efficiency in remediating security vulnerabilities, whether they are reported through HackerOne bug bounty program (or other external means, such as security@gitlab.com emails) or internally-reported. The average days to close issues in the GitLab project (project_id = '278964') that are have the label `security` and severity 1, severity 2, or severity 3; this excludes issues with variation of the security label (e.g. `security review`) and severity 4 issues. Issues that are not yet closed are excluded from this analysis. This means historical data can change as older issues are closed and are introduced to the analysis. The average age in days threshold is set at the daily level.
Target: https://about.gitlab.com/handbook/engineering/security/#severity-and-priority-labels-on-security-issues
Status: Problem
GCF security controls are conitnuously tested as parts of the Compliance team's continuous monitoring program, internal audits and external audits. A clear indicator of success is directly reflected in the control effectveness rating. Observations are a result of GCF security failure, indicating that the control is not implemented, designed effectively or is not operating effectively. These observations indicate a gap that requires remediation in order for the security control to be operating and audit ready.
Target: This will be determined in FY22 Q1 as part of GRC application onboarding
Status: Attention
This metric is focused around the financial impact of abusive accounts and their activity.
Target: Cost of abuse is below $10K/Mo
Status: Attention
This metric is focused around the volume and severity of paged incidents to the Security Engineer On-Call.
Target: Number of pages/month does not exceed +50% of monthly average of the last 12 months for 3 consecutive months
Status: Okay
The Field Security organization functions as a sales and customer enablement team therefore a clear indicator of success is directly reflected in the engagement of their services by Legal, Sales, TAMs and customers themselves.
Target: 60%
Chart (Sisense↗)
Status: Okay
Measures the feedback time for merge request pipeline failures to quantify how quickly the feedback is provided to team members that there is a failure which needs action.
Target: Below 30 minutes
Status: Unknown
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
Status: Attention
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching /source/handbook/engineering/quality/**over time.
Target: Above 35
Chart (Sisense↗)
Status: Attention
Continual decrease since the department FY20Q2 KR to raise contributions
We are making progress but still under target, continuing to champion small iterations and being handbook first
Measures the stability of our test tooling to enable engineering efficiency.
Target: Above 99%
Chart (Sisense↗)
Status: Attention
Remains above 90%, at 95% for January
Increasing review app utilization or stability has been de-prioritized in favor of pipeline improvements, engineering metrics, and Community Contribution workstreams
Measures the stability of the GitLab project master pipelines to accelerate cycle time of merge requests, continuous deployments and GitLab EE releases.
Target: Above 95%
Chart (Sisense↗)
Status: Attention
Success rate has been at or above 90% for 10 consecutive months
Increased coverage for master broken resolution by Development department has helped keep above 90%
Merge train was piloted for gitlab-org/gitlab for 8 hours and identified blockers to work through
Measures the average duration of our merge request pipelines to accelerate our development cycle time, and continuous deployments. See epics 1853 and 25 for individual work items.
Target: Below 30 minutes
Chart (Sisense↗)
Status: Attention
Increased from 53.9 to 58.5 due to new bottleneck jobs being added early in January and increase proportion of longer running pipeline types (e2e and frontend).
Refining pipeline performance indicators to identify a suite of pipeline metrics including time to failure.
FY20Q4 KR to further reduce pipeline duration to 45 minutes
Mean time to close of severity 1 bugs. Measured from time severity triage to bug closure.
Target: Below 30 days
Chart (Sisense↗)
Status: Attention
Decreased in December 2020 from 26.8 to 13.4 and 3 month moving average has decreased from 23.2 to 20.8.
Future revision to bug PIs to add average age and bug SLO percent achieved
Mean time to close of severity 2 bugs. Measured from time severity triage to bug closure.
Target: Below 60 days
Chart (Sisense↗)
Status: Attention
Decreased from 94.6 to 54.3 during December 2020 but 3 month moving average increased slightly from 77.6 to 78.2
3 month average has been increasing since 2020-07-01
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/**` or `/source/handbook/support/**` over time.
Target: Above 50
Chart (Sisense↗)
Status: Unknown
The System Usability Scale (SUS) is an industry-standard survey that measures overall system usability based on 10 questions. Moving a SUS score upward even a couple of points on a large system is a significant change. The goal of this KPI is to understand how usability of the GitLab product rates against industry standards and then track trends over time. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change. See our grading scale for details on interpreting scores.
Target: Above 75 (out of 100)
Chart (Sisense↗)
Status: Problem
Perceived usability rates as a C+ and has steadily declined over 7 quarters.
Working with PM to prioritize usability issues. Efforts include Q4 FY21 KRs to fix 50 usability issues, make Settings a great experience, and identify the top 10 usability problems in the product (these research results will feed into a Q1 FY22 OKR).
To make the SUS less of a lagging indicator (due to long version upgrade cycles on self managed), we ran a pilot in Q3 to source participants who are SaaS users (from our data warehouse) instead of just our First Look panel (SaaS + self managed and longer tenured users). The pilot succesfully shows that we can shift to SaaS participants only in Q4 without any impact to historical metrics.
Our goal is to use customer research to validate problems and solutions to ensure we are building the right things in the right way. We use many research methods, including interviews, surveys, usability studies, findability/navigation studies, and analytics. Hypothesis that there is a connection between this KPI and SUS KPI.
Target: At or greater than 2 validation issues per Product Designer per quarter
Chart (Sisense↗)
Status: Attention
We're under target in Q2 - Q4 FY21.
To reduce the friction associated with moderated testing, an unmoderated testing solution was set in place in Nov 2020. This allows for easier research with faster results. As a result, we expect to see an increase in solution validation studies in Q1 FY22.
We will further investigate why there have been fewer solution validation studies during Q2 - Q4 FY21.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
Status: Okay
