This page acts as the raw materials for the Engineering Key meeting, where we review Key Performance Indicators (KPIs) and Objectives & Key Results (OKRs) with stakeholders.
The Google document for note taking is confidential within the company because customers may be referenced. Please see the calendar invite for the URL.
This meeting covers keys for the entire Engineering Division as well as the Development, Quality, Security, and UX departments. the Infrastructure and Support departments within Engineering have their own key meetings.
The following performance indicators are limited to just those defined as keys. They are sorted by ascending health (so problems are at the top). And they are grouped by the org to which they roll up.
This is the total number of MRs that go into the product on a monthly basis (regardless if the author is a GitLab team member or an open source contributor) divided by the total headcount of the GitLab Inc R&D (the Product Management and Engineering divisions, but not the Support department). See Merge Request Rate for more detail on the MR Rate taxonomy. This metric measures how efficiently we are able to convert our investor's capital and the energy of our community to develop product and deliver value to users. The largest negative driver of this metric have been periods of intense recruiting and explosive headcount growth (100% in 2018, 130% in 2019). And the largest positive drivers are our iteration value (work breakdown) and open source contributions.
Target: Greater than 6 MRs per month
Chart (Sisense↗)
Status: Attention
This is the total number of MRs authored by non-GitLab team members that go into the product on a monthly basis divided by the total headcount of the GitLab Inc. R&D (the Product Management and Engineering divisions, but not the Support department). The reason we sometimes track the Wider rate is that MRs from our community take effort from one of our coaches to merge, and we want to incentivize that behavior.
Target: TBD
Chart (Sisense↗)
Status: Attention
Need to set target
Need to verify data
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor for team members hired within the past 3 months so hiring managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in more efficient location factor areas with another hire. The historical average location factor represents the average location factor for only new hires in the last three months, excluding internal hires and promotions. The calculation for the three-month rolling average location factor is the location factor of all new hires in the last three months divided by the number of new hires in the last three months for a given hire month. The data source is BambooHR data.
Target: 0.58
Chart (Sisense↗)
Status: Attention
We're about 0.4 over our target
We need to zoom into some Engineering departments that are over and tail them in
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
Status: Okay
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/**` or `/source/handbook/support/**` over time.
Target: Unknown until we reach maturity level 2
Chart (Sisense↗)
Status: Okay
The MTTM metric is an indicator of our efficiency in mitigating security vulnerabilities, whether they are reported through HackerOne bug bounty program (or other external means, such as security@gitlab.com emails) or internally-reported. The average days to close issues in the GitLab CE project (project_id = '278964') that are have the label `security` and severity::1, severity::2, or severity::3; this excludes issues with variation of the security label (e.g. `security review`) and severity::4 issues. Issues that are not yet closed are excluded from this analysis. This means historical data can change as older issues are closed and are introduced to the analysis. The average age in days threshold is set at the daily level.
Target: TBD
Status: Attention
This metric is focused around the volume and severity of paged incidents to the Security Incident Response Team.
Target: TBD
Status: Attention
Volume of pages for critical security incidents in Q2 2020 (to date) has increased in accordance to projections based on data from the previous 3 quarters
Due to unforeseen circumstances affecting our hiring capability, the team is operating with a low, sub-optimal number of people on the on-call roster, which is impacting project-based work
GCF security controls are conitnuously tested as parts of the Compliance team's continuous monitoring program, internal audits and external audits. A clear indicator of success is directly reflected in the control effectveness rating. Observations are a result of GCF security failure, indicating that the control is not implemented, designed effectively or is not operating effectively. These observations indicate a gap that requires remediation in order for the security control to be operating and audit ready.
Target: TBD
Status: Attention
This metric is focused around the Financial impact of abusive accounts.
Target: TBD
Status: Attention
The Field Security organization functions as a sales and customer enablement team therefore a clear indicator of success is directly reflected in the engagement of their services by Legal, Sales, TAMs and customers themselves.
Target: TBD
Status: Okay
Field Security completed Sales security enablement training for new Sales staff on July 17th specifically focused on when and how to engage.
There is a significant amount of reliance being placed the SOC 2 Type 1 report produced by the Security Compliance orgranization by GitLab customers. This is a strong enablement tool.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
Status: Unknown
Currently finance tells me when there is a problem, I’m not self-service.
Get the budget captured in a system
Chart budget vs. actual over time in periscope
Frequency of Community contribution merged MRs for all `is_part_of_product` projects over the last 12 months.
Target: TBD
Chart (Sisense↗)
Status: Problem
Measure the number of bugs past the priority SLO timeline.
Target: TBD
Chart (Sisense↗)
Status: Problem
Measure the number of bugs past the priority SLO timeline.
Target: Target to be determined, we recently adjust SLO to base on severity. Working on providing bug SLO to all product groups.
Chart (Sisense↗)
Status: Problem
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching /source/handbook/engineering/quality/**over time.
Target: Above 35
Chart (Sisense↗)
Status: Attention
Sustained improvement from 14 to 22 in Q2
We are making progress but still under target, continuing to champion small iterations and being handbook first
Measures the stability of our test tooling to enable engineering efficiency.
Target: Above 99%
Chart (Sisense↗)
Status: Attention
Measures the stability of the GitLab project master pipelines to accelerate cycle time of merge requests, continuous deployments and GitLab EE releases.
Target: Above 95%
Chart (Sisense↗)
Status: Attention
Master broken triage process is effective
Decrease in 2020-07 from 97% to 93% due to increase in time to resolve.
Inadequate master-broken coverage during UTC 16:00 to 01:00 has been leading to slower resolutions. Future team additions should have a preference towards those working hours. 56% of all 2020-07 master-broken issues occurred during this timeframe and average MTTR is 20% higher than other failures. Time to close has improved in since the end of July 2020.
Measures the average duration of our merge request pipelines to accelerate our development cycle time, and continuous deployments. See epics 1853 and 25 for individual work items.
Target: Below 30 minutes
Chart (Sisense↗)
Status: Attention
Continued increase from critical path jobs duration increasing
Review app deployment adds 8-10 minutes per frontend pipeline and make up 29% of pipelines
Measure the cost per pipeline to measure engineering efficiency. This is calculated by taking the total cost of all merge request pipelines divided by the number of pipelines.
Target: Below "$1.25"
Chart (Sisense↗)
Status: Attention
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching `/source/handbook/engineering/**` or `/source/handbook/support/**` over time.
Target: Above 50
Chart (Sisense↗)
Status: Unknown
The System Usability Scale (SUS) is an industry-standard survey that measures overall system usability based on 10 questions. Moving a SUS score upward even a couple of points on a large system is a significant change. The goal of this KPI is to understand how usability of the GitLab product rates against industry standards and then track trends over time. Even though UX will be responsible for this metric, they will need other departments such as PM and Development to positively affect change. See Table 1 for grading details.
Target: At or greater than B+
Chart (Sisense↗)
Status: Problem
Perceived usability rates as a C+ and has steadily declined over 6 quarters.
Work with PM to prioritize usability issues. First big focus areas are Navigation and Settings.
Our goal is to use customer research to validate problems and solutions to ensure we are building the right things in the right way. We use many research methods, including interviews, surveys, usability studies, findability/navigation studies, and analytics. Hypothesis that there is a connection between this KPI and SUS KPI.
Target: At or greater than 2 validation issues per Product Designer per quarter
Chart (Sisense↗)
Status: Attention
In Q2 FY21, we had 30 product Designers on staff and 51 total validation issues. This is slightly under our target of 60. For Q3, it's unclear whether we're back on track.
There is a hypothesis that less research is happening because our current reliance on moderated testing is labor intensive. It takes too much effort to complete and too much time to see results. We're exploring introducing an unmoderated research tool for simpler studies that will decrease effort, increase speed, and improve participant diversity.
We need to spend our investors' money wisely. We also need to run a responsible business to be successful, and to one day go on the public market.
Target: Unknown until FY21 planning process
Status: Okay
Actionable insights originate from user research. They always have the 'Actionable Insight' label applied to the resulting issue and a clear follow up that needs to take place as a result of the research observation or data. An actionable insight both defines the insight and clearly calls out the next step as a recommendation. The goal of this KPI is to ensure we're documenting research insights that are actionable and tracking their closure rate.
Target: TBD
Chart (Sisense↗)
Status: Okay
