Engineering is on plan. But we are lending some of our recruiters to sales for this quarter. And we just put in place a new "one star minimum" rule that might decrease offer volume.
We have prioritized efforts. We will likely not trend downwards until the backlog of older bugs are closed.
Improve the chart by changing time series to bug age not creation month.
Define threshold and migrate into periscope.
Key Performance Indicators
Quality Hiring Actual vs Plan
Are we able to hire high quality workers to build our product vision in a timely manner? Hiring information comes from BambooHR where employees are in the division `Engineering` and department is `quality`. This KPI is tracked and reported on a monthly basis but year to date (headcount v plan) is also measured.
Engineering is on plan. But we are lending some of our recruiters to sales for this quarter. And we just put in place a new "one star minimum" rule that might decrease offer volume.
Currently finance tells me when there is a problem, I’m not self-service.
Get the budget captured in a system
Chart budget vs. actual over time in periscope
Maturity: Level 2 of 3
Quality Average Location Factor
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.
The handbook is essential to working remote successfully, to keeping up our transparency, and to recruiting successfully. Our processes are constantly evolving and we need a way to make sure the handbook is being updated at a regular cadence. This data is retrieved by querying the API with a python script for merge requests that have files matching /source/handbook/engineering/quality/**over time.
Rate is 80% as of March 2020, 10 day moving average increased from 81% to 85%.
Our master broken triage process is effective.
Our defensive measure with rebasing master with pipeline results seems to be improving the stability.
Maturity: Level 3 of 3
Average merge request pipeline duration for GitLab
Measures the average duration of our merge request pipelines to accelerate our development cycle time, and continuous deployments. See epics 1853 and 25 for individual work items.
Average duration continues to trend slightly lower, we are at 65 mins as of March 2020.
We optimized jest frontend unit test jobs.
We are working on more improvements with caching and reducing unneeded test jobs.
Maturity: Level 3 of 3
New issue first triage SLO
Measure our speed to triage new issues. We currently have ~400 new issues every week in CE/EE. We need to go through all of them and identify valid issues and high severity bugs.
We have prioritized efforts. We will likely not trend downwards until the backlog of older bugs are closed.
Improve the chart by changing time series to bug age not creation month.
Define threshold and migrate into periscope.
Maturity: Level 2 of 3
Regular Performance Indicators
Quality Discretionary Bonus Rate
Discretionary bonuses offer a highly motivating way to reward individual GitLab team members who really shine as they live our values. Our goal is to award discretionary bonuses to 10% of GitLabbers in the Quality department every month.
Target: 10%
URL(s)
Health: Unknown
We currently track bonus percentages in aggregate, but there is no easy way to see the percentage for each individual department.
Maturity: Level 2 of 3
Review App deployment success rate for GitLab FOSS
Measures the stability of our test tooling to enable engineering efficiency. See this epic for specific items.
Rate is above 78% as of December 2019 but we need to make sure it continues to go up.
Maturity: Level 3 of 3
Average duration of end-to-end test suite execution on CE/EE master branch
Measures the average duration of our full QA/end-to-end test suite in the master branch to accelerate cycle time of merge requests, and continuous deployments.
Defect trends on a high level is increasing for mature areas of the product, but also due to double counting from duplicate issues. We need to raise more awareness and encourage teams to look at this metric and evaluate themselves regularly.
Add this metric into every stage groups' dashboard to raise awareness.
Define threshold and migrate into Periscope. https://gitlab.com/gitlab-data/analytics/issues/1792
Define a mechanism to filter duplicate issues.
Maturity: Level 1 of 3
Needs a target
Mean time to resolve S1-S2 functional defects
Tells us the monthly average time to resolve high severity defects.
We have prioritized efforts and trend is stabilizing. We have equiped engineering groups with better weeky triage reports. We will likely not trend downwards until the backlog of older bugs are closed.
Migrate to GitLab Insights dashboard https://gitlab.com/groups/gitlab-org/-/insights.
Add integration into Periscope as a GitLab feature.
Add this metric into every stage groups' dashboard to raise awareness.
Define a mechanism to filter duplicate issues.
Maturity: Level 1 of 3
Needs a target
Ratio of bugs triaged with Severity (and priority)
Measure our ability to differentiate high severity defects from the pool so we can prioritize fixing them above trivial bugs.
The number of untriaged bugs have been stable but we can still do better. We need to establish an actionable threshold for each team.
Convert chart to time series data and define threshold.
Migrate into Native GitLab Insights dashboard https://gitlab.com/groups/gitlab-org/-/insights.
Add integration into Periscope as a GitLab feature.
Providing a weekly bug report with a distribution matrix of priority and severity. - https://gitlab.com/gitlab-org/quality/triage-ops/issues/186
Provide training for product/engineering groups on how to effectively use Priority and Severity. - https://gitlab.com/gitlab-org/quality/team-tasks/issues/148
Enforce triaging escalations (based on stage group labels) for bugs with no severity/priority labels.
Maturity: Level 1 of 3
Needs a target
The ratio of closed (not merged MRs) vs merged MRs over time.
Has a description, target, and Sisense embed (if public) or URL (or not).
Level 2 of 3
Missing one of: description, target, or Sisense embed (if public) or URL (or not).
Level 1 of 3
Missing two of: description, target, or Sisense embed (if public) or URL (or not).
Level 0 of 3
Missing a description, a target, and Sisense embed (if public) or URL (or not).
Health
Level
Meaning
Okay
The KPI is at an acceptable level compared to the threshold
Attention
This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
Problem
We'll prioritize our efforts here
Unknown
Unknown
How to work with pages like this
Data
The heart of pages like this is a data file called /data/performance_indicators.yml which is in YAML format. Almost everything you need to do will involve edits to this file. Here are some tips:
Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :
urls: should be an array (indented lines starting with dashes) even if you only have one url
Both maturity.level and health.level take a number between 0 and 3 (inclusive)
Two flags to note:
parent can be used when a PI or KPI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
public can be set to false where a PI or KPI meets our not public guidelines.
Pages
Pages like /handbook/engineering/performance-indicators/ are rendered by and ERB template.
These ERB templates call the helper function performance_indicators() that is defined in /helpers/custom_helpers.rb. This helper function calls in several partial templates to do it's work.
This function takes a required argument named org in string format that limits the scope of the page to a portion of the data file. Possible valid values for this org argument are listed in the org property of each element in the array in /data/performance_indicators.yml.
