Engineering is on plan. But we are lending some of our recruiters to sales for this quarter. And we just put in place a new "one star minimum" rule that might decrease offer volume.
We have prioritized efforts. We will likely not trend downwards until the backlog of older bugs are closed.
Improve the chart by changing time series to bug age not creation month.
Define threshold and migrate into periscope.
Key Performance Indicators
Quality Hiring Actual vs Plan
Are we able to hire high quality workers to build our product vision in a timely manner? Hiring information comes from BambooHR where employees are in the division `Engineering` and department is `quality`. This KPI is tracked and reported on a monthly basis but year to date (headcount v plan) is also measured.
Engineering is on plan. But we are lending some of our recruiters to sales for this quarter. And we just put in place a new "one star minimum" rule that might decrease offer volume.
Health: Monitor health closely
Maturity: Level 3 of 3
Quality Average Location Factor
We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.
Rate is above 63% as of December 2019 but we need to make sure it continues to go up.
Alerting of broken master is noisy and leading to bystander effect for longer lasting ~"master:broken" issues
Maturity: Level 3 of 3
Average merge request pipeline duration for GitLab
Measures the average duration of our merge request pipelines to accelerate our development cycle time, and continuous deployments. See epics 1853 and 25 for individual work items.
Duration is below 75 minutes as of December 2019 and is still going down.
Maturity: Level 3 of 3
New issue first triage SLO
Measure our speed to triage new issues. We currently have ~400 new issues every week in CE/EE. We need to go through all of them and identify valid issues and high severity bugs.
Rate is above 78% as of December 2019 but we need to make sure it continues to go up.
Maturity: Level 3 of 3
Average duration of end-to-end test suite execution on CE/EE master branch
Measures the average duration of our full QA/end-to-end test suite in the master branch to accelerate cycle time of merge requests, and continuous deployments.
Defect trends on a high level is increasing for mature areas of the product, but also due to double counting from duplicate issues. We need to raise more awareness and encourage teams to look at this metric and evaluate themselves regularly.
Add this metric into every stage groups' dashboard to raise awareness.
Define threshold and migrate into Periscope. https://gitlab.com/gitlab-data/analytics/issues/1792
Define a mechanism to filter duplicate issues.
Maturity: Level 1 of 3
Needs a target
Mean time to resolve S1-S2 functional defects
Tells us the monthly average time to resolve high severity defects.
We have prioritized efforts and trend is stabilizing. We have equiped engineering groups with better weeky triage reports. We will likely not trend downwards until the backlog of older bugs are closed.
Migrate to GitLab Insights dashboard https://gitlab.com/groups/gitlab-org/-/insights.
Add integration into Periscope as a GitLab feature.
Add this metric into every stage groups' dashboard to raise awareness.
Define a mechanism to filter duplicate issues.
Maturity: Level 1 of 3
Needs a target
Ratio of bugs triaged with Severity (and priority)
Measure our ability to differentiate high severity defects from the pool so we can prioritize fixing them above trivial bugs.
The number of untriaged bugs have been stable but we can still do better. We need to establish an actionable threshold for each team.
Convert chart to time series data and define threshold.
Migrate into Native GitLab Insights dashboard https://gitlab.com/groups/gitlab-org/-/insights.
Add integration into Periscope as a GitLab feature.
Providing a weekly bug report with a distribution matrix of priority and severity. - https://gitlab.com/gitlab-org/quality/triage-ops/issues/186
Provide training for product/engineering groups on how to effectively use Priority and Severity. - https://gitlab.com/gitlab-org/quality/team-tasks/issues/148
Enforce triaging escalations (based on stage group labels) for bugs with no severity/priority labels.
Maturity: Level 1 of 3
Needs a target
The ratio of closed (not merged MRs) vs merged MRs over time.
Missing one of: description, target, or periscope data.
Level 1 of 3
Missing two of: description, target, or periscope data.
Level 0 of 3
Missing a description, a target, and periscope data.
Health
Level
Meaning
Okay
The KPI is at an acceptable level compared to the threshold
Attention
This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
Problem
We'll prioritize our efforts here
Unknown
Unknown
How to work with pages like this
Data
The heart of pages like this is a data file called /data/performance_indicators.yml which is in YAML format. Almost everything you need to do will involve edits to this file. Here are some tips:
Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :
urls: should be an array (indented lines starting with dashes) even if you only have one url
Both maturity.level and health.level take a number between 0 and 3 (inclusive)
Pages
Pages like /handbook/engineering/performance-indicators/ are rendered by and ERB template.
These ERB templates call the helper function performance_indicators() that is defined in /helpers/custom_helpers.rb. This helper function calls in several partial templates to do it's work.
This function takes a required argument named org in string format that limits the scope of the page to a portion of the data file. Possible valid values for this org argument are listed in the org property of each element in the array in /data/performance_indicators.yml.
