Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Quality Department Performance Indicators

Executive Summary

KPI Health Reason(s)
Quality Hiring Actual vs Plan Okay
  • Engineering is on plan. But we are lending some of our recruiters to sales for this quarter. And we just put in place a new "one star minimum" rule that might decrease offer volume.
  • Health: Monitor health closely
  • Quality Average Location Factor Attention
  • We are at our target of 0.58 exactly overall, but trending upward.
  • We need to get the target location factors in the charts.
  • It will probably be cleaner if we get this in periscope.
  • We need to set the target location factors on our vacancies and make sure recruiting is targeting the right areas of the globe on a per role basis.
  • Review App deployment success rate for GitLab Problem
  • Rate is fluctuating between 25% and 55%.
  • Improve failure troubleshooting.
  • Improve cluster alerting.
  • Scheduled pipeline success rate for master branch in GitLab project Attention
  • Rate is fluctuating between 60% and 75%.
  • Average merge request pipeline duration for GitLab Attention
  • Duration is fluctuating between 70 and 80 minutes.
  • New issue first triage SLO Unknown
  • We haven’t started measuring it yet. We have made progress on fanning out first triage to Engineers in the Quality Department.
  • Define an automated mechanism to collect data in Periscope.
  • Define threshold.
  • Fan out triaging to all of Engineering and not just the Quality Department.
  • P1/P2 open bugs past target SLO Attention
  • We have very little P1 but a large number of P2
  • We have prioritized efforts. We will likely not trend downwards until the backlog of older bugs are closed.
  • Improve the chart by changing time series to bug age not creation month.
  • Define threshold and migrate into periscope.
  • Key Performance Indicators

    Quality Hiring Actual vs Plan

    Are we able to hire high quality workers to build our product vision in a timely manner? Hiring information comes from BambooHR where employees are in the division `Engineering`.

    Target: 24 by November 1, 2019

    URL(s)

    Health: Okay

    Maturity: Level 2 of 3

    Quality Average Location Factor

    We remain efficient financially if we are hiring globally, working asynchronously, and hiring great people in low-cost regions where we pay market rates. We track an average location factor by function and department so managers can make tradeoffs and hire in an expensive region when they really need specific talent unavailable elsewhere, and offset it with great people who happen to be in low cost areas.

    Target: 0.58

    URL(s)

    Health: Attention

    Maturity: Level 2 of 3

    Review App deployment success rate for GitLab

    Measures the stability of our test tooling to enable engineering efficiency. See this epic for specific tasks.

    Target: 99%

    Chart (Periscope↗)

    Health: Problem

    Maturity: Level 3 of 3

    Scheduled pipeline success rate for master branch in GitLab project

    Measures the stability of our `master` scheduled pipelines to accelerate cycle time of merge requests, continuous deployments and GitLab EE releases.

    Target: 95%

    Chart (Periscope↗)

    Health: Attention

    Maturity: Level 3 of 3

    Average merge request pipeline duration for GitLab

    Measures the average duration of our merge request pipelines to accelerate our development cycle time, and continuous deployments. See epics 1853 and 25 for individual work items.

    Target: 30 minutes

    Chart (Periscope↗)

    Health: Attention

    Maturity: Level 3 of 3

    New issue first triage SLO

    Measure our speed to triage new issues. We currently have ~400 new issues every week in CE/EE. We need to go through all of them and identify valid issues and high severity bugs.

    Target: 5 days

    URL(s)

    Health: Unknown

    Maturity: Level 2 of 3

    P1/P2 open bugs past target SLO

    Measure the number of bugs past the priority SLO timeline.

    Target: 0 P1, 10 P2 for customer labelled bugs.

    URL(s)

    Health: Attention

    Maturity: Level 2 of 3

    Regular Performance Indicators

    Review App deployment success rate for GitLab FOSS

    Measures the stability of our test tooling to enable engineering efficiency. See this epic for specific items.

    Target: 99%

    Chart (Periscope↗)

    Health: Problem

    Maturity: Level 3 of 3

    Scheduled pipeline success rate for master branch in GitLab FOSS project

    Measures the stability of our master scheduled pipelines to accelerate GitLab CE releases.

    Target: 95%

    Chart (Periscope↗)

    Health: Attention

    Maturity: Level 3 of 3

    Average duration of end-to-end test suite execution on CE/EE master branch

    Measures the average duration of our full QA/end-to-end test suite in the master branch to accelerate cycle time of merge requests, and continuous deployments.

    URL(s)

    Health: Unknown

    Maturity: Level 1 of 3

    Ratio of quarantine vs total end-to-end tests in CE/EE master branch

    Measures the stability and effectiveness of our QA/end-to-end tests running in the master branch.

    URL(s)

    Health: Unknown

    Maturity: Level 1 of 3

    Monthly new bugs per stage group

    Tells us the hit rate of defects per each stage group on a monthly basis

    URL(s)

    Health: Problem

    Maturity: Level 1 of 3

    Mean time to resolve S1-S2 functional defects

    Tells us the monthly average time to resolve high severity defects.

    URL(s)

    Health: Problem

    Maturity: Level 1 of 3

    Ratio of bugs triaged with Severity (and priority)

    Measure our ability to differentiate high severity defects from the pool so we can prioritize fixing them above trivial bugs.

    URL(s)

    Health: Attention

    Maturity: Level 1 of 3

    The ratio of closed (not merged MRs) vs merged MRs over time.

    Measure amount of throw away work vs merged.

    URL(s)

    Health: Attention

    Maturity: Level 1 of 3

    Other PI Pages

    Legends

    Maturity

    Level Meaning
    Level 3 of 3 Has a description, target, and periscope data.
    Level 2 of 3 Missing one of: description, target, or periscope data.
    Level 1 of 3 Missing two of: description, target, or periscope data.
    Level 0 of 3 Missing a description, a target, and periscope data.

    Health

    Level Meaning
    Okay The KPI is at an acceptable level compared to the threshold
    Attention This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
    Problem We'll prioritize our efforts here
    Unknown Unknown

    How to work with pages like this

    Data

    The heart of pages like this is a data file called /data/performance_indicators.yml which is in YAML format. Almost everything you need to do will involve edits to this file. Here are some tips:

    Pages

    Pages like /handbook/engineering/performance-indicators/ are rendered by and ERB template.

    These ERB templates call the helper function performance_indicators() that is defined in /helpers/custom_helpers.rb. This helper function calls in several partial templates to do it's work.

    This function takes a required argument named org in string format that limits the scope of the page to a portion of the data file. Possible valid values for this org argument are listed in the org property of each element in the array in /data/performance_indicators.yml.