We are organizing for and regularly reviewing Performance Indicators (PIs) in order to enable a dialog between ourselves (all of R&D) about most effective use of our R&D efforts towards the most impactful improvements. As a result we should:
As a communication tool, Performance Indicators are only as useful as the range of the audience. As a result we should:
| Stage/Group | PI Type | PI | Health | SaaS / SM | Implementation Status |
|---|---|---|---|---|---|
| Ops | Section CMAU | Sum of Ops Section SMAU | Okay | Both | Complete |
| Ops | Section SpU | Ops Section Contribution to Stages Per User | Attention | Both | Complete |
| Verify, Verify:CI | GMAU, SMAU | Unique users triggering pipelines | Okay | Both | Complete |
| Verify:Pipeline Authoring | GMAU | Number of unique users interacting with gitlab-ci.yml file | Okay | Both | Implementation |
| Verify:Runner | Other | Total downloads of runner binaries and docker images per month | Unknown | SM | Definition |
| Verify:Testing | Paid GMAU | Count of active paid testing feature users | Unknown | Both | Complete |
| Package:Package | SMAU, GMAU | Count of users publishing packages | Okay | Both | Complete |
| Release:Release | SMAU | Count of users triggering deployments using GitLab | Okay | Both | Complete |
| Configure:Configure | SMAU, GMAU | MAU of GitLab Managed Terraform State | Okay | Both | Complete |
| Monitor:Monitor | SMAU, GMAU | Unique users that interact with Alerts and Incidents | Okay | Both | Complete |
A sum of all SMAUs across the five stages in the Ops Section (Verify, Package, Release, Configure & Monitor)
Funnel Description: Handbook Link
SaaS/SM: Both
Target: 1.55M by End of FY22-Q1
Is Primary PI: true
URL(s)
Metric Name: ci_pipelines, incident_management_total_unique_counts_monthly, deployments, packages, Managed Terraform State
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Tertiary Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
In order to get a sense of the increase in overall stages per user attributed to the Ops Section stages we divide Section CMAU / Overall UMAU. This is analagous to Product-wide KPIs of CMAU and SMAU and helps us visualize the relative dispersion of the completeness of adoption across stages. The nuber is not a percent - but is the actual stages count from the overall stages per user that was provided by Ops Section stages.
SaaS/SM: Both
Target: 0.5 by end of Q1
Is Primary PI: true
URL(s)
Chart (Sisense↗)
Health: Attention
Implementation Status: Complete
Lessons Learned (April):
A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.
SaaS/SM: Both
Target: In FY22 Q1, we anticipate accomplishing this target of 1.1M unique users as predicted in our [tertiary chart](https://app.periscopedata.com/app/gitlab/798616/WIP-Linear-Predicted-Dashboard?widget=10611135).
Is Primary PI: true
Metric Name: ci_pipelines
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
Count the number of users that are pushing MRs to gitlab-ci.yml files. Since we'd like to count the on-going usage of our users with pipelines.
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target for FY22-Q1 is 150,000 unique users interacting with the gitlab-ci.yml file on SaaS.
Is Primary PI: true
Metric Name: redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly
Chart (Sisense↗)
Health: Okay
Implementation Status: Implementation
Lessons Learned (April):
Count of projects that have the first passing pipeline in the 24 hours and the 28 days after project creation as a percentage of all new CI projects that created in the time period. This ratio helps us understand how effectively we are highlighting and enabling the GitLab CI/CD capabilities to new project creators. Our desire is an increasing percentage of users complete in the first 24 hours.
SaaS/SM: SaaS
Target: We'd expect this to increase 1% each month for both metrics (24hr and 28 days).
Metric Name: green_project
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Okay
Implementation Status: Target
Lessons Learned (April):
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
This chart displays the split between CI minutes executed on GitLab SaaS Shared Runners compared to CI minutes on self-hosted runners for GitLab SaaS customers in Ultimate and Premium tiers.
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target percentage composition is open for discussion. A few key questions that we need to consider are as follows. First is the majority of the usage of self-hosted runners by GitLab SaaS customers due to internal security and compliance requirements that dictate the use of self-hosted runners. Secondly, are the CI build needs of GitLab SaaS customers not being met by the current Shared Runner offer, including different levels of computing performance or other capabilities such as GPU support.
Metric Name: shared-runner-vs-self-managed
Chart (Sisense↗)
Health: Okay
Implementation Status: Target
Lessons Learned (April):
This chart displays the split between CI minutes executed on GitLab SaaS Shared Runners compared to CI minutes on self-hosted runners for GitLab SaaS customers in the Free tier.
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target percentage composition is open for discussion.
Metric Name: shared-runner-vs-self-managed-free
Chart (Sisense↗)
Health: Okay
Implementation Status: Target
Lessons Learned (April):
The count of CI minutes used per month on GitLab.com Linux Shared Runners with the Docker executor. These are CI minutes as defined by those pipelines that are executed on runner managers with the label shared-runners-manager-X. One of our strategic focus areas is increasing combined monthly active users (CMAU). Providing customers on GitLab SaaS, a performant, and full-featured build infrastructure supports that strategy. Therefore, we believe that measuring CI minutes usage by platform is a leading indicator of future performance. Also, there are several potential future investment areas for Linux Runners on GitLab SaaS, such as offering multiple virtual machine types. We must evaluate these investments with the strategic focus areas in mind.
Funnel Description: Handbook Link
SaaS/SM: SaaS
Target: Growth rate of 5% monthly.
Metric Name: NA
Chart (Sisense↗)
Health: Okay
Implementation Status: Target
Lessons Learned (April):
The count of CI minutes used per month on GitLab.com Windows Shared Runners with the Shell executor. These are CI minutes as defined by those pipelines that are executed on runner managers with the label windows-shared-runners-manager-X. We must have a clear understanding of use and adoption to more effectively forecast the impact of future investments in GitLab SaaS Windows Runners on our strategic focus areas.
Funnel Description: Handbook Link
SaaS/SM: SaaS
Target: Growth rate of 20% monthly.
Metric Name: NA
Chart (Sisense↗)
Health: Okay
Implementation Status: Target
Lessons Learned (April):
Count of downloads of runner binaries grouped by platform, gitlab-runner and gitlab-runner helper container images. Measuring runner binaries' downloads will provide insight into whether we are making the right investments in core runner features and capabilities by platform (operating system and computing architecture). The assumption is that if self-managed users continue to maintain and upgrade their GitLab Runner build infrastructure, this may suggest continued long-term use of GitLab as their DevOps platform.
Funnel Description: Handbook Link
SaaS/SM: SM
Target: TBD
Is Primary PI: true
Metric Name: NA
Health: Unknown
Implementation Status: Definition
Lessons Learned (April):
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
This chart displays the SLO attainment trends for GitLab SaaS CI Runners. The error-rate and Apdex score are the two metrics (Service Level Indicators) used to calculate the SLO attainment metric.
SaaS/SM: Both
Target: 99.95% SLO attainment monthly.
Metric Name: gitlab-saas-ci-runners-slo-trends
Chart
Health: Okay
Implementation Status: Target
Lessons Learned (April):
A rolling count of unique users who have interacted with a paid testing feature in the last 28 days.
Funnel Description: Handbook Link
SaaS/SM: Both
Target: We are targeting 17,500 unique paid testing users by end of FY22-Q1.
Is Primary PI: true
Metric Name: NA
Chart (Sisense↗)
Health: Unknown
Implementation Status: Complete
Lessons Learned (April):
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target for FY22-Q1 is 40,000 unique users publishing packages across SaaS and self-managed combined.
Is Primary PI: true
Metric Name: user_packages_total_unique_counts_monthly
Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
A count of events in which a package is published to the Package Registry on GitLab.com.
Funnel Description: Handbook Link
SaaS/SM: SaaS
Target: The FY22 Q1 target (April 30, 2021) is 400k packages published per month on GitLab.com. We exceeded this number in the first month of the quarter. The new target will be 475k packages published.
Metric Name: NA
Chart (Sisense↗)
Health: Okay
Implementation Status: Target
Lessons Learned (April):
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
A rolling count of unique users triggering deployments with Gitlab in the last 28 days.
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target for FY22-Q1 is 453,000 for unique users triggering deployments, and 213,000 for paid unique users triggering deployments.
Is Primary PI: true
URL(s)
Metric Name: deployments
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Tertiary Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
A rolling count of users creating releases in the last 28 days.
SaaS/SM: Both
Target: Our FY22 Q1 is 30%.
Metric Name: release_management
Chart (Sisense↗)
Health: Okay
Implementation Status: Target
A rolling count of unique users that have at least one project on their environment dashboard in the last 28 days.
SaaS/SM: Both
Target: TBD
Metric Name: TBD
Health: Unknown
Implementation Status: Definition
A rolling count of unique users that have at least one project on their environment dashboard in the last 28 days.
SaaS/SM: Both
Target: TBD
URL(s)
Metric Name: Environment Instances
Chart (Sisense↗)
Health: Okay
Implementation Status: Definition
A rolling count of unique users that have at least one release associated to a group milestone in the last 28 days.
SaaS/SM: Both
Target: TBD
Metric Name: TBD
Health: Unknown
Implementation Status: Definition
A rolling count of unique users that have a Pages project in the last 28 days.
SaaS/SM: SM
Target: TBD
URL(s)
Metric Name: pages
Chart (Sisense↗)
Health: Okay
Implementation Status: Instrumentation done for Self-Managed
A rolling count of users triggering feature flag toggles with Gitlab in the last 28 days.
SaaS/SM: SaaS
Target: FY22Q1 increase of 10% in Unique users triggering feature flags.
URL(s)
Metric Name: feature_flags
Chart (Sisense↗)
Health: Okay
A rolling count of users triggering automatic deployments (in without any manual activity) with Gitlab in the last 28 days.
SaaS/SM: Both
Target: TBD
Metric Name: TBD
Health: Unknown
Implementation Status: Definition
A rolling count of unique users triggering pipelines which created a review app with Gitlab in the last 28 days.
SaaS/SM: Both
Target: TBD
URL(s)
Metric Name: review_apps
Chart (Sisense↗)
Health: Okay
Implementation Status: Definition
A rolling count of users using advanced deployments, namely, canary and incremental rollouts with Gitlab in the last 28 days.
SaaS/SM: Both
Target: TBD
URL(s)
Metric Name: advanced_deployments
Chart (Sisense↗)
Health: Okay
Implementation Status: Definition
A rolling count of users using using AutoDevOps or Ci templates to deploy to AWS based on cloud_service:$CI_CLUSTER_CLOUD_SERVICE_TYPE in the last 28 days.
SaaS/SM: Both
Target: TBD
Metric Name: Users deploying to AWS
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Okay
Usage ping data showing users who are interacting with the CI/CD Analytics page
SaaS/SM: Both
Target: TBD
Metric Name: Users viewing DORA4 metrics
Chart (Sisense↗)
Health: Okay
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
Monthly active users of GitLab Managed Terraform State
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target for FY22-Q1 is 5000 monthly active users of GitLab Managed Terraform State
Is Primary PI: true
URL(s)
Metric Name: redis_hll_counters.terraform.p_terraform_state_api_unique_users_monthly
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
A rolling count of the number of GitLab Kubernetes Agent instances in the last 28 days
Funnel Description: Handbook Link
SaaS/SM: Both
Target: 100 by the end of Q2FY22
URL(s)
Metric Name: kubernetes_agents
Chart (Sisense↗)
Health: Attention
Implementation Status: Target
Lessons Learned (April):
A rolling count of unique users that have triggered pipelines with a HashiCorp Vault Variable in the last 28 days.
SaaS/SM: Both
Target: no target
URL(s)
Chart (Sisense↗)
Health: Okay
Implementation Status: Dashboard
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
Unique users that interact with alerts or incidents. (This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority.)
Funnel Description: Handbook Link
SaaS/SM: Both
Target: The target for FY22-Q1 is 15,000 estimated total unique users that interact with alerts and incidents.
Is Primary PI: true
URL(s)
Metric Name: incident_management_total_unique_counts_monthly
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
Count of projects creating incidents
SaaS/SM: Both
Target: 15,000 projects by the end of FY22Q1
Metric Name: projects_creating_incidents
Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
Count of users performing an action that involves an alert which includes creating an alert, editing an alert, or interacting with features and actions in the side bar of the alert detail page
SaaS/SM: Both
Target: TBD
Metric Name: incident_management_alerts_total_unique_user_counts
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Problem
Implementation Status: Complete
Lessons Learned (April):
Count of users performing an action that involves an incident which includes creating an incident, editing an incident, or interacting with features and actions in the side bar of the incident detail page
SaaS/SM: Both
Target: TBD
Metric Name: incident_management_incidents_total_unique_user_counts
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Okay
Implementation Status: Complete
Lessons Learned (April):
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 20 minutes per month
Health: Problem
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name} |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section) |
section |
Optional | the product section (Ex: dev) as defined in sections.yml |
stage |
Optional | the product stage (Ex: release) as defined in stages.yml |
group |
Optional | the product group (Ex: progressive_delivery) as defined in stages.yml |
category |
Optional | the product group (Ex: feature_flags) as defined in categories.yml |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI. |
health.level |
Optional | indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI. |
health.reasons |
Optional | indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
urls |
Optional | list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url |
funnel |
Optional | indicates there is a handbook link for a description of the funnel for this PI. Should be a URL |
sisense_data |
Optional | allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes. |
sisense_data.chart |
Optional | indicates the numeric Sisense chart/widget ID. For example: 9090628 |
sisense_data.dashboard |
Optional | indicates the numeric Sisense dashboard ID. For example: 634200 |
sisense_data.shared_dashboard |
Optional | indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b |
sisense_data.embed |
Optional | indicates the Sisense embed version. For example: v2 |
sisense_data_secondary |
Optional | allows a second Sisense dashboard to be embeded. Same as sisense data |
sisense_data_secondary.chart |
Optional | Same as sisense_data.chart |
sisense_data_secondary.dashboard |
Optional | Same as sisense_data.dashboard |
sisense_data_secondary.shared_dashboard |
Optional | Same as sisense_data.shared_dashboard |
sisense_data_secondary.embed |
Optional | Same as sisense_data.embed |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
pi_type |
Optional | indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI) |
product_analytics_type |
Optional | indicates if the metric is available on SaaS, SM (self-managed), or Both. |
is_primary |
Optional | boolean flag that indicates if this is the Primary PI for the Product Group. |
implementation |
Optional | indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI. |
implementation.status |
Optional | indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI. |
implementation.reasons |
Optional | indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
lessons |
Optional | indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI. |
lessons.learned |
Optional | learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned |
monthly_focus |
Optional | indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI. |
monthly_focus.goals |
Optional | indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal |
metric_name |
Optional | indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name. |
Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.
