Ops Section Product Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Ops Section Product Performance Indicators

On this page

• Intent
• Section PI Overview
• Regular Performance Indicators
  • Ops - Section TMAU - Sum of Ops Section SMAU
  • Ops - Section SpU - Ops Section Contribution to Stages Per User
  • Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines
  • Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file
  • Verify:Pipeline Authoring - Other - % first green project
  • Verify:Runner - Other - Linux Shared (SaaS) Runners Minutes Used
  • Verify:Runner - Other - Windows Shared Runners (SaaS) Minutes Used
  • Verify:Runner - Other - Total downloads of runner binaries and docker images per month
  • Verify:Testing - Paid GMAU - Count of active paid testing feature users
  • Package:Package - SMAU, GMAU - Count of users publishing packages
  • Package:Package - Other - Packages published using the Package Registry
  • Release:Release - SMAU - Count of users triggering deployments using GitLab
  • Release:Release - AMAU - Unique users creating Releases
  • Release:Release - AMAU - Unique users with at least 1 project on their environments dashboard
  • Release:Release Management - AMAU - Unique users Associating Group Milestones to Releases
  • Release:Release - AMAU - Unique users of Pages
  • Release:Release - AMAU - Unique users triggering feature flags feature flag toggles
  • Release:Release - AMAU - Unique users triggering automatic deployments pipelines with automatic deployments (no manual jobs)
  • Release:Release - AMAU - Unique users triggering review apps pipelines that created a review app
  • Release:Release - AMAU - Count of unique users using canary or incremental rollouts users using Canary or Incremental Rollout
  • Release:Release - AMAU - Count of users deploying to AWS AutoDevOps or Ci templates to deploy to AWS
  • Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters
  • Configure:Configure - Adoption PI - Number of projects with GitLab Managed Terraform State
  • Configure:Configure - Adoption PI - Number of Gitlab Kubernetes Agent instances
  • Configure:Configure - AMAU - Unique users running pipelines that have a HashiCorp Vault Variable
  • Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents
  • Monitor:Monitor - Other - Count of projects where incidents are being created
  • Monitor:Monitor - AMAU - Alerts Active Users
  • Monitor:Monitor - AMAU - Incidents Active Users
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Intent

We are organizing for and regularly reviewing Performance Indicators (PIs) in order to enable a dialog between ourselves (all of R&D) about most effective use of our R&D efforts towards the most impactful improvements. As a result we should:

• focus on singular Performance Indicators
• focus on the art of the possible, what we can measure and react to quickly
• focus on business metrics, particularly SMAU
• prioritize for the intent of representing return on business investment (SMAU/GMAU that covers SaaS and Self-Managed) over the intent of building a rapid experimentation capability. This will have the practical effect of PMs prioritizing instrumentation and funnel building in Usage Ping over Snowplow.
• continue beyond measurement and review of our Performance Indicators to building a Growth Model, a Growth funnel and supporting metrics and segmentation

As a communication tool, Performance Indicators are only as useful as the range of the audience. As a result we should:

• store PI metrics, funnels, growth models, status and next steps handbook first
• expose our product groups to PI metrics regularly
• communicate transparently about adjustments to PIs within our groups
• reference PIs in planning issues and prioritization discussions

Section PI Overview

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Ops	Section TMAU	Sum of Ops Section SMAU	Problem	Both	Implementation
Ops	Section SpU	Ops Section Contribution to Stages Per User	Okay	Both	Complete
Verify, Verify:CI	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Verify:Pipeline Authoring	GMAU	Number of unique users interacting with gitlab-ci.yml file	Unknown	Both	Implementation
Verify:Runner	Other	Total downloads of runner binaries and docker images per month	Unknown	SM	Definition
Verify:Testing	Paid GMAU	Count of active paid testing feature users	Unknown	Both	Data Availability
Package:Package	SMAU, GMAU	Count of users publishing packages	Attention	Both	Data available
Release:Release	SMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Okay	Both	Complete

Regular Performance Indicators

Ops - Section TMAU - Sum of Ops Section SMAU

A sum of all SMAUs across the five stages in the Ops Section (Verify, Package, Release, Configure & Monitor)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 1.5M by End of Q4

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/781120/Ops-Section-Dashboard

Metric Name: ci_pipelines, incident_management_total_unique_counts_monthly, deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Problem

• We did not achieve our FY21 Q4 Goal despite adding another Stage MAU in the final month of the quarter.
• Our trend over the quarter has been down - we'd considered December to a seasonal blip, but it appears to be part of a trend.
• There is still data completeness concerns around January's data that we'll need to revisit in March.

Implementation Status: Implementation

• Still missing well defined Configure SMAU.

Lessons Learned (January):

• Insight - There remains a disconnect between user and non-user activity and our MAU metrics. We seen increases in non-userfied metrics (Terraform State, Projects with Incident Management, Runner Downloads, Packages uploaded, etc) that aren't reflective of in unique user counts. This points to a completeness of metric capture concern.

Ops - Section SpU - Ops Section Contribution to Stages Per User

In order to get a sense of the increase in overall stages per user attributed to the Ops Section stages we divide Section TMAU / Overall UMAU. This is analagous to Product-wide KPIs of TMAU and SMAU and helps us visualize the relative dispersion of the completeness of adoption across stages. The nuber is not a percent - but is the actual stages count from the overall stages per user that was provided by Ops Section stages.

SaaS/SM: Both

Target: 0.5 by end of Q1

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/781120/Ops-Section-Dashboard

Chart (Sisense↗)

Health: Okay

• Target recently set for FY22Q1

Implementation Status: Complete

• Target recently set, Feb data seems to be showing despite not being complete. Hoping to rebound from the down-trend in Jan.

Lessons Learned (January):

• Improvement - Focusing on cross-stage collaboration and moving users through funnels (Verify->Package and Verify->Release) specifically.

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: We delivered on 92% of our FY21 Q4 target of 1 million unique users triggering pipelines, and fell short of our 10% MoM growth. In FY22 Q1, we anticipate accomplishing this target of 1.1M unique users as predicted in our [tertiary chart](https://app.periscopedata.com/app/gitlab/798616/WIP-Linear-Predicted-Dashboard?widget=10611135).

Is Primary PI: true

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Metric is instrumented for both SaaS and Self-managed

Implementation Status: Complete

• With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new gitlab-ci.yml files created.
• We are also working to instrument other performance indicators for Verify:CI
• Updating dashboard to have both SM / SaaS and Paid / All breakouts

Lessons Learned (January):

• Insight - In January 2021 we saw a slight uptick in users again, mostly concentrated in the Paid, self-managed user base for continuous integration. Although, we have not delivered on our ambitious 1.14M unique users triggering pipelines, we are close at 917K users meaning only ~220K users shy of attaining our goal.
• Improvement - With the plateau in monthly active users, we proceeding with an evaluation of maturity of our core Jobs To Be Done in Continuous Integration to offer a challenge or affirmation for the "Loveable" status.
• Improvement - We are finalizing the research in adoption hurdles for SCM to Verify to further expand opportunities for SpO as well as increase MAU. We will be prioritizing the next generation of features in CI that target these gaps once we invest in addressing technical debt, bugs, and UI polish in the 1H of FY22.
• Improvement - Lastly, we are beginning to explore the idea of NextGen CI, which is related to the SCM to Verify adoption challenges, while tackling other interesting problem sets like ML/AI in CI.

Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file

Count the number of users that are pushing MRs to gitlab-ci.yml files. Since we'd like to count the on-going usage of our users with pipelines.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: N/A

Chart

Health: Unknown

• Not instrumented

Implementation Status: Implementation

• Instrumentation for .com is undergoing and currently is in-review, we should have some of the data ready on our next PI meeting.
• Still waiting on instrumentation for usage ping data which is scheduled for 13.10. It appears that the data exists; however, the data team was facing challenges extracting it.
• Evaluated for FY21-Q4 OKRs but not part of FY21-Q3 KRs

Lessons Learned (January):

• Insights - We delivered the first MVC for Pipeline Editor, which has recieved great traction with preliminary signals on social such as the LinkedIn post got 149 likes & 1800+ views in 24 hours in addition to it receiving the most clicks from the "What's New" in 13.8 so far.
• Improvements - Secondary improvements that will continue to drive this metric, will include the empty state for pipeline editor, which will allow new users to easily start using pipeline editor. Creating an versatile empty template, which will empower users to interact with the YAML file in any number of ways that are currently not supported today, and Branch selector for pipeline editor to allow users to work with the pipeline editor from any non-master branch.

Verify:Pipeline Authoring - Other - % first green project

Count of projects that have the first passing pipeline in the 24 hours and the 28 days after project creation as a percentage of all new projects created in the time period. This ratio helps us understand how effectively we are highlighting and enabling the GitLab CI/CD capabilities to new project creators. Our desire is an increasing percentage of users complete in the first 24 hours.

SaaS/SM: SaaS

Target: We'd expect this to increase 1% each month for both metrics (24hr and 28 days).

Metric Name: green_project

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• We'd like to improve our onboarding experience since we know that our convertion rate multiplied if users are adopting CI

Implementation Status: Target

• The data currently in Sisense is only instrumented for .com and not self-managed.

Lessons Learned (January):

• Improvements - 2 metrics were added - Ratio of projects that have a first green in the first 24 hours and 28 days after creation.
• Insights - We have seen a slight increase in the ratio of projects with a first green pipeline in 24 hours after creation, about .3% (which is still far off from our target) and about 6K additional project in absolute numbers. We can likely attribute this to the efforts we have added to the pipeline authoring experience around linting, as well as the editor. It may take sometime before we will see the 28 days after creation to start to shift, but that is remaining flat from December.

Verify:Runner - Other - Linux Shared (SaaS) Runners Minutes Used

The count of CI minutes used per month on GitLab.com Linux Shared Runners with the Docker executor. These are CI minutes as defined by those pipelines that are executed on runner managers with the label shared-runners-manager-X. One of our strategic focus areas is increasing combined monthly active users (CMAU). Providing customers on GitLab SaaS, a performant, and full-featured build infrastructure supports that strategy. Therefore, we believe that measuring CI minutes usage by platform is a leading indicator of future performance. Also, there are several potential future investment areas for Linux Runners on GitLab SaaS, such as [offering multiple virtual machine types](https://gitlab.com/groups/gitlab-org/-/epics/2426). We must evaluate these investments with the strategic focus areas in mind.

Funnel Description: Handbook Link

SaaS/SM: SaaS

Target: Growth rate of 5% monthly.

Metric Name: NA

Chart (Sisense↗)

Health: Okay

• Metric is implemented and dashboards are created in Sisense.

Implementation Status: Target

Lessons Learned (January):

• Insights - Total Linux Docker executor minutes of 35,625,529 used in January 2021 reflect a 13% increase over December 2020. Since we expected to see significantly less usage in December due to the holidays, the actual growth rate is closer to 3%. We believe that the Verify stage's strength and continued improvements in CI is the current primary growth driver. The usage data also suggests that in addition to the CI offering's value proposition, users and customers derive value from the Runner Cloud build fleet on GitLab SaaS. A focus on this area, including various machine options, marketing, positioning, and competitive pricing, may yield dividends from a growth perspective.
• Improvements - The Linux Shared Runners with Docker Executor on .com is an area that will need continued investment as this solution addresses a significant percentage of CI use cases for users and customers on GitLab SaaS. From a future investment perspective, implementing new autoscaling technology can improve service quality and reduce infrastructure costs. We will also need to evaluate offering more options in terms of GCP machine types. This translates to customer value in several areas, such as reducing job execution time for those jobs that can benefit from larger CPU and memory VM instance types.

Verify:Runner - Other - Windows Shared Runners (SaaS) Minutes Used

The count of CI minutes used per month on GitLab.com Windows Shared Runners with the Shell executor. These are CI minutes as defined by those pipelines that are executed on runner managers with the label windows-shared-runners-manager-X. We must have a clear understanding of use and adoption to more effectively forecast the impact of future investments in GitLab SaaS Windows Runners on our strategic focus areas.

Funnel Description: Handbook Link

SaaS/SM: SaaS

Target: Growth rate of 20% monthly.

Metric Name: NA

Chart (Sisense↗)

Health: Okay

• Metric is implemented and dashboards are created in Sisense.

Implementation Status: Target

• We need to refine the dashboards to simplify self-service reporting.

Lessons Learned (January):

• Insight - Total Windows minutes of 400,635 used in January 2021 and 1.19 million for the fiscal year 2021. Windows usage on .com has been growing positively month over month, primarily driven by organic growth. The data suggest that more investment in Windows as a platform will yield a higher rate of growth. However, our current strategy is to prioritize the Linux and macOS fleet investments ahead of Windows in the first half of FY22.

Verify:Runner - Other - Total downloads of runner binaries and docker images per month

Count of downloads of runner binaries grouped by platform, gitlab-runner and gitlab-runner helper container images. Measuring runner binaries' downloads will provide insight into whether we are making the right investments in core runner features and capabilities by platform (operating system and computing architecture). The assumption is that if self-managed users continue to maintain and upgrade their GitLab Runner build infrastructure, this may suggest continued long-term use of GitLab as their DevOps platform.

Funnel Description: Handbook Link

SaaS/SM: SM

Target: TBD

Is Primary PI: true

Metric Name: NA

Health: Unknown

• Not instrumented

Implementation Status: Definition

• We need to start collecting the AWS S3 bucket metrics and integrate them into the data warehouse. Issue for tracking is in the Infra Project pending scheduling.

Lessons Learned (January):

• Insight - Total downloads of 19,400,000 of the gitLab-runner docker image in January 2021 reflects a 4% increase from December 2020, with an average growth in downloads per month from November 2020 to January 2021 at 3.42%. From the current data, we see a continued upward trajectory in the adoption and use of GitLab CI. In thinking about the usefulness of this data, we can posit that a material downward trajectory in downloads could be a leading indicator that we are losing mindshare in the market.

Verify:Testing - Paid GMAU - Count of active paid testing feature users

A rolling count of unique users who have interacted with a paid testing feature in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: We are targeting X unique paid testing users by end of FY22 Q1.

Is Primary PI: true

Metric Name: NA

Chart (Sisense↗)

Health: Unknown

• Individual Metrics are implemented and in the Dashboard.
• An aggregate metric for unique users interacting with any paid Testing Feature is on track to review after 13.9 release.

Implementation Status: Data Availability

• The target is missing in the chart.

Lessons Learned (January):

• Insights - Considering growth trends in the previous chart, between December 2020 and January 2021 there was a 13% growth of snowplow sessions, which is showing a level of retention in addition to further engagement from either the existing user or more users.
• Insights - Comparing overall clicks on the Code Quality tab against Unique Users of the feature, even with a small data set, implies strong repeat usage.
• Improvements - Instrumentation of unique user tracking for all existing paid features on track to complete in 13.9. This has moved up since the last review.
• Improvements - There are several bug fixes in progress and scheduled for Browser Performance Testing and the Group Code Coverage page that should increase engagement in those features and overall unique users.

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Health: Attention

• Improvements - As of January 1st, 2021, we now have user level data for Package.

Implementation Status: Data available

• Improvements - We expect a bit of a long tail as Self-Managede (pinging) instances upgrade and send data

Package:Package - Other - Packages published using the Package Registry

A count of events in which a package is published to the Package Registry on GitLab.com.

Funnel Description: Handbook Link

SaaS/SM: SaaS

Target: We have exceeded the original target of 240k packages published per month for FY21 Q4 and have met our modified target of 300k. The FY22 Q1 target (April 30, 2021) will be 400k packages published per month. Reminder the aforementioned goals are for GitLab.com.

Metric Name: NA

Chart (Sisense↗)

Health: Okay

• Improvements - We modified this metric to only look at the number of packages published and not installed. The reason being that installing packages is a passive action and may not signal user activity for Package.

Implementation Status: Target

• As of January 1st, we now have user-level data for Package.
• The target is missing in the chart.

Lessons Learned (January):

• Improvement - In 13.9 we plan to remove the strict naming convention for npm packages. This popular issue has also been identified by several customers as a blocker in adopting the Package Registry.
• Improvement - In 13.9 we plan to add support for v2 of Composer. This popular issue is frequently requested and has been a source of some negative Community sentiment.
• Insight - Although we see growth across all formats, we are beginning to see steady growth for generic packages. This feature has also been mentioned as important on the last 7 customer calls.

Release:Release - SMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 301,000 for SMAU, 124,000 for paid SMAU

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/640196/New-WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU
• https://app.periscopedata.com/app/gitlab/661492/Temp---Eli-Release-Stage-Examples?widget=8578558&udv=0

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - Our FY21 Q4 is 30% and this month, and we did not meet our target. The increase in deployments was only at 11% in this quarter.
• Insight - We have exceeded our yearly target for paid SMAU. We are currently at 134k users. Target was 124k.
• Improvements - Even with the end of year deployment freeze, there was still an increase of 6K users this month.
• Improvements - Release has recently shipped some of the DORA4 metrics and specifically deployment frequency which gives the user insights into their deployment efficiency and should encourage them to increase deployments.
• Improvements - In the next few milestones, we will add in-product guidance for users that have pipelines that are not triggering deployments.

Implementation Status: Complete

• We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
• We need to Expand Revenue Attribution to Release SMAU
• Improvements - Instrumentation was changed to the formal metric recommended by the Product Intelligence team, making the data more accurate.
• Improvements - Counter was instrumented for 28-days
• Improvements - User and unique count of deployments are instrumented
• Improvements - SaaS and self-managed data are both available on a single chart
• Updating dashboard to have Paid / All breakouts
• Improvements - Target has been fixed and reflected in the charts.

Release:Release - AMAU - Unique users creating Releases

A rolling count of users creating releases in the last 28 days.

SaaS/SM: Both

Target: Our FY21 Q4 is 10%.

Metric Name: release_management

Chart (Sisense↗)

Health: Okay

• Insights - We have exceeded our quaterly goal and reached a 13.2% increase in users creating releases!
• Improvements - We have published the release-cli binaries to S3 making it easier for users to consume.

Implementation Status: Target

• We are currently tracking this data for both .com and self-managed.

Release:Release - AMAU - Unique users with at least 1 project on their environments dashboard

A rolling count of unique users that have at least one project on their environment dashboard in the last 28 days.

SaaS/SM: Both

Target: TBD

Metric Name: TBD

Health: Unknown

• Not yet instrumented

Implementation Status: Definition

• Improvements - Defined the metrics and scheduled issues for implementation in gitlab&4198

Release:Release Management - AMAU - Unique users Associating Group Milestones to Releases

A rolling count of unique users that have at least one release associated to a group milestone in the last 28 days.

SaaS/SM: Both

Target: TBD

Metric Name: TBD

Health: Unknown

• Not yet instrumented

Implementation Status: Definition

• Improvements - Defined the metrics and scheduled issues for implementation in gitlab&4198

Release:Release - AMAU - Unique users of Pages

A rolling count of unique users that have a Pages project in the last 28 days.

SaaS/SM: Both

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard?widget=10598857&udv=0

Metric Name: pages

Chart (Sisense↗)

Health: Okay

• Instrumentation done for Self-Managed

Implementation Status: Instrumentation done for Self-Managed

• Improvements - There was an increase of 6.8% in creation of custom domains.
• Improvement - Added direct links to pages templates to help users get started with pages.

Release:Release - AMAU - Unique users triggering feature flags feature flag toggles

A rolling count of users triggering feature flag toggles with Gitlab in the last 28 days.

SaaS/SM: Both

Target: FY22Q1 increase of 10% in Unique users triggering feature flags.

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard?widget=10279590&udv=0

Metric Name: feature_flags

Chart (Sisense↗)

Health: Okay

• Insights - The number of unique users toggling feature flags has increased by nearly 12% from last month.

Release:Release - AMAU - Unique users triggering automatic deployments pipelines with automatic deployments (no manual jobs)

A rolling count of users triggering automatic deployments (in without any manual activity) with Gitlab in the last 28 days.

SaaS/SM: Both

Target: TBD

Metric Name: TBD

Health: Unknown

• Not yet instrumented

Implementation Status: Definition

• Improvements - Create the instrumentation Instrument AMAU of CD

Release:Release - AMAU - Unique users triggering review apps pipelines that created a review app

A rolling count of unique users triggering pipelines which created a review app with Gitlab in the last 28 days.

SaaS/SM: Both

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard?widget=10279590&udv=0

Metric Name: review_apps

Chart (Sisense↗)

Health: Okay

• Instrunmentation in place for self-managed.

Implementation Status: Definition

• Improvements - Increase of ~4% from previous month

Release:Release - AMAU - Count of unique users using canary or incremental rollouts users using Canary or Incremental Rollout

A rolling count of users using advanced deployments, namely, canary and incremental rollouts with Gitlab in the last 28 days.

SaaS/SM: Both

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard?widget=10279628&udv=0

Metric Name: advanced_deployments

Chart (Sisense↗)

Health: Okay

• Instumentation done for Canary for SaaS

Implementation Status: Definition

• Improvements - Create the instrumentation Instrument AMAU of Canary Rollouts
• Improvements - Create the instrumentation Instrument AMAU of Incremental Rollouts

Release:Release - AMAU - Count of users deploying to AWS AutoDevOps or Ci templates to deploy to AWS

A rolling count of users using using AutoDevOps or Ci templates to deploy to AWS based on cloud_service:$CI_CLUSTER_CLOUD_SERVICE_TYPE in the last 28 days.

SaaS/SM: Both

Target: TBD

Metric Name: Users deploying to AWS

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Snowplow data instrumented

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

SaaS/SM: Both

Target: 18000

Is Primary PI: true

Metric Name: project_clusters_enabled, group_clusters_enabled, instance_clusters_enabled

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Improvement - We are working towards moving cluster creation over to Terraform in order to provide customizable, IaC based cluster provisioning. This requires a production ready GitLab Managed Terraform State.

Implementation Status: Complete

• The target was fixed, and there is a simple pattern how it can be adjusted in the widget
• Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
• Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
• Improvement - The GitLab Kubernetes Agent has been released, and it's first usage pings are coming in. See below.
• Updating dashboard to have both SM / SaaS and Paid / All breakouts
• The Data Team strongly advises against estimating this value given that it is already an estimate. An estimate of an estimate yields very inaccurate results.

Lessons Learned (January):

• Insight - As the metrics got fixed in August, we are still trying to get an idea about the natural variation versus trends in the data. As a result, we reach the target from month to month even though the target is for the quarter. As the most recent iteration in this quest, the target was modified to 18000 SMAU.

Configure:Configure - Adoption PI - Number of projects with GitLab Managed Terraform State

A rolling count of the number of projects using GitLab Managed Terraform State in the last 28 days

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 6000 by end of Q1FY22

URL(s)

• https://app.periscopedata.com/app/gitlab/677628/Infrastructure-as-Code

Metric Name: projects_with_terraform_states

Chart (Sisense↗)

Health: Okay

• Improvement - The target was added to the chart using a simple to maintain pattern

Implementation Status: Target

• We are still missing the namespace/tier based visualisations for SaaS, currently SaaS is counted as an Ultimate user
• We have shipped a MAU based metric in 13.8, but no data arrived yet

Lessons Learned (January):

• Insight - We've reached our target in 2 months again! We already have 4000+ GitLab Managed Terraform States being reported.
• Insight - We see a clear interest among Ultimate users
• Improvement - We want to ship two requested features Protected Terraform states and State version comparison to increase adoption further
• Improvement - We are working towards moving cluster creation over to Terraform in order to provide customizable, IaC based cluster provisioning. This requires a production ready GitLab Managed Terraform State.

Configure:Configure - Adoption PI - Number of Gitlab Kubernetes Agent instances

A rolling count of the number of GitLab Kubernetes Agent instances in the last 28 days

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 100 by the end of Q4FY21

URL(s)

• https://app.periscopedata.com/app/gitlab/656814/Kubernetes-integration

Metric Name: kubernetes_agents

Chart (Sisense↗)

Health: Problem

Implementation Status: Target

• The instrumentation was already added to usage ping, we're waiting for the first data points to come home
• We want to switch to MAU based metric once adoption picks up a bit

Lessons Learned (January):

• Insight - adoption takes longer. The initial MVC requires a new workflow from our users, and we were missing two important features for serious adoption
• Insight - we see the first customers showing interest in using the agent, but no measurable usage is reported yet.
• Insight - getting a new component out to gitlab.com takes several months even despite outstanding support from the GitLab Infra team
• Improvement - we want to provide a CI tunnel to provide the security benefits of the agent without requiring a change in the user workflows. This might allow an upgrade path from the certificate based setups, if we decide to deprecate it.
• Improvement - we've proved that the Agent works with OpenShift, unlike most of the certificate-based integrations.

Configure:Configure - AMAU - Unique users running pipelines that have a HashiCorp Vault Variable

A rolling count of unique users that have triggered pipelines with a HashiCorp Vault Variable in the last 28 days.

SaaS/SM: Both

Target: no target

URL(s)

• https://app.periscopedata.com/app/gitlab/788824/Secrets-Management

Chart (Sisense↗)

Health: Okay

• Removing the target. It was 700 AMAU, and we've had around 150 instead. As Secrets Management is not a high priority direction for the team, I'm removing the target now.
• Improvement - Added a target. It's an educated guess, but at least it's a line in the sand. It's based on the adoption trend of Terraform state files.
• Instrumentation was added, we're waiting for the first data to arrive

Implementation Status: Dashboard

• We have data!

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interact with alerts or incidents. (This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority.)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 12,000 Estimated total SMAU

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• We met and surpassed our FY21Q4 usage target of 12k SMAU

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (January):

• Insight - There is a data issue for January 2021. Insights for Monitor SMAU/GMAU remain the same as those for December 2020.
• Insight - SMAU for December has decreased from November 2020. We believe this to be due to the holiday season.
• Insight - Monthly interactions on Alerts are far lower than those on Incidents. As an example, we can compare the AMAU for users changing the status on alerts from December (38) to the number of users who manually created an incident in December, which was ~2,300. Our hypothesis is two-fold. First, we believe configuring alerts is challenging. We addressing this by making it easier to integrate alerting sources into GitLab by enabling users to customize the endpoints. Second, we know that it is very common for teams to create incidents manually. They do this for two reasons - 1. Many teams do not have performance alerting automation set up and therefore rely on a human reporting a problem and 2. Teams that do have automatic performance alerting still want to create incident manually because they want a human to triage the alert first.
• Improvements - We are making it easier to integrate alerting sources into GitLab by enabling users to customize the endpoints.
• Improvements - We are adding on-call schedule management, which is a required feature set for Incident Management to compete in market.

Monitor:Monitor - Other - Count of projects where incidents are being created

Count of projects creating incidents

SaaS/SM: Both

Target: 12,000 projects where incidents are being created for GitLab.com by the end of Q4 - January 2021

Metric Name: projects_creating_incidents

Chart (Sisense↗)

Health: Okay

• We crushed our target for FY21Q4 by doubling it. We've set a new target for FY22Q1 assuming 100% retention and 50% growth.

Implementation Status: Complete

• Instrumentation complete - issue

Lessons Learned (January):

• Insight - Number of projects creating incidents continues to increase indicating adoption and retention.
• Insight - We have the most projects creating incidents in Ultimate. This is because Ultimate organizations tend to be bigger and have far more projects than customers in other tiers.
• Insight - We have the second most projects creating incidents in Core. This is because this is our largest user base.
• Improvement - We are ramping up sales enablement efforts for Incident Management as we approach the 'complete' maturity. Incident Management will be positioned as a value add for Premium & Ultimate.

Monitor:Monitor - AMAU - Alerts Active Users

Count of users performing an action that involves an alert which includes creating an alert, editing an alert, or interacting with features and actions in the side bar of the alert detail page

SaaS/SM: Both

Target: TBD

Metric Name: incident_management_alerts_total_unique_user_counts

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Problem

• Even though we are seeing growth, the overall number of users utilizing Alerts is very low. We are focusing our efforts here.

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (January):

• Insights - Interactions on Alerts are much lower than Incidents. This means users are primarily responding to problems manually. We hope to see an uptick in Alerts utlization after we make it easier to integrate alerting sources with GitLab.
• Improvements - We are focusing on this area of Incident Management by making it easier to integrate alert sources into GitLab.

Monitor:Monitor - AMAU - Incidents Active Users

Count of users performing an action that involves an incident which includes creating an incident, editing an incident, or interacting with features and actions in the side bar of the incident detail page

SaaS/SM: Both

Target: TBD

Metric Name: incident_management_incidents_total_unique_user_counts

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Number of users interacting with incidents continues to increase month over month.

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (January):

• Insights - Number of users interacting with incidents recovered following the holiday season, which we can see on the Incident WAU chart. Incidents MAU continues to grow month over month increasing 15% in total December to January.
• Improvements - We've received an increasing amount of feedback on how incidents interact on issue boards. We are doing discovery into how we can improve this experience. Follow along on this issue.

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Threat Management Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a Red stepped-line for timeseries target.
  • Directional targets will be used:
    • Above ...
    • Below ...
    • At ...
    • At or above ...
    • At or below ...
  • Optional: Use a single Green bar for the current month (in progress). Preferred for metrics which accumulate through a month but optional due to the performance impact to render the chart.
  • Optional: Use a Black line for rolling average.
• Each Sisense dashboard for KPIs should consider the following settings to ensure timely updates:
  • Setting up auto-refresh for a frequency that fits the KPI
  • Excluding Dashboards from Auto Archive
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
• Multi-series performance indicators should consider the following guidelines:
  • If series are mutually exclusive, use stacked bars for each series with a monthly time series
  • If series are not mutually exclusive, use grouped bars for each series with a monthly time series
  • Do not graph any targets in the chart.
  • Current month styling guidelines will not apply
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :