Stage and Group Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Stage and Group Performance Indicators

On this page

• Stage PI Summary
• Group PI Summary
• Stage PI Details
  • Manage - SMAU - MAU Viewing Analytics Features
  • Plan - SMAU - MAU Interacting With Issues
  • Create - SMAU - MAU Conducting Git Write Operations
  • Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner - SMAU, GMAU - Unique users triggering pipelines
  • Package, Package:Package - SMAU, GMAU - Count of users publishing/installing packages
  • Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab
  • Configure, Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters
  • Monitor - SMAU - Proxy user count based on prometheus integration
• Group PI Details
  • Manage:Access - GMAU - MAU Using Enhanced Authentication
  • Manage:Access - Paid GMAU - MAU Using Paid SAML
  • Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  • Manage:Import - GMAU - MAU importing
  • Manage:Analytics - GMAU - MAU Viewing Analytics Features
  • Plan:Project Management - GMAU - MAU Interacting With Issues
  • Plan:Project Management - Paid GMAU - MAU Interacting With Issues
  • Plan:Portfolio Management - Paid GMAU - MAU creating Epics each Month
  • Plan:Certify - Paid GMAU - MAU Interacting with Requirements
  • Create:Source Code - GMAU - MAU Conducting Git Write Operations to project repository
  • Create:Knowledge - GMAU - MAU that viewed or uploaded designs
  • Create:Static Site Editor - GMAU - MAU that committed via the SSE
  • Create:Editor - GMAU - MAU Conducting Git Write Operations from web editors
  • Create:Ecosystem - GMAU - MAU of any integration
  • Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner - SMAU, GMAU - Unique users triggering pipelines
  • Verify:Testing - Paid GMAU - Count of active paid testing feature users
  • Package, Package:Package - SMAU, GMAU - Count of users publishing/installing packages
  • Package:Package - Paid GMAU - Count of paid users using the Dependency Proxy
  • Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab
  • Configure, Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters
  • Monitor:Health - GMAU - Unique users that interact with Alerts and Incidents
• Other PI Pages
• Legends
  • Health
  • Instrumentation
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Stage PI Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	PI Workflow
Manage	SMAU	MAU Viewing Analytics Features	Attention	Both
Plan	SMAU	MAU Interacting With Issues	Okay	Both
Create	SMAU	MAU Conducting Git Write Operations	Unknown	Both
Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner	SMAU, GMAU	Unique users triggering pipelines	Okay	Both	Complete
Package, Package:Package	SMAU, GMAU	Count of users publishing/installing packages	Unknown	Both	Instrument Tracking
Release, Release:Progressive Delivery, Release:Release Management	SMAU, GMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure, Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor	SMAU	Proxy user count based on prometheus integration	Okay	Both	Complete

Group PI Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	PI Workflow
Manage:Access	Paid GMAU	MAU Using Paid SAML	Unknown	Both
Manage:Compliance	Paid GMAU	MAU Viewing Compliance Features	Unknown	Both
Manage:Analytics	GMAU	MAU Viewing Analytics Features	Okay	Both
Plan:Project Management	GMAU	MAU Interacting With Issues	Okay	Both
Plan:Project Management	Paid GMAU	MAU Interacting With Issues	Attention	Both
Plan:Portfolio Management	Paid GMAU	MAU creating Epics each Month	Okay	Both
Plan:Certify	Paid GMAU	MAU Interacting with Requirements	Unknown	Both
Create:Source Code	GMAU	MAU Conducting Git Write Operations to project repository	Attention	Both
Create:Knowledge	GMAU	MAU that viewed or uploaded designs	Attention	SaaS
Create:Static Site Editor	GMAU	MAU that committed via the SSE	Attention	Both
Create:Ecosystem	GMAU	MAU of any integration	Unknown	Both
Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner	SMAU, GMAU	Unique users triggering pipelines	Okay	Both	Complete
Verify:Testing	Paid GMAU	Count of active paid testing feature users	Unknown	SaaS	Instrument Tracking
Package, Package:Package	SMAU, GMAU	Count of users publishing/installing packages	Unknown	Both	Instrument Tracking
Release, Release:Progressive Delivery, Release:Release Management	SMAU, GMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure, Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor:Health	GMAU	Unique users that interact with Alerts and Incidents	Unknown	Both	Instrument Tracking

Stage PI Details

Manage - SMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page.

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Chart (Sisense↗)

Health: Attention

• Classified as "Attention" since we're able to measure SMAU, but need to add additional instrumentation before we can set a stage goal
• Planned improvements in 13.4 for Analytics - instance-level overview, issues analytics improvements, and DevOps report
• Planned improvements in 13.5 for Analytics views - group-level DevOps Report

Instrumentation: Attention

• First iteration of SMAU (analytics views) merged into 13.3 for self-managed and GitLab.com
• Incorporating other groups into SMAU currently pending instrumentation updates in 13.4 (see https://gitlab.com/groups/gitlab-org/-/epics/3690)
• After instrumentation is verified, please see this issue to track dashboard updates

Plan - SMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

Free/Paid: Both

SaaS/SM: Both

Target: 196400

Is Primary PI: true

Chart (Sisense↗)

Health: Okay

• Insight - 9.5% MoM growth from July to August driven by 37% growth in self-managed Project Management GMAU.
• Driver - Iterations & Epic Swimlanes should drive wider adoption for Plan - especially Paid SMAU

Instrumentation: Okay

• Dashboarding Update - dashboard now includes UUs interacting with issues across .com and self-managed. Thanks to the data team for pulling this together.

Create - SMAU - MAU Conducting Git Write Operations

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.

Free/Paid: Both

SaaS/SM: Both

Target: Our FY21 Q3 Target is to exceed 1m unique users performing Git write operations. As instances upgrade to GitLab 13.3 with improved instrumentation, we intend to switch to growth rate target aligned with user growth rates.

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/230481
• https://gitlab.com/gitlab-org/gitlab/-/issues/234027
• https://gitlab.com/gitlab-data/analytics/-/issues/5629

Health: Unknown

• Insight - Git write operations are performed by a large number of users. On GitLab.com, approximately 10% of users perform a Git write operation every month. For licensed self-hosted instances, this number should be much higher.
• This is the Create SMAU and the Gitaly GMAU. Please note, the primary Gitaly metric is Availability.

Instrumentation: Attention

• Enabled on GitLab.com for testing, and is scheduled for self-hosted in GitLab 13.3. https://gitlab.com/gitlab-org/gitlab/-/issues/230481
• Dashboard under construction https://gitlab.com/gitlab-data/analytics/-/issues/5629
• AMAU is being tracked per repository type, and which means that we are under counting slightly by using project repositories at the moment. https://gitlab.com/gitlab-org/gitlab/-/issues/234027

Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner - SMAU, GMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: Our FY21 Q3 Target is to see 10% growth. We are tracking towards a goal of 500k unique users triggering pipelines by end of FY21 (Jan 31, 2021).

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/verify/continuous-integration/#usage-funnel
• https://about.gitlab.com/handbook/engineering/development/ci-cd/verify/runner/#usage-funnel

Chart (Sisense↗)

Health: Okay

• Rationale - CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features. Unique users who start pipelines are clearly engaging with CI, and so this is a clean measure that we can start with to track this.
• Improvement - One area of improvement we want to invest in is the authoring experience, particularly around getting to your first successful pipeline quickly. This will help more teams adopt GitLab CI, improving SMAU.
• Improvement - Users engaging with pipelines but not necessarily triggering them is also important, but we are not yet considering that.

Instrumentation: Okay

• With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new .gitlab-ci.yml files created.
• We are also working to instrument other performance indicators for Verify:CI
• We are also adding instrumentation for time to first green build to measure Verify:Pipeline Authoring efficiency.

Package, Package:Package - SMAU, GMAU - Count of users publishing/installing packages

The maximum distinct count of users who either published a package or interacted with an installed package from the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Instrument Tracking

Target: n/a

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/package/#aarrr-framework

Health: Unknown

• Improvements - In 13.4, we intend to start measuring GMAU/SMAU for the Package stage via usage ping.

Instrumentation: Problem

• We have an issue open to measure this data via the Usage ping. This work is on-track to be completed in 13.4.

Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: Our FY21 Q3 Target is to see a growth of 24%.

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/release/#release-smau--release-management--progressive-delivery-gmau-usage-funnel
• https://app.periscopedata.com/app/gitlab/640196/New-WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU
• https://app.periscopedata.com/app/gitlab/661492/Temp---Eli-Release-Stage-Examples?widget=8578558&udv=0

Chart (Sisense↗)

Health: Okay

• Insight - In August we surpassed 100K unique users triggering deployments, nearly doubling the percent growth from the prior month. FY21 Q2 at 21%. We hope to hit this target again and slightly increase it in FY21 Q3.
• Improvements - We are seeing a very steep increase since June for self-managed users - nearly 15,000 users, with SaaS users trending as expected with ~2K users. We see that the greatest percentage increase came from Silver and Gold accounts.
• Improvements - By delivering more at scale features in Release Management, to include sharing releases and environments at the group level we expect to positively influence the reach to users planning releases and deploying to many environments. On the Progressive Delivery front, continued investment in making it easy to deploy to AWS will reduce friction to deploying to AWS as target.

Instrumentation: Okay

• We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
• We need to Expand Revenue Attribution to Release SMAU
• Improvements - Counter was instrumented for 28-days
• Improvements - User and unique count of deployments are instrumented
• Improvements - SaaS and self-managed data are both available on a single chart

Configure, Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: 5k Configure SMAU by the end of Q3FY21 (20% growth)

Is Primary PI: true

Chart (Sisense↗)

Health: Okay

• Insight - 17% increase from Jul/Aug puts the metric on target for hitting Q3 goal.

Instrumentation: Attention

• Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
• Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
• Improvement - we want to start collecting activity-based data with the GitLab Kubernetes Agent to be released soon

Monitor - SMAU - Proxy user count based on prometheus integration

A count of all users in projects that have a connected Prometheus integration because project users receive benefit from the cluster without direct usage.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: 11k Monitor SMAU by the end of Q3FY21 (20% growth)

Is Primary PI: true

Chart (Sisense↗)

Health: Okay

• Insight - Far surpassed goal
• Insight - We are investigating the sudden spike in this metric

Instrumentation: Attention

• Utilizes proxy SMAU that we need to replace

Group PI Details

Manage:Access - GMAU - MAU Using Enhanced Authentication

The number of unique users logging into GitLab with LDAP, SAML, OAuth, or Smartcard strategies on any tier of GitLab.

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Health: Unknown

• No target defined yet
• The Manage:Access group's focus is Paid GMAU so GMAU won't be an area of focus in the short term

Instrumentation: Problem

• Defined, instrumentation in 13.4

Manage:Access - Paid GMAU - MAU Using Paid SAML

The number of unique users logging into GitLab using SAML SSO for Groups.

Free/Paid: Paid

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Unknown

• No target defined yet
• Launched the ability to configure the default role assigned for SSO linked users in 13.3
• The ability to configure SSO in Gitlab.com without giving users access to all groups is planned for 13.4
• We are refining SSO group sync and plan to start in 13.5

Instrumentation: Problem

• Defined, instrumentation in 13.4

Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features

The number of unique users viewing Audit Events, Compliance Dashboard, or Credential Inventory.

Free/Paid: Paid

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Unknown

• Export audit events to CSV expected in 13.4
• Add a Projects view to the Compliance Dashboard. Planned for Q3 2020.
• Improve PAT & SSH credential management (epic). Ongoing with expected completion in 13.6
• Add additional credentials (epic) to the credential inventory. Planned for Q4 2020

Instrumentation: Attention

• Defined and instrumented in 13.3. Data availability anticipated in mid-Sept.

Manage:Import - GMAU - MAU importing

A monthly count of the unique number of users performing an import.

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Health: Unknown

• No target defined yet
• Improvement - To enable users to self-service GitLab migrations, the Import group is currently focused on starting to iterate toward the Complete solution for GitLab-to-GitLab migrations.
• Improvement - Continue to enhance the GitHub Importer with emphasis on supporting larger migrations.

Instrumentation: Problem

• Defined, instrumentation in progress, targeted for 13.4

Manage:Analytics - GMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page (https://docs.gitlab.com/ee/user/analytics/).

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Chart (Sisense↗)

Health: Okay

• Classified as "Okay" - expect to see an increase with the introduction of new throughput analytics page)
• Planned improvements in 13.4 - instance-level overview, issues analytics improvements, and DevOps report
• Planned improvements in 13.5 - Instance statistics & Enterprise DevOps Reports

Instrumentation: Attention

• Dashboard issue pending development

Plan:Project Management - GMAU - MAU Interacting With Issues

The number of unique users interacting with Issues each month.

Free/Paid: Both

SaaS/SM: Both

Target: 196400

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/722212/Plan:-Project-Management-GMAU

Chart (Sisense↗)

Health: Okay

• Result - 9.5% MoM growth from July to August driven by 37% growth in self-managed GMAU.
• Improvement - Work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
• Improvement - Continue making progress on quality of life improvements for existing features.

Instrumentation: Attention

• Implementation - Missing many interaction events from self-managed that cannot performantly be unioned until telemetry adds a new counter type so the target / current MAU is likely much lower than once full implementation is complete.

Plan:Project Management - Paid GMAU - MAU Interacting With Issues

The number of paid unique users interacting with Issues each month.

Free/Paid: Paid

SaaS/SM: Both

Target: 50400

Is Primary PI: true

URL(s)

• https://about.gitlab.com/direction/plan/issue_tracking/

Chart (Sisense↗)

Health: Attention

• Result - 0% MoM growth from July to August, with small increases in self-managed cancelling out small decreases in SaaS.
• Improvement - Continue high level efforts on simplifying groups/projects and making issues more extensible while also making incremental progress on iterations.
• Improvement - Continue making incremental, iterative progress towards Plan's core JTBD as outlined in the Issue Tracking direction.

Instrumentation: Attention

• Implementation - Lacking a standardized data model to generate a blended dashboard that accurately distinguishes between Paid and Free users.

Plan:Portfolio Management - Paid GMAU - MAU creating Epics each Month

The number of unique users creating Epics each month.

Free/Paid: Paid

SaaS/SM: Both

Target: 1,500 Unique Users for Self Managed and 1,200 Unique Users for SaaS

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5280

Chart (Sisense↗)

Health: Okay

• Insight - GMAU dipped slightly, but this is in line with the 2-3mo cycle we consistently see as larger customers tend to heavily plan quarterly. Overall monthly epic creation hit an all time high of +6200
• Improvement - Swimlanes will be a significant driver of MAU growth for Portfolio Management in the nearterm (n2m) - giving visibility to the feature on boards and adding additional value to better visualize your in-flight issues through swimlanes. Epic Boards should continue the trend into the end of FY21 by enabling Portfolio-level planning.

Instrumentation: Okay

• Dashboarding - we've instrumented core metrics but are looking to expand the MAU definition to include all epic usage, not just creation, for a more accurate GMAU metric

  Issue. 2020-09 update - we are identifying the additional instrumentation needed to accomplish this.

Plan:Certify - Paid GMAU - MAU Interacting with Requirements

The number of unique users interacting (create, view, reference) each month

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Unknown

• Dashboarding not implemented yet so health is a proxy of our Certify's "other" metric and adhnoc anlaysis. Tracking issue
• Improvement - UX improvements for Requirements Management based on completed research study, Requirement Import functionality, Upcoming Quality Management MVC

Instrumentation: Attention

• Dashboarding - Requirement Creation is instrumented and dashboarods are created. Work to track additional interactions is in progress. Tracking issue here

Create:Source Code - GMAU - MAU Conducting Git Write Operations to project repository

A rolling count of the number of unique users who performed a Git write operation to a project repository within the last 28 days

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0
• https://gitlab.com/gitlab-org/gitlab/-/issues/230481
• https://gitlab.com/gitlab-org/gitlab/-/issues/230481
• https://gitlab.com/gitlab-data/analytics/-/issues/5629

Chart (Sisense↗)

Health: Attention

• No target defined
• Released in 13.3, not enough data
• Improvement - work towards providing users more insights while navigating their code with Code Intelligence
• Improvement - work towards providing more control over managing repository size

Instrumentation: Okay

• Enabled on GitLab.com for testing
• Released for self-hosted in GitLab 13.3

Create:Knowledge - GMAU - MAU that viewed or uploaded designs

Monthly unique users that have viewed or uploaded designs.

Free/Paid: Both

SaaS/SM: SaaS

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/234041
• https://gitlab.com/gitlab-data/analytics/-/issues/4761

Chart (Sisense↗)

Health: Attention

• No target defined yet
• Recent improvements - Initially we want to drive usage and feature adoption. We moved the feature to core and moved designs up in the UI so they will be seen by users. The chart above just shows Design Views on .com
• Improvement - We plan to show designs in more places in the app to increase adoption

Instrumentation: Attention

• Have .com data for design views
• Instrumenting usage ping and updates/creates for 13.5 - this pushed back due to holidays/parental leave

Create:Static Site Editor - GMAU - MAU that committed via the SSE

Monthly unique users that have made at least one commit via the static site editor.

Free/Paid: Free

SaaS/SM: Both

Target: 110 MAU by September 30, 2020

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/684213/WIP-Static-Site-Editor-Usage-Metrics
• https://about.gitlab.com/direction/create/static_site_editor/

Chart (Sisense↗)

Health: Attention

• Goal - drive early adoption, with a focus on internal GitLab team members editing the Handbook
• Insight - Strong growth driven by making the Static Site Editor available for all Handbook pages
• Improvement - Add MR Title and Description when submitting changes from the Static Site Editor
• Improvement - Address issues reported from Handbook editors that block full-time adoption

Instrumentation: Attention

• Usage Ping data is missing so the data doesn't reflect self-hosted users
• Improvement - Instrumenting for Usage Ping in 13.5

Create:Editor - GMAU - MAU Conducting Git Write Operations from web editors

A rolling count of the number of unique users who performed a Git write operation to any respository using the Web IDE, Web Editor or Snippet within a stage in the last 28 days

Free/Paid: Free

SaaS/SM: Both

Target: TBD

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4014

Health: Unknown

• Data is not reporting yet for this metric, will continue to watch as data collection begins.

Instrumentation: Attention

• The metric has been instrumented and turned on for GitLab.com (as of 8 Sept. 2020). It is expected to be turned on in the GitLab 13.4 release to start collecting data from self-managed.

Create:Ecosystem - GMAU - MAU of any integration

Monthly Unique Users of any GitLab integration.

Free/Paid: Both

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/225678
• https://gitlab.com/gitlab-data/analytics/-/issues/5850

Health: Unknown

• No target defined
• Improvement - Adding Group and Instance integration will allow larger sets of projects to be integrated easily
• Improvement - Improvements and new functionality in the Jira integration should increase adoption, expanding on the success of our most popular integration

Instrumentation: Problem

• Awaiting implementation of data in Usage Ping
• Awaiting implementation of Sisense dashboard (blocked by Usage Ping implementation)

Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner - SMAU, GMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: Our FY21 Q3 Target is to see 10% growth. We are tracking towards a goal of 500k unique users triggering pipelines by end of FY21 (Jan 31, 2021).

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/verify/continuous-integration/#usage-funnel
• https://about.gitlab.com/handbook/engineering/development/ci-cd/verify/runner/#usage-funnel

Chart (Sisense↗)

Health: Okay

• Rationale - CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features. Unique users who start pipelines are clearly engaging with CI, and so this is a clean measure that we can start with to track this.
• Improvement - One area of improvement we want to invest in is the authoring experience, particularly around getting to your first successful pipeline quickly. This will help more teams adopt GitLab CI, improving SMAU.
• Improvement - Users engaging with pipelines but not necessarily triggering them is also important, but we are not yet considering that.

Instrumentation: Okay

• With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new .gitlab-ci.yml files created.
• We are also working to instrument other performance indicators for Verify:CI
• We are also adding instrumentation for time to first green build to measure Verify:Pipeline Authoring efficiency.

Verify:Testing - Paid GMAU - Count of active paid testing feature users

A rolling count of unique users who have interacted with a paid testing feature in the last 28 days

Free/Paid: Paid

SaaS/SM: SaaS

PI Workflow: Instrument Tracking

Target: TBD

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/verify/testing/#performance-indicators

Chart (Sisense↗)

Health: Unknown

• Insights - We enabled tracking for one feature of Visual Reviews and have found very low usage especially compared to views of the docs page.
• Improvements - We will be recording a walk through of how to setup and use Visual Reviews to add to the documentation page before the next review.
• Improvements - We are targeting delivery of Group Code Coverage features and tracking in 13.4.

Instrumentation: Attention

• Improvements - Implementation of page views for existing paid features is in progress.
• Improvements - We are targeting tracking of Web Performance engagement in 13.5.

Package, Package:Package - SMAU, GMAU - Count of users publishing/installing packages

The maximum distinct count of users who either published a package or interacted with an installed package from the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Instrument Tracking

Target: n/a

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/package/#aarrr-framework

Health: Unknown

• Improvements - In 13.4, we intend to start measuring GMAU/SMAU for the Package stage via usage ping.

Instrumentation: Problem

• We have an issue open to measure this data via the Usage ping. This work is on-track to be completed in 13.4.

Package:Package - Paid GMAU - Count of paid users using the Dependency Proxy

The maximum distinct count of paid users who either published a package or interacted with an installed package from the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Free/Paid: Paid

SaaS/SM: Both

PI Workflow: Instrument Tracking

Target: n/a

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/package/#usage-funnels

Health: Unknown

• Not yet instrumented

Instrumentation: Problem

• We have an issue open to measure Dependency Proxy adoption via the Usage ping. It is currently scheduled for milestone 13.6.

Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: Our FY21 Q3 Target is to see a growth of 24%.

Is Primary PI: true

URL(s)

• https://about.gitlab.com/handbook/engineering/development/ci-cd/release/#release-smau--release-management--progressive-delivery-gmau-usage-funnel
• https://app.periscopedata.com/app/gitlab/640196/New-WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU
• https://app.periscopedata.com/app/gitlab/661492/Temp---Eli-Release-Stage-Examples?widget=8578558&udv=0

Chart (Sisense↗)

Health: Okay

• Insight - In August we surpassed 100K unique users triggering deployments, nearly doubling the percent growth from the prior month. FY21 Q2 at 21%. We hope to hit this target again and slightly increase it in FY21 Q3.
• Improvements - We are seeing a very steep increase since June for self-managed users - nearly 15,000 users, with SaaS users trending as expected with ~2K users. We see that the greatest percentage increase came from Silver and Gold accounts.
• Improvements - By delivering more at scale features in Release Management, to include sharing releases and environments at the group level we expect to positively influence the reach to users planning releases and deploying to many environments. On the Progressive Delivery front, continued investment in making it easy to deploy to AWS will reduce friction to deploying to AWS as target.

Instrumentation: Okay

• We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
• We need to Expand Revenue Attribution to Release SMAU
• Improvements - Counter was instrumented for 28-days
• Improvements - User and unique count of deployments are instrumented
• Improvements - SaaS and self-managed data are both available on a single chart

Configure, Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Complete

Target: 5k Configure SMAU by the end of Q3FY21 (20% growth)

Is Primary PI: true

Chart (Sisense↗)

Health: Okay

• Insight - 17% increase from Jul/Aug puts the metric on target for hitting Q3 goal.

Instrumentation: Attention

• Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
• Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
• Improvement - we want to start collecting activity-based data with the GitLab Kubernetes Agent to be released soon

Monitor:Health - GMAU - Unique users that interact with Alerts and Incidents

Unique users that interacted with alerts or incidents

Free/Paid: Both

SaaS/SM: Both

PI Workflow: Instrument Tracking

Target: TBD once we understand current activity

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/668012/Health-Group-Dashboard
• https://about.gitlab.com/handbook/engineering/development/ops/monitor/health/#north-star-metric

Health: Unknown

• Improvements - Health Group is focused on table-stakes features to build user base. In the next few milestones we are going to finish building out Incidents as a type of issue and move on to enabling users to integrate any tool with GitLab via custom integrations. Making it easier to integrate alert sources into GitLab will increase adoption of all Incident Management features.

Instrumentation: Problem

• Improvements - Instrument GMAU as an aggregate counter in the usage ping for significant AMAUs - progress tracked in this issue.

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Threat Management Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Secure & Defend Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• Data Team KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

Instrumentation

Value	Level	Meaning
3	Okay	We are correctly capturing and measuring data for GitLab.com and self-managed instances.
2	Attention	Missing one of: GitLab.com or self-managed data.
1	Problem	Missing both: GitLab.com and self-managed data.
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI.
urls	Optional	list of urls associated with the (K)PI
sisense_data	Optional	contains elements/properties related to Sisense, including chart (numeric Sisense widget ID), dashboard (numeric Sisense dashboard ID), shared_dashboard (Sisense shared dashboard ID), embed (v2)
sisense_data_secondary	Optional	same as sisense_data. Allows a second sisense chart to be included.
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU)
plan_type	Optional	indicates if the metric is Free, Paid, or Both.
telemetry_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
instrumentation	Optional	has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI.
pi_workflow	Optional	indicates the PI Workflow status

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a single Green bar for the current month (in progress).
  • Use a Red stepped-line for timeseries target.
  • Directional targets will be used:
    • Above ...
    • Below ...
    • At ...
    • At or above ...
    • At or below ...
  • Optional: Use a Black dashed-line for rolling average.
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :
• urls: should be an array (indented lines starting with dashes) even if you only have one url
• Both maturity.level and health.level display a value between 0 and 3 (inclusive).