Stage and Group Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Stage and Group Performance Indicators

On this page

• SMAU Summary
• GMAU Summary
• Other Stage PIs Summary
• Other Group PIs Summary
• SMAU PI Details
  • Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
  • Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
  • Create - SMAU - MAU Conducting Git Write Operations
  • Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines
  • Package:Package - SMAU, GMAU - Count of users publishing packages
  • Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab
  • Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters
  • Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents
  • Sec, Secure - Section MAU, SMAU - Unique users who have used a Secure scanner
  • Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI
• GMAU PI Details
  • Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
  • Manage:Access - Paid GMAU - MAU Using Paid SAML
  • Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  • Manage:Import - GMAU - MAU importing
  • Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
  • Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
  • Create:Source Code - GMAU - MAU Conducting Git Write Operations to project repository
  • Create:Editor - GMAU - MAU Conducting Git Write Operations from web editors
  • Create:Ecosystem - GMAU - MAU of any integration
  • Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary
  • Enablement:Global Search - Paid GMAU - The number of unique paid users per month
  • Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com
  • Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines
  • Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file
  • Verify:Testing - Paid GMAU - Count of active paid testing feature users
  • Package:Package - SMAU, GMAU - Count of users publishing packages
  • Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab
  • Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters
  • Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents
  • Secure:Static Analysis - GMAU - Users running Static Analysis jobs
  • Secure:Dynamic Analysis - GMAU - Users running DAST
  • Secure:Composition Analysis - GMAU - Users running any SCA scanners
  • Secure:Fuzz Testing - GMAU - Users running fuzz testing
  • Secure:Threat Insights - GMAU - Users interacting with Secure UI
  • Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI
• Other Stage PIs Details
  • Enablement - Section PPI, Stage PPI - Median End User Latency
• Other Group PIs Details
  • Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab
  • Enablement:Memory - Group PPI - Memory Consumed
  • Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)
  • Verify:Runner - Other - Total downloads of runner binaries and docker images per month for self-managed
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

SMAU Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Analytics Features	Okay	Both	Complete
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Create	SMAU	MAU Conducting Git Write Operations	Okay	Both	Complete
Verify, Verify:CI	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Package:Package	SMAU, GMAU	Count of users publishing packages	Unknown	Both	In Progress
Release, Release:Progressive Delivery, Release:Release Management	SMAU, GMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Okay	Both	Complete
Sec, Secure	Section MAU, SMAU	Unique users who have used a Secure scanner	Okay	Both	Complete
Protect:Container Security	SMAU, GMAU	Users running Container Scanning or interacting with Protect UI	Okay	Both	Complete

GMAU Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Analytics Features	Okay	Both	Complete
Manage:Access	Paid GMAU	MAU Using Paid SAML	Okay	Both	Complete
Manage:Compliance	Paid GMAU	MAU Viewing Compliance Features	Unknown	Both	Complete
Manage:Import	GMAU	MAU importing	Unknown	Both	Complete
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Plan:Product Planning	Paid GMAU	MAU creating Epics each Month	Okay	Both	Complete
Create:Source Code	GMAU	MAU Conducting Git Write Operations to project repository	Okay	Both	Complete
Create:Editor	GMAU	MAU Conducting Git Write Operations from web editors	Unknown	Both	Complete
Create:Ecosystem	GMAU	MAU of any integration	Unknown	Both	Instrumentation
Enablement:Geo	Paid GMAU	Number of unique users utilizing a Geo secondary	Unknown	Both	Instrumentation
Enablement:Global Search	Paid GMAU	The number of unique paid users per month	Okay	Both	Complete
Enablement:Infrastructure	Paid GMAU	Number of unique paid users that perform an action on GitLab.com	Okay	SaaS	Complete
Verify, Verify:CI	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Verify:Pipeline Authoring	GMAU	Number of unique users interacting with gitlab-ci.yml file	Unknown	Both	Implementation
Verify:Testing	Paid GMAU	Count of active paid testing feature users	Problem	Both	Implementation
Package:Package	SMAU, GMAU	Count of users publishing packages	Unknown	Both	In Progress
Release, Release:Progressive Delivery, Release:Release Management	SMAU, GMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Okay	Both	Complete
Secure:Static Analysis	GMAU	Users running Static Analysis jobs	Okay	Both	Complete
Secure:Dynamic Analysis	GMAU	Users running DAST	Okay	Both	Complete
Secure:Composition Analysis	GMAU	Users running any SCA scanners	Okay	Both	Complete
Secure:Fuzz Testing	GMAU	Users running fuzz testing	Attention	Both	Definition
Secure:Threat Insights	GMAU	Users interacting with Secure UI	Attention	SaaS	Definition
Protect:Container Security	SMAU, GMAU	Users running Container Scanning or interacting with Protect UI	Okay	Both	Complete

Other Stage PIs Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement	Section PPI, Stage PPI	Median End User Latency	Attention	Both	Dashboard

Other Group PIs Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement:Distribution	Group PPI	Percentage of installations on the 3 most recent versions of GitLab	Attention	Both	Complete
Enablement:Memory	Group PPI	Memory Consumed	Attention	Both	Complete
Enablement:Database	Group PPI	Database Query Apdex (100ms target, 250ms tolerable)	Attention	Both	Complete
Verify:Runner	Other	Total downloads of runner binaries and docker images per month for self-managed	Unknown	SM	Definition

SMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page.

SaaS/SM: Both

Target: 40,150 per month (SaaS only). Self-managed target TBD.

Is Primary PI: true

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Starting in October, self-managed data is coming in as a result of implementing usage ping in 13.5. Analytics feature adoption on self-managed instances is at 64% of the estimated uplift. Overall usage (SaaS and self-managed) is at 96% of target. For paid users, usage is at 91% of target.

Implementation Status: Complete

• Please see this issue to track dashboard updates

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 437,490 for All, 101,835 for Paid

Is Primary PI: true

URL(s)

• https://about.gitlab.com/direction/plan/issue_tracking/
• https://app.periscopedata.com/app/gitlab/722212/Plan:-Project-Management-GMAU

Metric Name: issues

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Result - Surpassed GMAU goal of 196k (and 200k benchmark) recorded Self-Managed + SaaS SMAU. Set a new target for Q4 inclusive of estimated self-managed uplift.
• Driver - Iterations, Epic Swimlanes, Roadmap enhancements, Epic Boards, and Plan feature parity between an Instance, Group, and Project should drive wider adoption for Plan - especially Paid SMAU - over the next 3-6 months.
• Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
• Improvement - For All, continue making progress on quality of life improvements for existing features.
• Improvement - For Paid, continue high level progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible,and continuing to make incremental progress on iterations.
• Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption.
• Improvement - For Paid, continue making incremental, iterative progress towards Plan's core JTBD as outlined in the Issue Tracking direction.
• Insight - In Sep and Oct, MAU growth within existing instances far outpaced attrition, which is a strong signal of sustainable, organic adoption. More analysis is needed to filter by tier and identify top paying customers with elevated attrition and/or growth.
• Insight - During Q3, SaaS Trial users that interacted with Plan increased from 17% to 33% (94%). This is a strong signal that Plan is becoming more of a key consideration during the initial sales cycles.
• Insight - Plan adoption by version has improved from 33% in 12.7 to 45% in 13.4 – which represents a net improvement in adoption of 36% over the last 10 releases.

Implementation Status: Complete

• Updating dashboard to combine both Paid / All usage.
• Implementation - Completed adding interaction events from self-managed which allows us to count unique users performing multiple actions within Issues.
• Implementation - Next we are working on enhancing current xMAU reporting to include aggregate distinc counts. We need to figure out how to gracefully transition to the new counter without perceived xMAU loss.

Create - SMAU - MAU Conducting Git Write Operations

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 10% for Paid SMAU.

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard
• https://gitlab.com/gitlab-org/gitlab/-/issues/234027

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - Git write operations are performed by a large number of users. On GitLab.com, approximately 10% of users perform a Git write operation every month. For licensed self-managed instances, this number should be much higher.
• AMAU is being tracked per repository type, and which means that we are under counting slightly by using project repositories at the moment gitlab#234027. From GitLab 13.6 this should be resolved.

Implementation Status: Complete

• Enabled on GitLab.com
• Released for self-managed in GitLab 13.3

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: Our FY21 Q4 target is to see 10% growth month over month. We are tracking towards a goal of 680,000 unique users triggering pipelines by end of FY21 (Jan 31, 2021) as combined total for SaaS and Reported Self-Managed.

Is Primary PI: true

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - While growth between September and October (at 8.5%) was less than the 10% monthly target, we have exceeded our previous target of 500k unique users triggering pipelines by end of FY21; 511k is current count based on SaaS and Recorded Self-Managed data. We have adjusted our target to 1 million unique users triggering pipelines by end of FY21.
• Insight - Growth of unique SaaS user triggering pipelines was less than 4% between Sep and Oct, which is less than half the growth of Recorded Self-Managed users (at over 10%). This needs more analysis and research to understand factors for slower SaaS growth.
• Insight - Based on Q3 growth (SaaS and Recorded Self-Managed combined), the projected growth in Q4 is 9% per month.
• Improvement - Users engaging with pipelines but not necessarily triggering them is also important and to provide pipeline analytics for these users. In progress is reviewing research and insights for the key metrics for analyzing pipelines.

Implementation Status: Complete

• With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new .gitlab-ci.yml files created.
• We are also working to instrument other performance indicators for Verify:CI
• Updating dashboard to have both SM / SaaS and Paid / All breakouts

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: packages

Health: Unknown

• Improvements - In 13.6, we intend to start measuring GMAU/SMAU for the Package stage via usage ping.
• Improvements - Due to scalability and performance concerns, we have to revisit how we collect and store this data.

Implementation Status: In Progress

• We have an issue open to measure this data via the Usage ping. In milestone 13.6 we learned that we needed to modify the MR to use Redis. As a result this work may slip into 13.7.

Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: Our FY21 Q4 target will be 30% growth, sustaining at least 10% growth MoM, given that retention is at 89%.

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/640196/New-WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU
• https://app.periscopedata.com/app/gitlab/661492/Temp---Eli-Release-Stage-Examples?widget=8578558&udv=0

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - Our FY21 Q4 is 30% and this month, we have grown in unique users by 9%, a little lower than our 10% target and due to the seasonality of release creation we may see a larger increase at the end of November.
• Improvements - Over 10K more users are triggering deployments in self-managed instances, which is very encouraging given that retention is not 100%. We continue to see a 2.5K increase from SaaS this month.
• Improvements - Release Management has recently shipped the addition of assets and binaries to the Releases object, which is necessary to successfully distribute artifacts generated from releases from GitLab, the work in gitlab#17838 will futher support greater release automation reaching the CI audience and introducing them to Release Mangement capabilities. We are also beginning to see greater interest in GitLab Native Runbooks, which will positively impact enterprise release capabilities. On the Progressive Delivery front, we expect to positively impact top of funnel users with the introduction of feature flags to core in [gitlab#212318](https://gitlab.com/gitlab-org/gitlab/-/issues/212318. The ability for users to deploy easier with GitLab, will be futhered by the ECS templates make available in 13.5 with gitlab#201742](https://gitlab.com/gitlab-org/gitlab/-/issues/201742).

Implementation Status: Complete

• We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
• We need to Expand Revenue Attribution to Release SMAU
• Improvements - Counter was instrumented for 28-days
• Improvements - User and unique count of deployments are instrumented
• Improvements - SaaS and self-managed data are both available on a single chart
• Updating dashboard to have Paid / All breakouts
• Target is missing / misaligned in the charts.

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

SaaS/SM: Both

Target: 16000

Is Primary PI: true

Metric Name: project_clusters_enabled, group_clusters_enabled, instance_clusters_enabled

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - What is our new target?
• Improvement - We are working towards moving cluster creation over to Terraform in order to provide customizable, IaC based cluster provisioning. This requires a production ready GitLab Managed Terraform State.

Implementation Status: Complete

• Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
• Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
• Improvement - The GitLab Kubernetes Agent has been released, and it's first usage pings are coming in. See below.
• Updating dashboard to have both SM / SaaS and Paid / All breakouts
• The Data Team strongly advises against estimating this value given that it is already an estimate. An estimate of an estimate yields very inaccurate results.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interacted with alerts or incidents. _This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority._

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 12,000 Estimated total SMAU

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insights - Growth between September and October 2020 has surpassed the target of being 10% greater than the previous month. We've set a new target inorporating estimated SMAU uplift for self-managed. The target of 12,000 users is based on 100% retention and 25% growth. We are still seeing the results of new explosive growth for incident management. As the we get closer to the complete maturity for the category, we expect this value to continue to grow at a steady rate.
• Improvements - To fuel this growth, the Monitor Group is working to mature Incident management to complete by making it easier to integrate alert sources into GitLab and adding on-call schedule management. These features create an end-to-end incident management workflow that can compete in market.

Implementation Status: Complete

• Instrumentation complete - issue.

Sec, Secure - Section MAU, SMAU - Unique users who have used a Secure scanner

The number of unique users who have run one or more Secure scanners.

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: user_unique_users_all_secure_scanners

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• currently instrumentated to meet OKRs (see instrumentation, Lessons Learned for more details)

Implementation Status: Complete

• Can't currently differentiate between free-to-OSS/education Gold/Ultimate users and paid Gold/Ultimate users
• Threat Insights data is not yet included in this count until we support user-level metrics in Snowplow for self-managed

Focus This Month (November):

• Identify a way to denote underreporting on Secure-related *MAU charts to remove confusion / clarify metrics interpretation
• Identify usage by product tier while removing free-to-OSS/education Gold/Ultimate users
• Support Data team on improving Self-Managed Uplift for Secure

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

The number of users who have run a container scanning job or interacted with the Protect UI in the last 28 days of the month.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >2%

Is Primary PI: true

Metric Name: user_container_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• On track
• Decline in chart from September to October is likely due to lost data. Other charts indicate growth continued as expected at ~2% month over month

Implementation Status: Complete

• Limitations in data (see SCA limitations above)
• Data is not available yet for the Security Orchestration category as it has not yet reached Minimal maturity

Lessons Learned (October):

• TBD

Focus This Month (November):

• Allow container security scans to be run against containers in production environments (https://gitlab.com/groups/gitlab-org/-/epics/3410)
• Project-level DAST Scan Schedule Policies (https://gitlab.com/groups/gitlab-org/-/epics/4598)

GMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page.

SaaS/SM: Both

Target: 40,150 per month (SaaS only). Self-managed target TBD.

Is Primary PI: true

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Starting in October, self-managed data is coming in as a result of implementing usage ping in 13.5. Analytics feature adoption on self-managed instances is at 64% of the estimated uplift. Overall usage (SaaS and self-managed) is at 96% of target. For paid users, usage is at 91% of target.

Implementation Status: Complete

• Please see this issue to track dashboard updates

Manage:Access - Paid GMAU - MAU Using Paid SAML

The number of unique users logging into GitLab using SAML SSO for Groups.

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: user_auth_by_provider (max value)

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Launching SSO group sync for GitLab.com MVC in 13.6
• Releasing OneLogin support for SCIM and SAML in 13.6

Implementation Status: Complete

• Defined, instrumentation in 13.4
• Charts show the max value of the user_auth_by_provider set.

Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features

The number of unique users viewing Audit Events, Compliance Dashboard, or Credential Inventory.

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Unknown

• Add a Projects view to the Compliance Dashboard. Planned for Q3 2020.
• Improve PAT & SSH credential management (epic). Ongoing with expected epic completion in 13.8.
• Add additional credentials (epic) to the credential inventory. Planned for Q4 2020
• Improve discoverability of Compliance features. Planned for Q1 2021.

Implementation Status: Complete

Manage:Import - GMAU - MAU importing

A monthly count of the unique number of users performing an import.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: projects_imported (max value)

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Unknown

• No target defined yet
• Improvement - To enable users to self-service GitLab migrations, the Import group is currently focused on starting to iterate toward the Complete solution for GitLab-to-GitLab migrations.
• Improvement - Continue to enhance the GitHub Importer with emphasis on supporting larger migrations.

Implementation Status: Complete

• Defined and instrumented in 13.5
• The charts take the max value of the projects_imported, which only takes into account the most important import actions. A new combined metric for all import actions is planned.

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 437,490 for All, 101,835 for Paid

Is Primary PI: true

URL(s)

• https://about.gitlab.com/direction/plan/issue_tracking/
• https://app.periscopedata.com/app/gitlab/722212/Plan:-Project-Management-GMAU

Metric Name: issues

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Result - Surpassed GMAU goal of 196k (and 200k benchmark) recorded Self-Managed + SaaS SMAU. Set a new target for Q4 inclusive of estimated self-managed uplift.
• Driver - Iterations, Epic Swimlanes, Roadmap enhancements, Epic Boards, and Plan feature parity between an Instance, Group, and Project should drive wider adoption for Plan - especially Paid SMAU - over the next 3-6 months.
• Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
• Improvement - For All, continue making progress on quality of life improvements for existing features.
• Improvement - For Paid, continue high level progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible,and continuing to make incremental progress on iterations.
• Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption.
• Improvement - For Paid, continue making incremental, iterative progress towards Plan's core JTBD as outlined in the Issue Tracking direction.
• Insight - In Sep and Oct, MAU growth within existing instances far outpaced attrition, which is a strong signal of sustainable, organic adoption. More analysis is needed to filter by tier and identify top paying customers with elevated attrition and/or growth.
• Insight - During Q3, SaaS Trial users that interacted with Plan increased from 17% to 33% (94%). This is a strong signal that Plan is becoming more of a key consideration during the initial sales cycles.
• Insight - Plan adoption by version has improved from 33% in 12.7 to 45% in 13.4 – which represents a net improvement in adoption of 36% over the last 10 releases.

Implementation Status: Complete

• Updating dashboard to combine both Paid / All usage.
• Implementation - Completed adding interaction events from self-managed which allows us to count unique users performing multiple actions within Issues.
• Implementation - Next we are working on enhancing current xMAU reporting to include aggregate distinc counts. We need to figure out how to gracefully transition to the new counter without perceived xMAU loss.

Plan:Product Planning - Paid GMAU - MAU creating Epics each Month

The number of unique users creating Epics each month.

SaaS/SM: Both

Target: 1,500 Unique Users for Self Managed and 1,200 Unique Users for SaaS

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5280

Metric Name: epics

Chart (Sisense↗)

Health: Okay

• Insight - GMAU dipped slightly, but this is in line with the 2-3mo cycle we consistently see as larger customers tend to heavily plan quarterly. Overall monthly epic creation hit an all time high of +6200
• Improvement - Swimlanes will be a significant driver of MAU growth for Portfolio Management in the nearterm (n2m) - giving visibility to the feature on boards and adding additional value to better visualize your in-flight issues through swimlanes. Epic Boards should continue the trend into the end of FY21 by enabling Portfolio-level planning.

Implementation Status: Complete

• Dashboarding - we've instrumented core metrics but are looking to expand the MAU definition to include all epic usage, not just creation, for a more accurate GMAU metric

  Issue. 2020-09 update - we are identifying the additional instrumentation needed to accomplish this.

Create:Source Code - GMAU - MAU Conducting Git Write Operations to project repository

A rolling count of the number of unique users who performed a Git write operation to a project repository within the last 28 days

SaaS/SM: Both

Target: Our FY21 Q3 Target is to exceed 1m unique users performing Git write operations. As instances upgrade to GitLab 13.3 with improved instrumentation, we intend to switch to growth rate target aligned with user growth rates.

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0
• https://gitlab.com/gitlab-org/gitlab/-/issues/234027

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - Git write operations are performed by a large number of users. On GitLab.com, approximately 10% of users perform a Git write operation every month. For licensed self-managed instances, this number should be much higher.
• AMAU is being tracked per repository type, and which means that we are under counting slightly by using project repositories at the moment gitlab#234027
• Improvement - work towards providing users more insights while navigating their code with Code Intelligence
• Improvement - work towards providing more control over managing repository size

Implementation Status: Complete

• Enabled on GitLab.com
• Released for self-managed in GitLab 13.3

Create:Editor - GMAU - MAU Conducting Git Write Operations from web editors

A rolling count of the number of unique users who performed a Git write operation to any respository using the Web IDE, Web Editor or Snippet within a stage in the last 28 days

SaaS/SM: Both

Target: 120K for All

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU

Metric Name: action_monthly_active_users_ide_edit

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Unknown

• Data collection is beginning, but it's too early to draw any insights as instances continue to upgrade and report data.

Implementation Status: Complete

• Instrumentation of the Web IDE is in place for both self-managed and SaaS.
• Instrumentation of Static Site Editor and Wiki to roll up into GMAU scheduled for 13.7
• New target GMAU to be set once we get cumulative, de-duped data from new categories

Create:Ecosystem - GMAU - MAU of any integration

Monthly Unique Users of any GitLab integration.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/225678
• https://gitlab.com/gitlab-data/analytics/-/issues/5850

Metric Name: TBD

Health: Unknown

• No target defined
• Improvement - Adding Group and Instance integration will allow larger sets of projects to be integrated easily
• Improvement - Improvements and new functionality in the Jira integration should increase adoption, expanding on the success of our most popular integration

Implementation Status: Instrumentation

• Awaiting implementation of data in Usage Ping
• Awaiting implementation of Sisense dashboard (blocked by Usage Ping implementation)

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Number of unique users utilizing a Geo secondary. This adoption metric helps us understand whether end users are actually seeing value in, and are using, geo secondaries.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Metric Name: geo_nodes

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Unknown

• No target defined yet; we can now track active Geo authentications on secondaries.
• Insight - The number of Geo nodes deployed has continued to grow over second half of 2020. We know that Geo has low penetration as a percentage of total deployments, but skews heavily toward the large enterprise with a 25% percentage of Premium+ user share.
• Improvement - Working towards tracking Git traffic on secondaries and replication events for other data via Geo Usage Ping Epic
• Improvement - We are working to add support for replicating all data types, so that Geo is on a solid foundation for both DR and Geo Replication.

Implementation Status: Instrumentation

• Geo is not available on GitLab.com today, so cannot use Snowplow or the .com database.
• First iteration merged in 13.6 - active Geo authentications on secondaries. Dashboard will be available in 13.7.

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

The number of unique active users and unique paid users interacting with either Basic Search or Advanced Search per month.

SaaS/SM: Both

Target: 10% month over month (SaaS and self-managed combined)

Is Primary PI: true

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Heavy growth is expected for Self-managed as we see customers ramp up migrating to verson 13.4+
• Growth is expected in SaaS as we have been adding new customers and features
• 100 new namespaces to the SaaS index every 7 days. We hit 10,000 on 2020-11-11
• We have a net new document count of 40,000 per hour
• Additional details regarding growth indications can be found in these slides.

Implementation Status: Complete

• Some data is not showing for SaaS Paid GMAU due to a label change. SaaS GMAU started in September.
• Self-managed data was implemented in 13.4.

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

The sum of all monthly active users on gitlab.com that roll up to paid accounts in a 28 day rolling period.

SaaS/SM: SaaS

Target: 111750

Is Primary PI: true

Metric Name: COUNT_EVENTS_ACTIVE_USERS_LAST_28_DAYS_BY_PLAN_WAS_PAID

Chart (Sisense↗)

Health: Okay

• On track for Q4 target
• Insight - Continuing to grow at a similar rate as last month (~7%). Growth this month was driven by an increase in the number of Gold monthly active users (+17% MoM)
• Improvement - Focusing on improving overall availability of GitLab.com by continuing work to achieve auto-rollbacks, iterating on our Disaster Recovery plan by testing failover using Geo, and increasing the reliability of Gitaly infrastructure via cgroups.

Implementation Status: Complete

• Instrumentation complete.

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: Our FY21 Q4 target is to see 10% growth month over month. We are tracking towards a goal of 680,000 unique users triggering pipelines by end of FY21 (Jan 31, 2021) as combined total for SaaS and Reported Self-Managed.

Is Primary PI: true

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - While growth between September and October (at 8.5%) was less than the 10% monthly target, we have exceeded our previous target of 500k unique users triggering pipelines by end of FY21; 511k is current count based on SaaS and Recorded Self-Managed data. We have adjusted our target to 1 million unique users triggering pipelines by end of FY21.
• Insight - Growth of unique SaaS user triggering pipelines was less than 4% between Sep and Oct, which is less than half the growth of Recorded Self-Managed users (at over 10%). This needs more analysis and research to understand factors for slower SaaS growth.
• Insight - Based on Q3 growth (SaaS and Recorded Self-Managed combined), the projected growth in Q4 is 9% per month.
• Improvement - Users engaging with pipelines but not necessarily triggering them is also important and to provide pipeline analytics for these users. In progress is reviewing research and insights for the key metrics for analyzing pipelines.

Implementation Status: Complete

• With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new .gitlab-ci.yml files created.
• We are also working to instrument other performance indicators for Verify:CI
• Updating dashboard to have both SM / SaaS and Paid / All breakouts

Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file

Count the number of users that are pushing MRs to gitlab-ci.yml files. Since we'd like to count the on-going usage of our users with pipelines.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: Average time to first green pipeline - PI The average amount of time it takes to go from running your first pipeline to your first green one.

Secondary Chart (Sisense↗)

Health: Unknown

• Not instrumented

Implementation Status: Implementation

• Waiting on instrumentation scheduled
• Evaluated for FY21-Q4 OKRs but not part of FY21-Q3 KRs
• Improvments to drive this metric, we are working on building an improved authoring experience starting from creating an MVC for a pipeline editor and creating an versatile empty template

Verify:Testing - Paid GMAU - Count of active paid testing feature users

A rolling count of unique users who have interacted with a paid testing feature in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: NA

Chart (Sisense↗)

Health: Problem

• Insights - Visits to the Group Code Coverage page continue to improve but downloads stayed flat. The additional features to the page were delayed beyond 13.5 which we believe is the contributing factor to this.
• Improvements - Additional enhancements of the feature are going live in 13.6 that we expect to increase unique user engagement with the page/feature and we will add a tracking metric (clicks to projects by unique users).
• Insights - Views of the Full Code coverage report have recovered but stayed about flat at previous levels (and are raw usage not unique userIDs).
• Improvements - We have prioritized the implementation tracking unique usersIDs in the 13.7 milestone.

Implementation Status: Implementation

• Waiting on SM instrumentation
• Improvements - This chart is currently counting snowplow sessions instead of users.
• Improvements - Instrumentation of unique user tracking for all existing paid features on track to complete in 13.8.
• The target is missing in the chart.

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: packages

Health: Unknown

• Improvements - In 13.6, we intend to start measuring GMAU/SMAU for the Package stage via usage ping.
• Improvements - Due to scalability and performance concerns, we have to revisit how we collect and store this data.

Implementation Status: In Progress

• We have an issue open to measure this data via the Usage ping. In milestone 13.6 we learned that we needed to modify the MR to use Redis. As a result this work may slip into 13.7.

Release, Release:Progressive Delivery, Release:Release Management - SMAU, GMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: Our FY21 Q4 target will be 30% growth, sustaining at least 10% growth MoM, given that retention is at 89%.

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/640196/New-WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU
• https://app.periscopedata.com/app/gitlab/661492/Temp---Eli-Release-Stage-Examples?widget=8578558&udv=0

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - Our FY21 Q4 is 30% and this month, we have grown in unique users by 9%, a little lower than our 10% target and due to the seasonality of release creation we may see a larger increase at the end of November.
• Improvements - Over 10K more users are triggering deployments in self-managed instances, which is very encouraging given that retention is not 100%. We continue to see a 2.5K increase from SaaS this month.
• Improvements - Release Management has recently shipped the addition of assets and binaries to the Releases object, which is necessary to successfully distribute artifacts generated from releases from GitLab, the work in gitlab#17838 will futher support greater release automation reaching the CI audience and introducing them to Release Mangement capabilities. We are also beginning to see greater interest in GitLab Native Runbooks, which will positively impact enterprise release capabilities. On the Progressive Delivery front, we expect to positively impact top of funnel users with the introduction of feature flags to core in [gitlab#212318](https://gitlab.com/gitlab-org/gitlab/-/issues/212318. The ability for users to deploy easier with GitLab, will be futhered by the ECS templates make available in 13.5 with gitlab#201742](https://gitlab.com/gitlab-org/gitlab/-/issues/201742).

Implementation Status: Complete

• We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
• We need to Expand Revenue Attribution to Release SMAU
• Improvements - Counter was instrumented for 28-days
• Improvements - User and unique count of deployments are instrumented
• Improvements - SaaS and self-managed data are both available on a single chart
• Updating dashboard to have Paid / All breakouts
• Target is missing / misaligned in the charts.

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

SaaS/SM: Both

Target: 16000

Is Primary PI: true

Metric Name: project_clusters_enabled, group_clusters_enabled, instance_clusters_enabled

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insight - What is our new target?
• Improvement - We are working towards moving cluster creation over to Terraform in order to provide customizable, IaC based cluster provisioning. This requires a production ready GitLab Managed Terraform State.

Implementation Status: Complete

• Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
• Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
• Improvement - The GitLab Kubernetes Agent has been released, and it's first usage pings are coming in. See below.
• Updating dashboard to have both SM / SaaS and Paid / All breakouts
• The Data Team strongly advises against estimating this value given that it is already an estimate. An estimate of an estimate yields very inaccurate results.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interacted with alerts or incidents. _This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority._

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 12,000 Estimated total SMAU

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Insights - Growth between September and October 2020 has surpassed the target of being 10% greater than the previous month. We've set a new target inorporating estimated SMAU uplift for self-managed. The target of 12,000 users is based on 100% retention and 25% growth. We are still seeing the results of new explosive growth for incident management. As the we get closer to the complete maturity for the category, we expect this value to continue to grow at a steady rate.
• Improvements - To fuel this growth, the Monitor Group is working to mature Incident management to complete by making it easier to integrate alert sources into GitLab and adding on-call schedule management. These features create an end-to-end incident management workflow that can compete in market.

Implementation Status: Complete

• Instrumentation complete - issue.

Secure:Static Analysis - GMAU - Users running Static Analysis jobs

The highest of the number of unique users who have run SAST or Secret Detection jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10%

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_sast_jobs, user_secret_detection_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• On track

Implementation Status: Complete

• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (October):

• Tracking correlation between in app clicks and upgrades is extremely difficult with existing product analytics system. Will have to work closely with Telemetry and Growth teams to successfully track SAST Core usage upgrades
• We continue to see only positive impact from moving SAST and Secret Detection down to core. It is effectively creating an upgrade path into ultimate from lower plan tieres, which will be formalized and tracked with the new Improved MR experience for non-ultimate users
• Delivered a number of major features in 13.6 including, MVC of SAST & Secret Detection Custom Rulesets (our most requested feature). A major rewrite of NodeJS SAST scanner to add 100+ new rules which will increase coverage and improve accuracy. We added highly requested iOS & Android SAST scanning. All of these features should help to drive paid adoption and extensibility of SAST and Secret Detection. We expect this will take many months to see full effect as these features are related to enterprise sales cycles.
• We have successfully started tracking SAST Configuration UI usage which will inform prioritization of Configuration UI for Secret Detection.

Focus This Month (November):

• Now that SAST & Secret Detection is availible to all plan types, creating a better experience for non-Ultimate users to help them discover value and become interested in upgrading. Improved MR experience & Improved Configuration Experience.
• Pivot focus to some enterprise configuration issues related to monorepos and advanced configurations now that we have Static Analysis growing organically.

Secure:Dynamic Analysis - GMAU - Users running DAST

Number of unique users who have run one or more DAST jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10% per-month

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_dast_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• Steady monthly increase over the last three months (according to PM created charts)

Implementation Status: Complete

• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (October):

• Charts from the data team do not match usage charts created by PM - investigation is underway to understand the difference
• Customers have stopped using DAST in the past because of the huge number of vulnerabilities produced in the dashboard
• Feedback from the effort to aggregate vulnerabilities should result in an uptick in DAST usage
• More profile options need to be introduced for on-demand DAST scans to have broad usage

Focus This Month (November):

• Aggregating noisy vulnerabilities to remove objections to using DAST because it overwhelms the Dashboard (https://gitlab.com/gitlab-org/gitlab/-/issues/254043)
• Add more options to Site profile (https://gitlab.com/groups/gitlab-org/-/epics/3771)
• DAST Site validation (https://gitlab.com/groups/gitlab-org/-/epics/4871)

Secure:Composition Analysis - GMAU - Users running any SCA scanners

The highest of the number of unique users who have run one or more Dependency Scanning jobs, or number of unique users who have run one or more License Scanning jobs.

SaaS/SM: Both

Target: 2% higher than current month (SaaS and self-managed combined) for All and Paid

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_license_management_jobs, user_dependency_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• On track

Implementation Status: Complete

• Based on discussion with Anoop, who will be documenting all MVC shared understandings, this MVC is as expected and will be shared as a trend and not as ACTUAL numbers and there will be a single source of truth to maintain for the known current accepted data limitations and assumptions.

Lessons Learned (October):

• All my work on my dashboard will be replaced by new work by the data team. See this dashboard.
• Concerned over a drop in both scans and users within the Self-hosted space but can't find a particular source.

Focus This Month (November):

• Continue work on auto remediation as part of Dependency Scanning to complete
• Continue to update the DBT and work to improve the data teams documentation, as well as design and them implement a new flattening algorithm.
• When there is time i would like to review in depth with engineering where in the cycle we trigger the count and update known caveats if needed.
• When there is time I would like to review with data or engineering all transformations that occur when ping is processed to send.
• When there is time I would like to review with data all transformations that occur after ping is sent and review in case we want to make adjustments.

Secure:Fuzz Testing - GMAU - Users running fuzz testing

The highest of the number of unique users who have run one or more Fuzz Testing jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10% per-month

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_coverage_fuzzing_jobs on self-managed, (options like '%gitlab-cov-fuzz%' OR options like '%FUZZAPI_REPORT%') on .com

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• We have a good growth rate and need more users to use fuzz testing.

Implementation Status: Definition

• We now have reporting on .com for both coverage and API fuzz testing.
• Self-managed likely to capture no results due to being keyed off of job name rather than job type. Tracking work on improving this in an issue. We can see the number of jobs, just not users.
• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (October):

• We learned how to use a new source for reporting .com usage until we can report on job artifacts. This data appears accurate so may not need to move off it for .com.
• We learned about changes needed to usage ping. Since users can set their own job names, we can't look just at job names.
• We observed .com Snowplow working correctly, as we had several users reporting data.
• Self-managed likely to capture no results due to being keyed off of job name rather than job type. Tracking work on improving this in an issue.

Focus This Month (November):

• API fuzz testing foundational work to use Security Dashboard (https://gitlab.com/gitlab-org/gitlab/-/issues/224726)
• Corpus management for coverage-guided fuzz testing (https://gitlab.com/groups/gitlab-org/-/epics/2915)
• Crash reproduction binaries for coverage-guided fuzz testing (https://gitlab.com/gitlab-org/gitlab/-/issues/233937)
• API Fuzzing of OpenAPI v3 specifications (https://gitlab.com/gitlab-org/gitlab/-/issues/228652)

Secure:Threat Insights - GMAU - Users interacting with Secure UI

Number of unique sessions viewing a security dashboard, pipeline security report, or expanding MR security report. Same as Paid until OSS projects can be separated.

SaaS/SM: SaaS

Target: Progressively increasing month-over-month, >2% per-month

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: n/a

Chart (Sisense↗)

Health: Attention

• Data numbers and trends may be misrepresented (see below)

Implementation Status: Definition

• Recent deep-dive into data indicates page view usage queries may not be accurate
• GitLab employee usage may be greatly overcounted for certain feature pages
• May require new data instrumentation to correct
• Collection is complete for .com only
• Snowplow data is session only, not unique users
• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.
• Large Snowplow data loss event in September and October obscure true performance
• Collection is complete for .com only
• Snowplow data is session only, not unique users
• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.
• Large Snowplow data loss event in September and October obscure true performance
• Collection is complete for .com only
• Snowplow data is session only, not unique users
• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.
• Cannot separate out GitLab employee usage from customers

Lessons Learned (October):

• Page URL pattern matching may be insufficiently deterministic to support accurate GMAU
• Data integrity problems make it impossible to make accurate performance and impact assessments

Focus This Month (November):

• Continue executing on roadmap based on input from other sensing mechanisms
• Conduct CMS evaluation for Viable, solution validation for future design vision
• MR security widget UX improvement

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

The number of users who have run a container scanning job or interacted with the Protect UI in the last 28 days of the month.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >2%

Is Primary PI: true

Metric Name: user_container_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• On track
• Decline in chart from September to October is likely due to lost data. Other charts indicate growth continued as expected at ~2% month over month

Implementation Status: Complete

• Limitations in data (see SCA limitations above)
• Data is not available yet for the Security Orchestration category as it has not yet reached Minimal maturity

Lessons Learned (October):

• TBD

Focus This Month (November):

• Allow container security scans to be run against containers in production environments (https://gitlab.com/groups/gitlab-org/-/epics/3410)
• Project-level DAST Scan Schedule Policies (https://gitlab.com/groups/gitlab-org/-/epics/4598)

Other Stage PIs Details

Enablement - Section PPI, Stage PPI - Median End User Latency

Median end user latency collected by Real User Monitoring. This metric captures how well we are doing in serving our primary customers, our end users, by measuring their end user experience.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: performanceTiming

Health: Attention

• We know we are slower than our primary competition
• We improved LCP of 5 most used pages by an average of 18% in Q3. We have another OKR for Q4 to continue to improve.

Implementation Status: Dashboard

• Waiting on Data team to prioritize. This is not a standard usage ping based metric, so not able to self-serve.
• No target defined, as metric is not dashboarded yet.

Lessons Learned (October):

• We have made significant improvements to LCP, however TTI/TBT are still significant gaps vs. our primary competition

Other Group PIs Details

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Of the total number of self-managed installs, what percentage are on one of the three most recent versions.

SaaS/SM: Both

Target: 40%

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10278985&udv=1102166
• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10279895&udv=1102166

Metric Name: versions_behind_latest

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• A target of 40% has been set. 40% is the upgrade rate that was reached in early 2019.
• Insights - The upgrade rate increased in from 28% to 29% in October 2020. This is continuing the trajectory of improvement since August, however this is still well below the target of 40%.
• Improvement - Considering switching to supporting 3 versions at a time, and extending support for PG11.

Implementation Status: Complete

• Primary PPI is complete, and target is set.

Lessons Learned (October):

• The steady decline in upgrade rate started in May with the release of 13.0 as the latest release. From discussions with TAMs and Support, this is the primary driver of the decrease in upgrade rate. This is born out in the data, where the vast majority of instances still on 12.9 are running pre-PG11 databases.
• PG11+ adoption continues as share of database versions, currently at 48%.

Enablement:Memory - Group PPI - Memory Consumed

Average memory consumed by all invididual GitLab processes

SaaS/SM: Both

Target: 1.5GB

Is Primary PI: true

Metric Name: topology.nodes[0].node_services

Chart (Sisense↗)

Health: Attention

• We are targeting to get below 2GB of total RAM usage, so we need to get these services further below that target.

Implementation Status: Complete

• Working to better support topology metrics for multi-node instances and Kubernetes
• Working to adjust chart to include how many of each process are started by default to better represent a default configuration.

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Database query Apdex, with 100ms the target and 250ms tolerable. This measures the ratio of queries which complete within the satisfactory time, informing how well the database is scaling and performing.

SaaS/SM: Both

Target: 0.99

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/227305

Metric Name: query_apdex_weekly_average

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• Apdex is currently trending in the wrong direction, although it is so close and only two datapoints, it could be noise.
• Improvement - We are focusing on partitioning the largest tables to improveme the performance and scalability of the database.

Implementation Status: Complete

• The PPI has been instrumented in 13.4, but data is just starting to come back.

Lessons Learned (October):

• Apdex does not seem to trend down based on scale of instance.
• Apdex is notably lower on EE instances than CE, see secondary chart. Current theory is that there are some EE-specific queries that do not run well, investigating further.

Verify:Runner - Other - Total downloads of runner binaries and docker images per month for self-managed

Count of downloads of runner binaries and docker images grouped by platform. We believe that measuring the downloads of runner binaries and Docker images by platform and analyzing the trends over month by month basis will provide insight into whether we are making the right investments in core runner features and capabilities. The assumption is that if self-managed users continue to maintain and upgrade their GitLab Runner build infrastructure, this may suggest continued long-term use of GitLab as their DevOps platform.

Funnel Description: Handbook Link

SaaS/SM: SM

Target: TBD

Is Primary PI: true

Metric Name: NA

Health: Unknown

• Not instrumented - On 2020-11-06, we started experimenting with tracking the daily downloads of the gitlab-runner and gitlab-runner-helper docker images from Docker Hub. The counter mechanism uses the Docker API and a simple function embedded in a Google spreadsheet. To date, and with only a small data set, we have noticed that the average daily downloads of the images are as follows. For gitlab-runner, the average daily downloads are 582,613, and for the gitlab-runner-helper, the average downloads are 115,440.

Implementation Status: Definition

• We need to start collecting the data from the source systems and integrated that data into the data warehouse.
• Waiting on dashboard to be created

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Threat Management Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• Data Team KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
lessons.learned	Optional	indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a Red stepped-line for timeseries target.
  • Directional targets will be used:
    • Above ...
    • Below ...
    • At ...
    • At or above ...
    • At or below ...
  • Optional: Use a single Green bar for the current month (in progress). Preferred for metrics which accumulate through a month but optional due to the performance impact to render the chart.
  • Optional: Use a Black line for rolling average.
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
• Multi-series performance indicators should consider the following guidelines:
  • If series are mutually exclusive, use stacked bars for each series with a monthly time series
  • If series are not mutually exclusive, use grouped bars for each series with a monthly time series
  • Do not graph any targets in the chart.
  • Current month styling guidelines will not apply
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :