Stage and Group Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Stage and Group Performance Indicators

On this page

• SMAU Summary
• GMAU Summary
• Other Stage PIs Summary
• Other Group PIs Summary
• SMAU PI Details
  • Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Optimize Features
  • Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
  • Create - SMAU - Unique users writing to a Git repository
  • Verify:Pipeline Execution - GMAU, SMAU - Unique users triggering pipelines
  • Package:Package - SMAU, GMAU - Count of users publishing packages
  • Release:Release - SMAU - Count of users triggering deployments using GitLab
  • Configure:Configure - SMAU, GMAU - MAU of GitLab Managed Terraform State
  • Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents
  • Sec, Secure - Stage MAU, SMAU - Unique users who have used a Secure scanner
  • Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI
• GMAU PI Details
  • Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Optimize Features
  • Manage:Access - Paid GMAU - MAU Using Paid SAML
  • Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  • Manage:Import - GMAU - MAU importing
  • Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
  • Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
  • Create:Source Code - GMAU - Unique users writing to a project Git repository
  • Create:Editor - GMAU - Unique users committing via any web editor
  • Create:Ecosystem - GMAU - MAU of any integration
  • Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary
  • Enablement:Geo - Paid GMAU - Number of potential Geo users
  • Enablement:Global Search - Paid GMAU - The number of unique paid users per month
  • Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com
  • Verify:Pipeline Execution - GMAU, SMAU - Unique users triggering pipelines
  • Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file
  • Verify:Testing - GMAU - Count of active testing users
  • Package:Package - SMAU, GMAU - Count of users publishing packages
  • Release:Release - SMAU - Count of users triggering deployments using GitLab
  • Configure:Configure - SMAU, GMAU - MAU of GitLab Managed Terraform State
  • Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents
  • Secure:Static Analysis - GMAU - Users running Static Analysis jobs
  • Secure:Dynamic Analysis - GMAU - Users running a Dynamic Analysis scan
  • Secure:Composition Analysis - GMAU - Users running any SCA scanners
  • Secure:Threat Insights - GMAU - Users interacting with Secure UI
  • Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI
• Other Stage PIs Details
  • Enablement - Section PPI, Stage PPI - Median End User Page Load Time
• Other Group PIs Details
  • Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab
  • Enablement:Memory - Group PPI - Memory Consumed
  • Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)
  • Verify:Runner - Other - Total downloads of runner binaries and docker images per month
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

SMAU Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Optimize Features	Okay	Both	Complete
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Create	SMAU	Unique users writing to a Git repository	Okay	Both	Complete
Verify:Pipeline Execution	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Package:Package	SMAU, GMAU	Count of users publishing packages	Okay	Both	Complete
Release:Release	SMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	MAU of GitLab Managed Terraform State	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Problem	Both	Complete
Sec, Secure	Stage MAU, SMAU	Unique users who have used a Secure scanner	Okay	Both	Complete
Protect:Container Security	SMAU, GMAU	Users running Container Scanning or interacting with Protect UI	Okay	Both	Complete

GMAU Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Optimize Features	Okay	Both	Complete
Manage:Access	Paid GMAU	MAU Using Paid SAML	Okay	SaaS	Complete
Manage:Compliance	Paid GMAU	MAU Viewing Compliance Features	Attention	Both	Complete
Manage:Import	GMAU	MAU importing	Okay	Both	Complete
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Plan:Product Planning	Paid GMAU	MAU creating Epics each Month	Okay	Both	Complete
Create:Source Code	GMAU	Unique users writing to a project Git repository	Okay	Both	Complete
Create:Editor	GMAU	Unique users committing via any web editor	Attention	Both	Complete
Create:Ecosystem	GMAU	MAU of any integration	Unknown	Both	Instrumentation
Enablement:Geo	Paid GMAU	Number of unique users utilizing a Geo secondary	Attention	Both	Instrumentation
Enablement:Geo	Paid GMAU	Number of potential Geo users	Attention	Both	Instrumentation
Enablement:Global Search	Paid GMAU	The number of unique paid users per month	Okay	Both	Complete
Enablement:Infrastructure	Paid GMAU	Number of unique paid users that perform an action on GitLab.com	Okay	SaaS	Complete
Verify:Pipeline Execution	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Verify:Pipeline Authoring	GMAU	Number of unique users interacting with gitlab-ci.yml file	Okay	Both	Implementation
Verify:Testing	GMAU	Count of active testing users	Attention	Both	Definition
Package:Package	SMAU, GMAU	Count of users publishing packages	Okay	Both	Complete
Release:Release	SMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	MAU of GitLab Managed Terraform State	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Problem	Both	Complete
Secure:Static Analysis	GMAU	Users running Static Analysis jobs	Okay	Both	Complete
Secure:Dynamic Analysis	GMAU	Users running a Dynamic Analysis scan	Okay	Both	Complete
Secure:Composition Analysis	GMAU	Users running any SCA scanners	Okay	Both	Complete
Secure:Threat Insights	GMAU	Users interacting with Secure UI	Attention	SaaS	Definition
Protect:Container Security	SMAU, GMAU	Users running Container Scanning or interacting with Protect UI	Okay	Both	Complete

Other Stage PIs Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement	Section PPI, Stage PPI	Median End User Page Load Time	Attention	Both	Complete

Other Group PIs Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement:Distribution	Group PPI	Percentage of installations on the 3 most recent versions of GitLab	Attention	Both	Complete
Enablement:Memory	Group PPI	Memory Consumed	Attention	Both	Complete
Enablement:Database	Group PPI	Database Query Apdex (100ms target, 250ms tolerable)	Attention	Both	Complete
Verify:Runner	Other	Total downloads of runner binaries and docker images per month	Unknown	SM	Definition

SMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Optimize Features

The number of unique users viewing any Optimize page.

SaaS/SM: Both

Target: 116050

Is Primary PI: true

Health: Okay

• Insights - DevOps Adoption was made available to SaaS users in 13.11. Previously it was only available at the Instance level for self-managed users. We received the first set of usage data in May, showing that 55 SaaS users tried DevOps Adoption within the first month of it being released. This feature is restricted to Ultimate users.
• In 13.11 we announced a series of improvements to group-level Value Stream Analytics and saw a 14% increase in adoption in May with the biggest increase being in the Ultimate tier.
• Improvements - Over the next 2 releases, we will be adding security scanning features to the DevOps Adoption report, making the report more useful for Ultimate customers.
• SaaS data is not available for the paid SMAU chart due to limitations of the redis counter that is used.

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Implementation Status: Complete

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 557,290 for All, 157,571 for Paid

Is Primary PI: true

Health: Okay

• Given the variables of missing SaaS data, customers purging their usage ping data, and the majority of self-managed customers not sending us usage ping data at all, it is not possible to draw any meaningul health-related conclusions from our data at this point in time.
• Driver - For Paid, continue progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible, and continuing to make incremental progress on iterations.
• Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
• Improvement - For All, continue making progress on quality of life improvements for existing features.

Metric Name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://about.gitlab.com/direction/plan/issue_tracking/
• https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)

Implementation Status: Complete

Lessons Learned (June):

• As of the time this update was written, 25% of SaaS usage ping data is missing from the last 8 weeks. > 60% of our paying self-managed customers do not have usage ping enabled. This makes an accurate analysis of overall trends impossible. Keep this in mind as you read on.
• Only 47% of instances are currently on a version with the new Project Management xMAU redis counters.
• May saw 1.9 net new Paid GMAU for every 1 churned Paid GMAU, which is a 19% improvement over May. (Self-Managed Only)
• From a macro perspective, it is worth highlighting that the total count of self-managed customers with usage ping enabled has remained relatively flat over the previous four reporting periods. This is strongly correlated with Dev Paid CMAU and Plan Paid MAU.
• After relatively strong MoM growth this year, extrapolated revenue per Paid MAU decreased by 7% from May to June (see cell V189 - internal only). This is by no means scientific, but it can be traced back to a strong decline in Self-Managed Ultimate Paid GMAU, which experienced a 19% decrease from May to June. It is unclear if this trend is local to Plan or could be related to larger Ultimate customers disabiling Using Ping and purging all historical data.
• Recorded xMAU has remained flat the past three reporting periods. The decline in All seems to be attributed to a drop in Estimated xMAU. It seems as though there may be room to improve how we estimate xMAU. For example, Estimated GMAU dropped by 3% from May to June while Recorded Self-Managed GMAU increased by only .5%, all while the overall percent of customers with usage ping enabled only increased by .6%.
• Total blended Plan MAU growth by tier from January to June - Core (18.3%), Trial (49.6%), EE Free (51.9%), Starter (-14.9%), Premium (72.2%), Ultimate (19.9%)

Focus This Month (July):

• Merge architectural blueprint for consolidating Epics, Requirements, and Issues into a single, parentable "work item" object.
• Finalize and validate the mobile-first MVC for the new issue UI/UX to set the foundation for long term improvements in SuS, reducing technical debt, improving performance, and making issues update in real-time. We will begin shipping iterations on the new issue view behind a feature flag starting in 14.2.
• Merge architectural blueprint for consolidating Groups and Projects. Complete a PoC for enabling Issues at the Group level with the proposed implementation path.
• Support the ongoing research Pricing and Packaging is conducting around the non-developer use cases.
• Collaborate closely with a large customer to build out a robust PoC to model their full SDLC within GitLab. The long term goal is enabling them to migrate away from Jira Align and Jira, which will drmatically improve the quality of life for the engineers. We will leverage the learnings to inform our product roadmap and create "blueprints" for enabling other larger organizations to migrate their planning needs to GitLab.

Create - SMAU - Unique users writing to a Git repository

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.

Is Primary PI: true

Health: Okay

• Insight - SMAU growth for SaaS appears to have stalled over the last 3 months while Self-managed numbers continue to steadily rise.
• Insight - On the median GitLab instance (self-managed), 33% of users perform a Git write operation each month (source). On the median Paid GitLab instance (self-managed), 50% of users perform a Git write operation each month (source).
• Improvement - Overall the Create stage is focused on maintaining growth via support for large MRs, implementing the Content Editor in Wikis, incremental back-ups for repositories, and preparing gitlab-ssh to replace Open SSH.
• Improvement - 3 of the 4 Groups in Create are focused on initiatives aimed at scaling GitLab for large customers. We expect it to contribute to retention of enterprise customers and we expect some growth in the medium term as we meet the performance expectations of some large prospects. We do not expect immediate growth from this work.

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard

Implementation Status: Complete

• Create SMAU metrics are being deduplicated.

Verify:Pipeline Execution - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

SaaS/SM: Both

Target: Our projected unique users triggering pipelines by end of FY22-Q2 is 1.32M, so we are going to target just above that at 1.35M unique users triggering pipelines.

Is Primary PI: true

Health: Okay

• On track and healthy, we expect to be progressing to FY22-Q3 target as expected.

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Insight - We can see in June a general increase in Verify MAU from 1.18M to 1.20M, which is a less than 2% percent overall increase of users. This growth can be seen coming in users on both SaaS (+20K) and self-managed (+10K). Although, when looking at the Paid Segment, we can see a slight decrease from the 400K MAU in May to 399K MAU, which could be attributed to a data tracking error. We have seen a 5% increase in recorded usage of Paid SaaS of users triggering pipelines on GitLab.com in June, which is consistent with the trend of the past 3 months, and we will continue to monitor this to see how enforcing public projects on new namespaces will impact this usage.
• Insight - The Verify Funnel and Drop off and overtime Funnel view. shows in June, a sustained gap between SCM MAU and Verify MAU despite the slight increase in MAU in Verify. The dropoff remains 55%. Pipeline Authoring MAU continues to increase slightly toward their ~250K Q3 target MAU, sustaining the 86% drop off from last month between Verify MAU. Although, we have not seen an improvement in Testing MAU, which continues to decline. We have restated the focus of the testing group and look to add a new PI focus in the coming milestones to this funnel.

Lessons Learned (June):

• Insight - In June 2021, we are starting to see a retention rate decrease from 89% to 82% for self-managed users, which requires more investigation to see if this is a data integrity reporting concern or actual account attrition.
• Insight - In June 2021, to elaborate on the retention rate concern in comparison to May expanding instances have contracted slightly (by -9%), while new instances are slightly decreasing (by -18%) and shrinking instances are improving significantly (by 24%) overall, there is not much new usage although we are not losing any existing users either. This seems to be slightly contraian to the 82% retention rate indicated in June.

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 54,000 unique users publishing packages across SaaS and self-managed combined. This is an increase from the original target of 50,000 based on solid growth in May and in weekly user growth for June.

Is Primary PI: true

Health: Okay

• We hit the May target of 46k active users of the Package Registry.

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Implementation Status: Complete

• Generic package usage is not collecting data for usage ping. GitLab-#331455 is scheduled for 14.1 and will address this.

Lessons Learned (June):

• Insights - We hit the May target of 46k users per month!
• Insights - Growth was split evenly between SaaS and Self-Managed.
• Insights - From a funnel perspective, we saw significant growth in the packages pulled using a deploy token or by a Guest. Both are signs that the Package Registry is being integrated into our customers production workflows. The number of packages pulled using a Deploy token grew from 3mm to 4mm April to May 2021. The number of packages pulled by Guest users grew from 2mm to 3mm during the same time period.
• Improvements - In 14.1, we are investing in the container registry cleanup policies which will not have an impact on this metric.
• Improvements - Based on the above, in 14.2 we plan to resolve some known issues with deploy tokens. GitLab-#285495 will resolve an issue in which group deploy tokens don't work with sub-groups. GitLab-#332028 will resolve an issue in which deploy tokens do not work when a project or group has been updated from private to public or private to internal. We expect this will help drive retention.

Release:Release - SMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 360,000 for unique users triggering deployments

Is Primary PI: true

Health: Okay

• 10% growth between May and April

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard

Implementation Status: Complete

Lessons Learned (June):

• Improvements - Deployment frequency is available at the API, project, and group level. Lead time for Merge Request is available at the API, and Project level and will be available at the Group level after 14.0.
• Talking to enterprise customers, it is becoming more clear that GitLab doesn't have the release orchestration features needed to coordinate complex releases.
• Customers are also interested in better management of environments

Configure:Configure - SMAU, GMAU - MAU of GitLab Managed Terraform State

Monthly active users of GitLab Managed Terraform State

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 7000 monthly active users of GitLab Managed Terraform State

Is Primary PI: true

Health: Okay

• Healthy growth continues

Metric Name: redis_hll_counters.terraform.p_terraform_state_api_unique_users_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/511813/Configure-team-business-metrics
• https://app.periscopedata.com/app/gitlab/677628/Infrastructure-as-Code

Implementation Status: Complete

• We have MAU based metrics.

Lessons Learned (June):

• Insight - Self managed usage is very healthy. We plan an up-sell feature to be developed as one of the first issues for our new frontend engineer. Nota bene, we don't have a design for it to drive adoption further on SaaS.
• Insight - We saw a jump of MAU in all tiers in May. As self-managed instances lag behind SaaS by 3 monthy usually, these might be attributed to the release of the SMAU metric on 22th January.
• Insight - Given the MoM growth, we will likely miss our quarterly target.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interact with alerts or incidents. (This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority.)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 18,000 estimated total unique users that interact with alerts and incidents.

Is Primary PI: true

Health: Problem

• 2 months of declining users in SaaS

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (June):

• The Monitor SMAU is still largely driven by people creating incidents as an issue types.
• The infrastructure team is looking to adopt Incendent types in their incident response process
• To facilitate the dogfooding, we will prioritize some work for the infra team
• There are some emerging ideas to drive overall usage, such as Alerging on CI/CD pipeline event or deeper integration with Slack. The new Monitor PM will determine path forward.

Sec, Secure - Stage MAU, SMAU - Unique users who have used a Secure scanner

The number of unique users who have run one or more Secure scanners.

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Health: Okay

• currently instrumented to meet OKRs (see instrumentation, Lessons Learned for more details)

Metric Name: user_unique_users_all_secure_scanners

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Can't currently differentiate between free-to-OSS/education Gold/Ultimate users and paid Gold/Ultimate users
• Threat Insights data is not yet included in this count until we support user-level metrics in Snowplow for self-managed

Lessons Learned (June):

• Current Self-Managed Uplift is leading to massive underreporting as it doesn't fully account for non-reporting instances (offline environments, security-conscious enterprises)

Focus This Month (July):

• Overall growth was reported as flat. This is in opposition in our Ultimate growth during this period. As such, this will need to be investigated to confirm if there was data loss.
• Identify a way to denote underreporting on Sec-related *MAU charts to remove confusion / clarify metrics interpretation
• Work with the Data Analytics team on improving Self-Managed Uplift for Sec

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

The number of users who have run a container scanning job or interacted with the Protect UI in the last 28 days of the month.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >2%

Is Primary PI: true

Health: Okay

• Usage growth dropped slightly from its April high. We may continue to see a small, temporary decline in usage as customers adapt their pipelines to work with the new Trivy scanner. Long-term, this is expected to provide a significant boost to usage.
• Although MAU data of the Alert Dashboard is not yet included in the broader numbers for these charts, we are seeing positive early signs from the data we do have that usage is increasing sharply.

Metric Name: user_container_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Limitations in data (see SCA limitations above)
• Although we have released the Alert Dashboard, we do not yet have a good way of tracking monthly active users for this area. Instrumentation is expected to be complex (1-2 months of one developer's time) and usage is expected to remain low until additional alert times are added to the dashboard. Currently instrumenting these metrics is a low priority.

Lessons Learned (June):

• The switch from Clair to Trivy has allowed us to close 20+ outstanding customer issues and bugs. Adoption is expected to increase significantly after the 14.0 release.

Focus This Month (July):

• Replacing Clair with Trivy (https://gitlab.com/groups/gitlab-org/-/epics/5398) will be the default beginning in the 14.0 release and will set the foundation for us to be able to scan containers in production environments
• Allow Users to Edit yaml-mode Scan Execution Policies in the Policy UI (https://gitlab.com/groups/gitlab-org/-/epics/5362) is being worked on for a July release. This change will allow us to default the policy feature flag to on.

GMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Optimize Features

The number of unique users viewing any Optimize page.

SaaS/SM: Both

Target: 116050

Is Primary PI: true

Health: Okay

• Insights - DevOps Adoption was made available to SaaS users in 13.11. Previously it was only available at the Instance level for self-managed users. We received the first set of usage data in May, showing that 55 SaaS users tried DevOps Adoption within the first month of it being released. This feature is restricted to Ultimate users.
• In 13.11 we announced a series of improvements to group-level Value Stream Analytics and saw a 14% increase in adoption in May with the biggest increase being in the Ultimate tier.
• Improvements - Over the next 2 releases, we will be adding security scanning features to the DevOps Adoption report, making the report more useful for Ultimate customers.
• SaaS data is not available for the paid SMAU chart due to limitations of the redis counter that is used.

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Implementation Status: Complete

Manage:Access - Paid GMAU - MAU Using Paid SAML

The number of unique users logging into GitLab using SAML SSO for Groups.

SaaS/SM: SaaS

Target: Q3FY22 goal 8700

Is Primary PI: true

Health: Okay

• Insights - An increase of ~10% in MAU from last month. A huge increase in self-manage SMAU of 87% from the previous month!
• Insights - This month also showed a record high of new group memebers on a SMAL group - over 9k new group members (33% increase from last month)!

Metric Name: user_auth_by_provider (Group_SAML)

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features

The number of unique users viewing Audit Events, Compliance Dashboard, or Credential Inventory.

SaaS/SM: Both

Target: 10% quarter-over-quarter

Is Primary PI: true

Health: Attention

• We are biasing for action and setting a target with the information we have, rather than waiting. We expect this may need to be adjusted in the future. We can use our predictive models to set initial targets in the future.

Metric Name: i_compliance_audit_events_monthly, g_compliance_audit_events_monthly, g_compliance_dashboard_monthly, i_compliance_credential_inventory_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Implementation Status: Complete

• Completed

Lessons Learned (June):

• We saw a 2% decrease in our users this month looking at our "total compliance actions" metric, which combines group and instance-level audit events, compliance dashboard, and credential inventory usage. Looking at individual metrics, this was primarily caused by a fewer users viewing the audit events screen. Instance-level statistics were flat with no change. That said, the number of audit event users is still markedly (86%) than where it was two quarters ago in December, primarily driven by us opening the screen to more users.
• We continue to hear about the needs of compliance-concious users and needing to be able to apply fine-grained controls, as opposed to blanket "everyone gets enforcement" checkboxes. This further underscores the need of bringing group-level controls for our GitLab.com and self-managed customers. It also underscores the importance of focusing on subgroups, beyond just root groups, since the root group on GitLab.com is essentially equivalent to an instance-level setting for those users.
• There is a distinction between acting on "what is in production" and "what were all changes between release X and Y" for users in different compliance environments. Internal stakeholders will care about the former while external stakeholders, such as auditors and regulatory bodies, care about the latter. This give us good insight into the types of experiences we need to build for users.

Focus This Month (July):

• Add additional audit events (1, 2, 3)
• Group-level MR settings
• Continue refinement and validation on webhook-based audit events
• We will investigate using the predictive and regreession capabilities in Sisense to build projected usage models that will be live updated with data as it comes in. This will help us understand if we are having a "slow" or "fast" month in terms of growth compared to what we would expect.
• Assuming we can get predictive models set up in Sisense, we will begin to use those projections to help us inform our planning for the future in terms of maintaining a scalable infrastructure and deciding on the urgency of needing to make archtiectural changes. This will also help inform us on how urgent the need for webhook-based audit events is, which will offload some processing from GitLab, will become in the future.
• We will consider if this metric should be updated to include more parts of the product that Compliance group is focused on, such as MR settings or exteral status checks. This will allow us to better understand which parts of our product users are using, in addition to the areas we already are tracking.
• We will focus on backfilling open engineering roles to allow us to increase our team velocity.

Manage:Import - GMAU - MAU importing

A monthly count of the unique number of users performing an import

SaaS/SM: Both

Target: 88,000 MAU importing projects

Is Primary PI: true

Health: Okay

• Target - Initial target set to 88,000 by the end of the year, which is a 10% increase from the initial user engagement levels in March and April.
• Improvement - Reducing friction for GitLab-to-GitLab migrations - The feature parity with the existing Group Export/Import, which is the first milestone on this path is targeted for 14.2.
• Note - SaaS data is not available for the paid GMAU chart due to limitations of the redis counter used.

Metric Name: usage_activity_by_stage_monthly.manage.unique_users_all_imports

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation has been implemented in 13.9.
• The charts are completed and data is available starting with February 2021.

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 557,290 for All, 157,571 for Paid

Is Primary PI: true

Health: Okay

• Given the variables of missing SaaS data, customers purging their usage ping data, and the majority of self-managed customers not sending us usage ping data at all, it is not possible to draw any meaningul health-related conclusions from our data at this point in time.
• Driver - For Paid, continue progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible, and continuing to make incremental progress on iterations.
• Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
• Improvement - For All, continue making progress on quality of life improvements for existing features.

Metric Name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://about.gitlab.com/direction/plan/issue_tracking/
• https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)

Implementation Status: Complete

Lessons Learned (June):

• As of the time this update was written, 25% of SaaS usage ping data is missing from the last 8 weeks. > 60% of our paying self-managed customers do not have usage ping enabled. This makes an accurate analysis of overall trends impossible. Keep this in mind as you read on.
• Only 47% of instances are currently on a version with the new Project Management xMAU redis counters.
• May saw 1.9 net new Paid GMAU for every 1 churned Paid GMAU, which is a 19% improvement over May. (Self-Managed Only)
• From a macro perspective, it is worth highlighting that the total count of self-managed customers with usage ping enabled has remained relatively flat over the previous four reporting periods. This is strongly correlated with Dev Paid CMAU and Plan Paid MAU.
• After relatively strong MoM growth this year, extrapolated revenue per Paid MAU decreased by 7% from May to June (see cell V189 - internal only). This is by no means scientific, but it can be traced back to a strong decline in Self-Managed Ultimate Paid GMAU, which experienced a 19% decrease from May to June. It is unclear if this trend is local to Plan or could be related to larger Ultimate customers disabiling Using Ping and purging all historical data.
• Recorded xMAU has remained flat the past three reporting periods. The decline in All seems to be attributed to a drop in Estimated xMAU. It seems as though there may be room to improve how we estimate xMAU. For example, Estimated GMAU dropped by 3% from May to June while Recorded Self-Managed GMAU increased by only .5%, all while the overall percent of customers with usage ping enabled only increased by .6%.
• Total blended Plan MAU growth by tier from January to June - Core (18.3%), Trial (49.6%), EE Free (51.9%), Starter (-14.9%), Premium (72.2%), Ultimate (19.9%)

Focus This Month (July):

• Merge architectural blueprint for consolidating Epics, Requirements, and Issues into a single, parentable "work item" object.
• Finalize and validate the mobile-first MVC for the new issue UI/UX to set the foundation for long term improvements in SuS, reducing technical debt, improving performance, and making issues update in real-time. We will begin shipping iterations on the new issue view behind a feature flag starting in 14.2.
• Merge architectural blueprint for consolidating Groups and Projects. Complete a PoC for enabling Issues at the Group level with the proposed implementation path.
• Support the ongoing research Pricing and Packaging is conducting around the non-developer use cases.
• Collaborate closely with a large customer to build out a robust PoC to model their full SDLC within GitLab. The long term goal is enabling them to migrate away from Jira Align and Jira, which will drmatically improve the quality of life for the engineers. We will leverage the learnings to inform our product roadmap and create "blueprints" for enabling other larger organizations to migrate their planning needs to GitLab.

Plan:Product Planning - Paid GMAU - MAU creating Epics each Month

The number of unique users creating Epics each month.

SaaS/SM: Both

Target: 10,301 Combined Unique Users for Self Managed and SaaS

Is Primary PI: true

Health: Okay

• May 2021 - 7126 GMAU has a slight decline of 7.57% since April.
• Insight - All of our epic event actions saw increases, so not sure why there was a slight drop. We'll see if this continues in June.
• Only 17% of instances are currently on a version with the new Product Management xMAU metric counters (15% increase in May).
• Improvement - Epic Boards full feature will ship in 14.0!
• Our next target will be set after 14.0 and epic boards release and verifying our new graphs for GMAU.

Metric Name: epics

Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Implementation Status: Complete

• New Sisense graph WIP GMAU - Users interacting with Epics
• We've added trackers for epic board creation ahead of the release in 14.0 - Epic board tracking issue

Create:Source Code - GMAU - Unique users writing to a project Git repository

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.
Why: We want to ensure all the features in our group provide a great experience that ultimately will allow everyone to contribute more often. A great experience in SCM is a critical starting point for this.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
- Q1FY22 targets - 2.44M GMAU, 590k paid GMAU
- Q2FY22 targets - 2.67M GMAU, 644k paid GMAU
- Q3FY22 targets - 2.92M GMAU, 704k paid GMAU
- Q4FY22 targets - 3.19M GMAU, 769k paid GMAU

Is Primary PI: true

Health: Okay

• Insight - GMAU increased less than 1% from April to May but we are still on track to achieving our Q2FY22 target
• Insight - Paid GMAU increased 3% April to May which is slightly under target.
• Improvement - The team is currently focused on performance as our secondary PI. We do not anticipate this to drive immediate improvement in our primary PI but over time it will contribute to retention and enable us to continue to land bigger and bigger customers.

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0

Implementation Status: Complete

• Enabled on GitLab.com
• Released for self-managed in GitLab 13.3

Create:Editor - GMAU - Unique users committing via any web editor

A rolling count of the number of unique users who performed a Git write operation to any repository using the Web IDE, Web Editor or Snippet within a stage in the last 28 days

SaaS/SM: Both

Target: 4% growth month over month for All

Is Primary PI: true

Health: Attention

• Insight - Relatively flat for the third month, with self-managed somewhat slightly outpacing web editor usage. We'll be shifting focus back on the Web IDE in 14.1 so we hope to see this pick up again in late Q2 - early Q3.
• Insight - A higher percentage of self-managed users are committing write operations from web editors when compared to the total number of MAU writing to a git repo. It's not a dramatic difference (16.6% on self-managed vs 13.5% on SaaS) but investigation into the usage patterns here could uncover areas for faster growth in SaaS usage.
• Improvement - Source Editor is going to be used to render files in the repository view, making it possible to switch immediately into "edit" mode. Expectations are an uptick in commits over the next quarter by reducing friction and improving performance .

Metric Name: action_monthly_active_users_ide_edit

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU

Implementation Status: Complete

Create:Ecosystem - GMAU - MAU of any integration

Monthly Unique Users of any GitLab integration.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Unknown

• No target defined
• Improvement - Adding Group and Instance integration will allow larger sets of projects to be integrated easily
• Improvement - Improvements and new functionality in the Jira integration should increase adoption, expanding on the success of our most popular integration

Metric Name: TBD

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/5181
• https://gitlab.com/gitlab-org/gitlab/-/issues/297401
• https://gitlab.com/gitlab-data/analytics/-/issues/7974

Implementation Status: Instrumentation

• Awaiting implementation of data in Usage Ping
• Improvement - (March 2021) Updates are in progress, GMAU tracking is implemented and data is starting to flow in, will be on the dashboard once updates complete
• Awaiting implementation of Sisense dashboard (blocked by Usage Ping implementation)

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Number of unique users utilizing a Geo secondary. This adoption metric helps us understand whether end users are actually seeing value in, and are using, geo secondaries.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Attention

• We've enabled gathering usage data from Geo secondaries and have added git fetch metrics to the usage data. These metrics are not yet available; we are working on getting them plotted in Sisense. We also intend to add tracking for git push operations on a Geo secondary. This work is currently paused due to capacity constraints.
• GMAU is currently based on logging into the Secondary web interface. Git access is more common and we will begin tracking Git pulls in 13.11 with tracking for Git pushes planned as part of the Geo Usage Ping Epic. We also know that the UX of the secondary web interface is not good and we want to remove it, see Opportunity Canvas(internal). In order to assess the impact of our planned changes, having this (low) GMAU is really important. We've recently completed a PoC to make the secondary UI indistinguishable from the primary. Implementation work is in progress and can be tracked in the secondary mimicry epic.

Metric Name: geo_nodes

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Implementation Status: Instrumentation

• Geo is not available on GitLab.com today, so cannot use Snowplow or the .com database.

Lessons Learned (June):

• The number of users is still very low, likely because the WebUI is insufficient. We are planning to change this, see this opportunity canvas and in-progress implementation work in the secondary mimicry epic.

Enablement:Geo - Paid GMAU - Number of potential Geo users

Sum of all users who are using instances with Geo enabled. All these users are positively impacted due to Geo in case of a failover or restore from backup.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Attention

• For Disaster Recovery we are measuring potential Geo users - the number of active licenses. In an ideal world, no regular user would ever need to rely on Geo because there is no disaster, but if one occurs everyone benefits. Based on the node number distribution ~60% of our customers use Geo mainly for DR. I think this is worth measuring because setting up Geo is always a conscious decision by the customer - it must be configured.
• We are working to add support for replicating all data types, so that Geo is on a solid foundation for both DR and Geo Replication.
• In 13.12 we replicate ~86% of all data types (25 out of 29 in total) and verify ~48%. Shipped Terraform State File verification

Metric Name: geo_nodes

Chart (Sisense↗)

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Implementation Status: Instrumentation

• Geo customers and user data available but we can't distinguish DR use case from replication yet.

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

The number of unique active users and unique paid users interacting with either Basic Search or Advanced Search per month.

SaaS/SM: Both

Target: 10% month over month (SaaS and self-managed combined)

Is Primary PI: true

Health: Okay

• In May, Paid GMAU is up by 9%. SaaS up 10%, Self-Manged up 9%
• There is contiuning growth of more than 10% MoM since Oct 2020
• Advanced Search performance remains stable since scaling up to 12 nodes.
• UI enhancements planned for the next month to address top SUS feedback will help accelerate growth

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Data collected is incomplete for September and October 2020 and January 2021

Lessons Learned (June):

• The Recently Searched/Used Projects and Groups will be added to the dropdowns on the search results page. Our customers have been requesting this feature for over 5 years.

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

The sum of all monthly active users on gitlab.com that roll up to paid accounts in a 28 day rolling period.

SaaS/SM: SaaS

Target: 134000

Is Primary PI: true

Health: Okay

• Paid Monthly Active Users on GitLab.com increased by 4.3% in May to 130K, on track for our Q2 target.
• Net Retention on GitLab.com remains above the 130% target while Gross Retention is below the 90% target due to a dip starting in January 2021. We created a dashbord to investigate further and found that the drop was driven by a Large customer whose spend contracted this year citing uncertainty in their industry due to the pandemic and an SMB customer who decided to downgrade to Premium because they weren't using the Ultimate feature set. We expect Gross Retention to stay in the 85% range through the end of the calendar year since the metric is pulling in last year's spend from these two customers and comparing it to their current spend on a monthly basis. We will continue to monitor retention as a supporting metric for Paid GMAU moving forward.

Metric Name: COUNT_EVENTS_ACTIVE_USERS_LAST_28_DAYS_BY_PLAN_WAS_PAID

Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation complete.

Lessons Learned (June):

• Need to account for seasonal fluctuations into target projections.

Verify:Pipeline Execution - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

SaaS/SM: Both

Target: Our projected unique users triggering pipelines by end of FY22-Q2 is 1.32M, so we are going to target just above that at 1.35M unique users triggering pipelines.

Is Primary PI: true

Health: Okay

• On track and healthy, we expect to be progressing to FY22-Q3 target as expected.

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Insight - We can see in June a general increase in Verify MAU from 1.18M to 1.20M, which is a less than 2% percent overall increase of users. This growth can be seen coming in users on both SaaS (+20K) and self-managed (+10K). Although, when looking at the Paid Segment, we can see a slight decrease from the 400K MAU in May to 399K MAU, which could be attributed to a data tracking error. We have seen a 5% increase in recorded usage of Paid SaaS of users triggering pipelines on GitLab.com in June, which is consistent with the trend of the past 3 months, and we will continue to monitor this to see how enforcing public projects on new namespaces will impact this usage.
• Insight - The Verify Funnel and Drop off and overtime Funnel view. shows in June, a sustained gap between SCM MAU and Verify MAU despite the slight increase in MAU in Verify. The dropoff remains 55%. Pipeline Authoring MAU continues to increase slightly toward their ~250K Q3 target MAU, sustaining the 86% drop off from last month between Verify MAU. Although, we have not seen an improvement in Testing MAU, which continues to decline. We have restated the focus of the testing group and look to add a new PI focus in the coming milestones to this funnel.

Lessons Learned (June):

• Insight - In June 2021, we are starting to see a retention rate decrease from 89% to 82% for self-managed users, which requires more investigation to see if this is a data integrity reporting concern or actual account attrition.
• Insight - In June 2021, to elaborate on the retention rate concern in comparison to May expanding instances have contracted slightly (by -9%), while new instances are slightly decreasing (by -18%) and shrinking instances are improving significantly (by 24%) overall, there is not much new usage although we are not losing any existing users either. This seems to be slightly contraian to the 82% retention rate indicated in June.

Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file

Count the number of users that are pushing MRs to gitlab-ci.yml files. This indicates on-going usage of pipeline authoring by continually adjusting pipeline configuration.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The FY22-Q2 target is 250,000 unique users interacting with the gitlab-ci.yml file on self-managed and SaaS.

Is Primary PI: true

Health: Okay

• Based on last quarter growth and growing number of users that are using the pipeline editor

Metric Name: redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Implementation

• Insight - Numbers keep growing, which aligns with the sentiment we receive from our users regarding the pipeline editor. The editor came out early this year and had several feature improvements. As a result, I expect this number to keep growing over the next month. Last week we had a record number of 13.8K unique users that interact with our editor.

Lessons Learned (June):

• Improvement - We were able to release Branch selector to allow users to work with the pipeline editor from any non-master branch. In addition, our research had shown that templates provide an easy way for users to start working with CI we've added the template support to the pipeline editor and instrument tracking to understand the user behavior around CI templates.

Verify:Testing - GMAU - Count of active testing users

A rolling count of unique users who have interacted with a testing feature over the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Attention

• There is low testing feature adoption of existing CI users and growth is at or below overall stage growth rates.

Metric Name: NA

Chart (Sisense↗)

Implementation Status: Definition

• We are shifting away from PGMAU to GMAU to include our focus on Build Artifacts and top of funnel related work.

Lessons Learned (June):

• We had over 1700 Unique SaaS users interacting and over 1500 SM users interacting with Testing features in June.
• MAU growth of Test Summary Expansions was just under 5% between May and June indicating low awareness of the features.

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 54,000 unique users publishing packages across SaaS and self-managed combined. This is an increase from the original target of 50,000 based on solid growth in May and in weekly user growth for June.

Is Primary PI: true

Health: Okay

• We hit the May target of 46k active users of the Package Registry.

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Implementation Status: Complete

• Generic package usage is not collecting data for usage ping. GitLab-#331455 is scheduled for 14.1 and will address this.

Lessons Learned (June):

• Insights - We hit the May target of 46k users per month!
• Insights - Growth was split evenly between SaaS and Self-Managed.
• Insights - From a funnel perspective, we saw significant growth in the packages pulled using a deploy token or by a Guest. Both are signs that the Package Registry is being integrated into our customers production workflows. The number of packages pulled using a Deploy token grew from 3mm to 4mm April to May 2021. The number of packages pulled by Guest users grew from 2mm to 3mm during the same time period.
• Improvements - In 14.1, we are investing in the container registry cleanup policies which will not have an impact on this metric.
• Improvements - Based on the above, in 14.2 we plan to resolve some known issues with deploy tokens. GitLab-#285495 will resolve an issue in which group deploy tokens don't work with sub-groups. GitLab-#332028 will resolve an issue in which deploy tokens do not work when a project or group has been updated from private to public or private to internal. We expect this will help drive retention.

Release:Release - SMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 360,000 for unique users triggering deployments

Is Primary PI: true

Health: Okay

• 10% growth between May and April

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard

Implementation Status: Complete

Lessons Learned (June):

• Improvements - Deployment frequency is available at the API, project, and group level. Lead time for Merge Request is available at the API, and Project level and will be available at the Group level after 14.0.
• Talking to enterprise customers, it is becoming more clear that GitLab doesn't have the release orchestration features needed to coordinate complex releases.
• Customers are also interested in better management of environments

Configure:Configure - SMAU, GMAU - MAU of GitLab Managed Terraform State

Monthly active users of GitLab Managed Terraform State

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 7000 monthly active users of GitLab Managed Terraform State

Is Primary PI: true

Health: Okay

• Healthy growth continues

Metric Name: redis_hll_counters.terraform.p_terraform_state_api_unique_users_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/511813/Configure-team-business-metrics
• https://app.periscopedata.com/app/gitlab/677628/Infrastructure-as-Code

Implementation Status: Complete

• We have MAU based metrics.

Lessons Learned (June):

• Insight - Self managed usage is very healthy. We plan an up-sell feature to be developed as one of the first issues for our new frontend engineer. Nota bene, we don't have a design for it to drive adoption further on SaaS.
• Insight - We saw a jump of MAU in all tiers in May. As self-managed instances lag behind SaaS by 3 monthy usually, these might be attributed to the release of the SMAU metric on 22th January.
• Insight - Given the MoM growth, we will likely miss our quarterly target.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interact with alerts or incidents. (This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority.)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 18,000 estimated total unique users that interact with alerts and incidents.

Is Primary PI: true

Health: Problem

• 2 months of declining users in SaaS

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (June):

• The Monitor SMAU is still largely driven by people creating incidents as an issue types.
• The infrastructure team is looking to adopt Incendent types in their incident response process
• To facilitate the dogfooding, we will prioritize some work for the infra team
• There are some emerging ideas to drive overall usage, such as Alerging on CI/CD pipeline event or deeper integration with Slack. The new Monitor PM will determine path forward.

Secure:Static Analysis - GMAU - Users running Static Analysis jobs

The highest of the number of unique users who have run SAST or Secret Detection jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10%

Is Primary PI: true

Health: Okay

• On track, continued healthy growth of static analysis usage.

Metric Name: xMAU, static analysis - paid GMAU and all GMAU

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Implementation Status: Complete

• Can't currently differentiate between free OSS Ultimate users and paid Ultimate users.

Lessons Learned (June):

• Tracking upgrades is really difficult to today, but we're starting to get directional data to better understand how many and what vectors upgrades from SAST come from. I believe that our in-product ugprade paths are now starting to see conversion. Additional ideas are being pursued to continue this trend.

Focus This Month (July):

• Moving forward we are focused strongly on data accuracy and reduction of false positives. We are actively rolling out a new vulnerability tracking improvement from Vulnerability Research which should materially reduce false positives. This approach is appearing to work out and produce more accurate tracking. This also opens a new metrics problem that we need a way to track accuracy with real-world data. We intend to measure the improvements of this change via the new vulnerabilities data in Sisence to ensure this change results in material positive impact before rolling it out. This feature has gotten heavily delayed due to rapid action DB work and upstream bugs with vulnerability reports that our group does not directly control.
• We're working to improve benchmarking capabilities to be better equipped to compare security results between versions of our analyzers and other third party tools to help customers and prospects answer questions about the depth and breadth of what types of vulns can we detect.
• To increase efficiency and reduce maintenance costs of our 10 linter analyzers we are transitioning to SemGrep for many of our linter analyzers. If successful we will reduce ~10 open source analyzers into 1 while also improving the detection rules and proving a more consistent rule management experience across these analyzers, this will be materially impactful as we seek to build our proprietary SAST engine on top of these results. In 13.12 we released this capability for JS, TS, and Python and are pursuring additional language transitions via our GSoC project. We're already pushing 170k Semgrep jobs with ~8k more per week!

Secure:Dynamic Analysis - GMAU - Users running a Dynamic Analysis scan

Number of unique users who have run one or more DAST or Fuzzing jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10% per-month

Is Primary PI: true

Health: Okay

• On track, usage continues to trend upwards, not including the anomalous self-managed increase last month.

Metric Name: user_dast_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Implementation Status: Complete

• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (June):

• Self-managed usage has shrunk since last month, pointing to April's usage boost being due to trials or PoVs, not a true bump in users. A way to filter out trial/PoV users would be extremely helpful in GMAU metrics.
• Fuzzing is not currently included in the Dynamic Analysis charts, so there is not a good single source of truth for Dynamic Analysis GMAU.

Focus This Month (July):

• Finishing noisy vulnerabilities aggregation to stop DAST from overwhelming the Vulnerability Report.
• Adding browser-based authentication to replace the current authentication process. This provides better troubleshooting when authentication fails and addresses a major pain point for DAST users.
• DAST API scans switching to use the API Security engine.

Secure:Composition Analysis - GMAU - Users running any SCA scanners

The highest of the number of unique users who have run one or more Dependency Scanning jobs, or number of unique users who have run one or more License Scanning jobs.

SaaS/SM: Both

Target: 2% higher than current month (SaaS and self-managed combined) for All and Paid

Is Primary PI: true

Health: Okay

• On track

Metric Name: user_license_management_jobs, user_dependency_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Implementation Status: Complete

• Currently our data is a lagging indicator (the majority of our users are self hosted and update 3 months behind current), trends are accurate but due to extrapolation of self-hosted, low opt-in of self-hosted, overall missing granularity, and 28 day cycles actual numbers are imprecise.

Lessons Learned (June):

• Nothing through use of xMAU Metrics.

Focus This Month (July):

• 13.11 and 13.12 team was dedicated to rapid action, we still have a few lingering issues and are also working on Software Composition Analysis removals and deprecations for 14.0 and the delayed bugs from 13.11+. 14.0-14.1 we are focusing on improving testing to speed up and make easier development of features as well as make it easier for community contriutors to test. Auto remediation is still blocked by a token issue.

Secure:Threat Insights - GMAU - Users interacting with Secure UI

Number of unique sessions viewing a security dashboard, pipeline security report, or expanding MR security report. Same as Paid until OSS projects can be separated.

SaaS/SM: SaaS

Target: Progressively increasing month-over-month, >2% per-month

Is Primary PI: true

Health: Attention

• GMAU dipped 2.5%. Unsure if this is still a data integrity issue. Total (non-unique) page views are also down about 7%.

Metric Name: n/a

Chart (Sisense↗)

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5670

Implementation Status: Definition

• Collection is complete for .com only
• Snowplow data is session only, not unique users
• Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.
• Cannot separate out GitLab employee usage from customers

Lessons Learned (June):

• Unclear if April and May drops are data issues or a legitimate signal. Most significant drop in page views in a year.
• Even if data is incomplete, growth did occur MoM for most areas outside Security Center. Drop may be attributable entirely to decreased usage here.

Focus This Month (July):

• Begin to adjust roadmap based on Kano survey results
• Continue focus on key technical debt around performance, scalability, and maintainability
• Evaluate main drivers of Error Budget overages and work to address

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

The number of users who have run a container scanning job or interacted with the Protect UI in the last 28 days of the month.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >2%

Is Primary PI: true

Health: Okay

• Usage growth dropped slightly from its April high. We may continue to see a small, temporary decline in usage as customers adapt their pipelines to work with the new Trivy scanner. Long-term, this is expected to provide a significant boost to usage.
• Although MAU data of the Alert Dashboard is not yet included in the broader numbers for these charts, we are seeing positive early signs from the data we do have that usage is increasing sharply.

Metric Name: user_container_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Limitations in data (see SCA limitations above)
• Although we have released the Alert Dashboard, we do not yet have a good way of tracking monthly active users for this area. Instrumentation is expected to be complex (1-2 months of one developer's time) and usage is expected to remain low until additional alert times are added to the dashboard. Currently instrumenting these metrics is a low priority.

Lessons Learned (June):

• The switch from Clair to Trivy has allowed us to close 20+ outstanding customer issues and bugs. Adoption is expected to increase significantly after the 14.0 release.

Focus This Month (July):

• Replacing Clair with Trivy (https://gitlab.com/groups/gitlab-org/-/epics/5398) will be the default beginning in the 14.0 release and will set the foundation for us to be able to scan containers in production environments
• Allow Users to Edit yaml-mode Scan Execution Policies in the Policy UI (https://gitlab.com/groups/gitlab-org/-/epics/5362) is being worked on for a July release. This change will allow us to default the policy feature flag to on.

Other Stage PIs Details

Enablement - Section PPI, Stage PPI - Median End User Page Load Time

Median end user page performance collected by Real User Monitoring. This metric captures how well we are doing in serving our primary customers, our end users, by measuring their end user experience.

SaaS/SM: Both

Target: 1.3

Is Primary PI: true

Health: Attention

• Median page load times have not changed since April, however the outliers (p90 and p95) have slightly decreased.
• Now Median and p90 load times by country by week are available.

Metric Name: performanceTiming

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• We are working to directly tie page load data to routes.

Lessons Learned (June):

• Page load times are not significantly affected by geography, with median times of 1.1s for US, 1.2 for most of Europe, and 1.3 for Australia. It seems internet latency is not a significant driver of overall performance for .com, however pages with many async requests will see higher impact.
• We are working to directly tie page load data to routes, with some of this content already sent but not processed in our data pipeline.
• We are also working to include RUM data (frontend performance) in error budgets, but pulling it into Grafana has proven more difficult than originally thought. We will need to update our lambda to push to Prometheus.
• Also exploring a CI test suite to catch regressions before they make it to production.

Other Group PIs Details

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Of the total number of self-managed installs, what percentage are on one of the three most recent versions.

SaaS/SM: Both

Target: 40%

Is Primary PI: true

Health: Attention

• A target of 40% has been set. 40% is the upgrade rate that was reached in early 2019.
• Insights - % on latest 3 versions has been stable, median age of version has returned to increasing trend.
• More research is required to understand reasoning for jump in age across all versions and editions.

Metric Name: versions_behind_latest

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10278985&udv=1102166
• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10279895&udv=1102166

Implementation Status: Complete

• Primary PPI is complete, and target is set.

Lessons Learned (June):

• The most recent update to the median age of license shows an increase across the board. This will continue to be investigated and discussed to see how we can trend downward again. A new research topic to explore is, how can we prompt community edition users to upgrade more regularly, since they are notably the largest median age.
• Security was a central focus for admins to upgrade their version from the ease of upgrading study. The inital MVC has been merged, to prompt users to sign up for the security newsletter in the next steps docs after installation. The next iteration will be to prompt current users to sign up for the security newsletter in the admin console.
• Another important aspect for admins to upgrade was "when developers are asleep", or less active within an organization. The MVC will be to display this activity to admins, and showcase the time of least activity.
• Work has begun on better highlighting when instances are out of date to administrators. This will impact results once it has been completed, estimated for this milestone.
• Based on the ease of upgrading study admins also were open to being notified more prominently when an upgrade was available. Work has begun on creating MVC's for a few new ways to notify admins in their preferred communication channel, that an upgrade is available.

Enablement:Memory - Group PPI - Memory Consumed

Average memory consumed by all invididual GitLab processes

SaaS/SM: Both

Target: 1.5GB

Is Primary PI: true

Health: Attention

• Memory consumption remains stable; 14.0 data to early to analyze.
• In collaboration with the Secure stage the team reduced the memory impact for the worst 1% of calls to the security report endpoint from 7GB down to 21MB,a 99.7% reduction. This shows that increasing visibility of problems plus collaboration can make a huge difference!
• By moving a subset of SideKiq workers from the primary database to read-only secondaries, the team removed about 10% of transactions from the primary PostgreSQL database - even on busy days. This is important because it increases the stability and performance of GitLab.com
• To further reduce memory consumption, we are planning to work on splitting the application into functional parts to ensure that only needed code is loaded and on reducing the memory consumption for Puma and Sidekiq endpoints.

Metric Name: topology.nodes[0].node_services

Chart (Sisense↗)

Implementation Status: Complete

• Working to adjust chart to include how many of each process are started by default to better represent a default configuration.
• Plot displays a 13.13 release for unknown reasons. Investigating

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Database query Apdex, with 100ms the target and 250ms tolerable. This measures the ratio of queries which complete within the satisfactory time, informing how well the database is scaling and performing.

SaaS/SM: Both

Target: 0.99

Is Primary PI: true

Health: Attention

• The query Apdex has decreased in all charts after the release of 13.10, including the weekly one, which has a pretty constant number of instances reporting stats. No similar trend on the GitLab.com charts, nor any other clear signal indicating what may be affecting the Apdex. This is still present in 13.12 but the team has not focused on an investigation due to capacity constraints. Current focus is mitigating the Primary Key Overflow risk.
• We expect the updates performed in the scope of Research Database queries for performance and frequency to further improve the ratio of queries which complete within 250ms and lower the variance of the Apdex even further.
• We expect our work on automated database migration testing to indirectly improve the database performance by minimising the number of database related incidents during deployments.

Metric Name: query_apdex_weekly_average

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/227305

Implementation Status: Complete

• The PPI has been instrumented in 13.4.

Lessons Learned (June):

• Apdex on GitLab.com exceeds our group PPI (see 100ms - Tolerable 250ms and 50ms - Tolerable 100ms for master) but reflects with sharp drops the production incidents that related to the database.

Verify:Runner - Other - Total downloads of runner binaries and docker images per month

Count of downloads of runner binaries grouped by platform, gitlab-runner and gitlab-runner helper container images. Measuring runner binaries' downloads will provide insight into whether we are making the right investments in core runner features and capabilities by platform (operating system and computing architecture). The assumption is that if self-managed users continue to maintain and upgrade their GitLab Runner build infrastructure, this may suggest continued long-term use of GitLab as their DevOps platform.

Funnel Description: Handbook Link

SaaS/SM: SM

Target: Target monthly growth rate of 3% given monthly trends to date.

Is Primary PI: true

Health: Unknown

• Not instrumented

Metric Name: NA

Implementation Status: Definition

• We need to start collecting the AWS S3 bucket metrics and integrate them into the data warehouse. Issue for tracking is in the Infra Project pending scheduling.

Lessons Learned (June):

• Insight - Downloads of the gitlab-runner and the gitlab-runner helper container images continue to grow at an average rate of 3% monthly. The cumulative download count is now over 639M. The total downloads for June 2021 came in at 17.6M (gitlab-runner) and 8M (gitlab-runner helper).

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Centralized Dashboards
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Talent Acquisition Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
-1	Confidential	Metric & metric health are confidential
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.