Stage and Group Performance Indicators

Maintained by:

SMAU Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Analytics Features	Okay	Both	Complete
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Create	SMAU	Unique users writing to a Git repository	Okay	Both	Complete
Verify, Verify:CI	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Package:Package	SMAU, GMAU	Count of users publishing packages	Okay	Both	Data available
Release:Release	SMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Okay	Both	Complete
Sec, Secure	Section MAU, SMAU	Unique users who have used a Secure scanner	Okay	Both	Complete
Protect:Container Security	SMAU, GMAU	Users running Container Scanning or interacting with Protect UI	Attention	Both	Complete

GMAU Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Analytics Features	Okay	Both	Complete
Manage:Access	Paid GMAU	MAU Using Paid SAML	Okay	SaaS	Complete
Manage:Compliance	Paid GMAU	MAU Viewing Compliance Features	Attention	Both	Complete
Manage:Import	GMAU	MAU importing	Unknown	Both	Dashboard
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Plan:Product Planning	Paid GMAU	MAU creating Epics each Month	Okay	Both	Complete
Create:Source Code	GMAU	Unique users writing to a project Git repository	Okay	Both	Complete
Create:Code Review	GMAU	Unique users using Merge Requests	Okay	Both	Complete
Create:Editor	GMAU	Unique users committing via any web editor	Attention	Both	Complete
Create:Ecosystem	GMAU	MAU of any integration	Unknown	Both	Instrumentation
Enablement:Geo	Paid GMAU	Number of unique users utilizing a Geo secondary	Attention	Both	Instrumentation
Enablement:Global Search	Paid GMAU	The number of unique paid users per month	Attention	Both	Complete
Enablement:Infrastructure	Paid GMAU	Number of unique paid users that perform an action on GitLab.com	Attention	SaaS	Complete
Verify, Verify:CI	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Verify:Pipeline Authoring	GMAU	Number of unique users interacting with gitlab-ci.yml file	Unknown	Both	Implementation
Verify:Testing	Paid GMAU	Count of active paid testing feature users	Unknown	Both	Data Availability
Package:Package	SMAU, GMAU	Count of users publishing packages	Okay	Both	Data available
Release:Release	SMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	Proxy user count based on attached clusters	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Okay	Both	Complete
Secure:Static Analysis	GMAU	Users running Static Analysis jobs	Okay	Both	Complete
Secure:Dynamic Analysis	GMAU	Users running DAST	Okay	Both	Complete
Secure:Composition Analysis	GMAU	Users running any SCA scanners	Okay	Both	Complete
Secure:Fuzz Testing	GMAU	Users running fuzz testing	Attention	Both	Definition
Secure:Threat Insights	GMAU	Users interacting with Secure UI	Okay	SaaS	Definition
Protect:Container Security	SMAU, GMAU	Users running Container Scanning or interacting with Protect UI	Attention	Both	Complete

Other Stage PIs Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement	Section PPI, Stage PPI	Median End User Page Load Time	Attention	Both	Target

Other Group PIs Summary

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement:Distribution	Group PPI	Percentage of installations on the 3 most recent versions of GitLab	Attention	Both	Complete
Enablement:Memory	Group PPI	Memory Consumed	Attention	Both	Complete
Enablement:Database	Group PPI	Database Query Apdex (100ms target, 250ms tolerable)	Attention	Both	Complete
Verify:Runner	Other	Total downloads of runner binaries and docker images per month	Unknown	SM	Definition

SMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page.

SaaS/SM: Both

Target: 116050

Is Primary PI: true

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Insights - The Optimize data for January is inaccurate due to this usage ping issue. Many page views for self managed are showing as negative.
Insights - Page views by SaaS users continued a downward trend that started in December. SaaS usage dropped by 10% in December, and again by 13% in January. Views are down for all Optimize pages except Group Insights, which increased by 45%.
Improvements - The new DevOps Adoption feature was rolled back in January to pivot from the concept of Segments to only supporting groups in the table (https://gitlab.com/gitlab-org/gitlab/-/issues/296972). There is strong interest in this feature from enterprise customers wanting insights into how GitLab is being used in their organizations. Refactoring work commenced in 13.9 and will continue into 13.10 with the goal of renabling the feature in 13.10 or 13.11. This is expected to result in an increase in DevOps Reports usage. After this work is completed, the feature will be added at the group level, further increasing availability and adoption.
Improvements - Group-level VSA is being refactored to create more space on the page for new features and to simplify navigation. This work is scheduled to be completed in 13.9. Next up we are adding the ability to sort items in each VSA stage by cycle time, which will allow users to identify aging items and take action to reduce overall cycle time in their value streams. Additionally, we are refactoring project-level VSA code to use the group-level code which will improve usability and add functionality when viewing VSA at the project level, as well as allow us to iterate faster because we won't be maintaining two sets of code. All of these improvements combined will improve the usability of VSA and is expected in increase adoption as we announce each improvement. These are the foundational steps we are taking before embarking on our FY22 vision for VSA which will add many new features and a much richer value stream management experience.
SaaS data is not available for the paid SMAU chart due to limitations of the redis counter that is used.

Implementation Status: Complete

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 484,510 for All, 148,176 for Paid

Is Primary PI: true

URL(s)

Metric Name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Achieved - FY21 Q4 targets of 437,490 for All, 101,835 for Paid. New targets set for FY22 Q1.
Driver - For Paid, continue high level progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible, and continuing to make incremental progress on iterations.
Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
Improvement - For All, continue making progress on quality of life improvements for existing features.
Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption.
Improvement - For Paid, continue making incremental, iterative progress towards Plan's core JTBD as outlined in the Issue Tracking direction.

Implementation Status: Complete

xMAU data models are now updated with the new counter that aggregates unique users across many different issue actions.
New Plan xMAU dashboard is live.

Lessons Learned (February):

Only 28% of instances are currently on a version with the new Plan xMAU metric counters.
The ratio of net new Paid GMAU to churned Paid GMAU improved from 3.75 in December to 3.99 in January, which indicates net growth and retention for Plan Paid users is healthy.
MAU interacting with Iterations increased by 52% from December to January. The next step to drive feature adoption is more holistically integrating Iterations into Boards.

Focus This Month (March):

Driver - Determine technical feasibility for allowing an issue to be assigned to multiple groups, which would likely solve Plan adoption blockers (priorities 1 and 3).
Continue making progress on the 13.9 release plan and roll into 13.10 goals.
Open > 3 documentation MRs in support of better articulating Plan use cases.
Complete the validation track for saved filters/searches, customizable issue types, and interacting with a full issue from within a Board
Merge architectural blueprint for consolidating Groups and Projects on the backend.

Create - SMAU - Unique users writing to a Git repository

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.

Is Primary PI: true

URL(s)

https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Observation - December 2020 SMAU is less than November 2020 SMAU. A similar drop can be observed in GMAU as well, suggesting a common cause. The cause is likely seasonality consistent with the end of year. For this reason January SMAU and GMAU is expected to return to November 2020 levels and continue growing.
Insight - on the median GitLab instance, 33% of users perform a Git write operation each month (source).
Insight - on the median Paid GitLab instance (self-managed), 50% of users perform a Git write operation each month (source).

Implementation Status: Complete

Improvement - AMAU is currently over counted. A deduplicated counter was implemented gitlab#234027 in GitLab 13.6. The charts do not yet reflect the new counter gitlab#291943.

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: We delivered on 92% of our FY21 Q4 target of 1 million unique users triggering pipelines, and fell short of our 10% MoM growth. In FY22 Q1, we anticipate accomplishing this target of 1.1M unique users as predicted in our [tertiary chart](https://app.periscopedata.com/app/gitlab/798616/WIP-Linear-Predicted-Dashboard?widget=10611135).

Is Primary PI: true

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

Metric is instrumented for both SaaS and Self-managed

Implementation Status: Complete

With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new gitlab-ci.yml files created.
We are also working to instrument other performance indicators for Verify:CI
Updating dashboard to have both SM / SaaS and Paid / All breakouts

Lessons Learned (February):

Insight - In January 2021 we saw a slight uptick in users again, mostly concentrated in the Paid, self-managed user base for continuous integration. Although, we have not delivered on our ambitious 1.14M unique users triggering pipelines, we are close at 917K users meaning only ~220K users shy of attaining our goal.
Improvement - With the plateau in monthly active users, we proceeding with an evaluation of maturity of our core Jobs To Be Done in Continuous Integration to offer a challenge or affirmation for the "Loveable" status.
Improvement - We are finalizing the research in adoption hurdles for SCM to Verify to further expand opportunities for SpO as well as increase MAU. We will be prioritizing the next generation of features in CI that target these gaps once we invest in addressing technical debt, bugs, and UI polish in the 1H of FY22.
Improvement - Lastly, we are beginning to explore the idea of NextGen CI, which is related to the SCM to Verify adoption challenges, while tackling other interesting problem sets like ML/AI in CI.

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Health: Okay

Improvements - As of March 1st, 2021, we now have user level data for SaaS and Self-Managed.

Implementation Status: Data available

Lessons Learned (February):

Insights - We only have data for January and February, but we saw 45% growth in the number of SaaS users that published a package to GitLab.com.
Insights - npm and Maven are the most commonly used formats
Improvements - In 13.10, we will release a key improvement for npm that will relax GitLab's stric naming conventions.
Improvements - In 13.10, we will resolve a bug that has been creating some confusion for heavy users of Maven.
Improvements - We do expect the number of Self-Managed users to rise as more pinging instances upgrade to 13.8.

Release:Release - SMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 301,000 for SMAU, 124,000 for paid SMAU

Is Primary PI: true

URL(s)

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

Insight - Our FY21 Q4 is 30% and this month, and we did not meet our target. The increase in deployments was only at 11% in this quarter.
Insight - We have exceeded our yearly target for paid SMAU. We are currently at 134k users. Target was 124k.
Improvements - Even with the end of year deployment freeze, there was still an increase of 6K users this month.
Improvements - Release has recently shipped some of the DORA4 metrics and specifically deployment frequency which gives the user insights into their deployment efficiency and should encourage them to increase deployments.
Improvements - In the next few milestones, we will add in-product guidance for users that have pipelines that are not triggering deployments.

Implementation Status: Complete

We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
We need to Expand Revenue Attribution to Release SMAU
Improvements - Instrumentation was changed to the formal metric recommended by the Product Intelligence team, making the data more accurate.
Improvements - Counter was instrumented for 28-days
Improvements - User and unique count of deployments are instrumented
Improvements - SaaS and self-managed data are both available on a single chart
Updating dashboard to have Paid / All breakouts
Improvements - Target has been fixed and reflected in the charts.

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

SaaS/SM: Both

Target: 18000

Is Primary PI: true

Metric Name: project_clusters_enabled, group_clusters_enabled, instance_clusters_enabled

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

Improvement - We are working towards moving cluster creation over to Terraform in order to provide customizable, IaC based cluster provisioning. This requires a production ready GitLab Managed Terraform State.

Implementation Status: Complete

The target was fixed, and there is a simple pattern how it can be adjusted in the widget
Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
Improvement - The GitLab Kubernetes Agent has been released, and it's first usage pings are coming in. See below.
Updating dashboard to have both SM / SaaS and Paid / All breakouts
The Data Team strongly advises against estimating this value given that it is already an estimate. An estimate of an estimate yields very inaccurate results.

Lessons Learned (February):

Insight - As the metrics got fixed in August, we are still trying to get an idea about the natural variation versus trends in the data. As a result, we reach the target from month to month even though the target is for the quarter. As the most recent iteration in this quest, the target was modified to 18000 SMAU.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interact with alerts or incidents. (This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority.)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 12,000 Estimated total SMAU

Is Primary PI: true

URL(s)

https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

We met and surpassed our FY21Q4 usage target of 12k SMAU

Implementation Status: Complete

Instrumentation complete - issue.

Lessons Learned (February):

Insight - There is a data issue for January 2021. Insights for Monitor SMAU/GMAU remain the same as those for December 2020.
Insight - SMAU for December has decreased from November 2020. We believe this to be due to the holiday season.
Insight - Monthly interactions on Alerts are far lower than those on Incidents. As an example, we can compare the AMAU for users changing the status on alerts from December (38) to the number of users who manually created an incident in December, which was ~2,300. Our hypothesis is two-fold. First, we believe configuring alerts is challenging. We addressing this by making it easier to integrate alerting sources into GitLab by enabling users to customize the endpoints. Second, we know that it is very common for teams to create incidents manually. They do this for two reasons - 1. Many teams do not have performance alerting automation set up and therefore rely on a human reporting a problem and 2. Teams that do have automatic performance alerting still want to create incident manually because they want a human to triage the alert first.
Improvements - We are making it easier to integrate alerting sources into GitLab by enabling users to customize the endpoints.
Improvements - We are adding on-call schedule management, which is a required feature set for Incident Management to compete in market.

Sec, Secure - Section MAU, SMAU - Unique users who have used a Secure scanner

The number of unique users who have run one or more Secure scanners.

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: user_unique_users_all_secure_scanners

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

currently instrumentated to meet OKRs (see instrumentation, Lessons Learned for more details)

Implementation Status: Complete

Can't currently differentiate between free-to-OSS/education Gold/Ultimate users and paid Gold/Ultimate users
Threat Insights data is not yet included in this count until we support user-level metrics in Snowplow for self-managed

Focus This Month (March):

Identify a way to denote underreporting on Secure-related *MAU charts to remove confusion / clarify metrics interpretation
Identify usage by product tier while removing free-to-OSS/education Gold/Ultimate users
Support Data team on improving Self-Managed Uplift for Secure

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

The number of users who have run a container scanning job or interacted with the Protect UI in the last 28 days of the month.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >2%

Is Primary PI: true

Metric Name: user_container_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

Usage decreased slightly from December -> January. This was anticipated and expected. See the lessons learned section for more details.
It is probable that the company-wide data collection challenges in January had at least some impact on these numbers as well.
Usage growth is projected to continue to decline in the short-term. See lessons learned section for more details.

Implementation Status: Complete

Limitations in data (see SCA limitations above)
Data is not available yet for the Security Orchestration category as it has not yet reached Minimal maturity

Lessons Learned (February):

As Container Security has not been resourced for some time, we have a pent-up list of feature requests and on-going maintenance work that is presenting a strong headwind to increasing usage.
Usage growth is anticipated to be minimal or negative until we can make a dent in the outstanding usability challenges. Work is anticipated to be completed by March, with usage numbers recovering across the March-June timeframe.
The release of the Security Orchestration category (planned for February) will also bolster the Group's usage numbers once we instrument metrics tracking for the category.

Focus This Month (March):

Alert Dashboard MVC (https://gitlab.com/groups/gitlab-org/-/epics/3438) will be released this month
Project-level DAST Scan Schedule Policies (https://gitlab.com/groups/gitlab-org/-/epics/4598) is being worked on for a March release
Replacing Clair with Trivy (https://gitlab.com/groups/gitlab-org/-/epics/5398) will set the foundation for us to be able to scan containers in production environments

GMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page.

SaaS/SM: Both

Target: 116050

Is Primary PI: true

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Insights - The Optimize data for January is inaccurate due to this usage ping issue. Many page views for self managed are showing as negative.
Insights - Page views by SaaS users continued a downward trend that started in December. SaaS usage dropped by 10% in December, and again by 13% in January. Views are down for all Optimize pages except Group Insights, which increased by 45%.
Improvements - The new DevOps Adoption feature was rolled back in January to pivot from the concept of Segments to only supporting groups in the table (https://gitlab.com/gitlab-org/gitlab/-/issues/296972). There is strong interest in this feature from enterprise customers wanting insights into how GitLab is being used in their organizations. Refactoring work commenced in 13.9 and will continue into 13.10 with the goal of renabling the feature in 13.10 or 13.11. This is expected to result in an increase in DevOps Reports usage. After this work is completed, the feature will be added at the group level, further increasing availability and adoption.
Improvements - Group-level VSA is being refactored to create more space on the page for new features and to simplify navigation. This work is scheduled to be completed in 13.9. Next up we are adding the ability to sort items in each VSA stage by cycle time, which will allow users to identify aging items and take action to reduce overall cycle time in their value streams. Additionally, we are refactoring project-level VSA code to use the group-level code which will improve usability and add functionality when viewing VSA at the project level, as well as allow us to iterate faster because we won't be maintaining two sets of code. All of these improvements combined will improve the usability of VSA and is expected in increase adoption as we announce each improvement. These are the foundational steps we are taking before embarking on our FY22 vision for VSA which will add many new features and a much richer value stream management experience.
SaaS data is not available for the paid SMAU chart due to limitations of the redis counter that is used.

Implementation Status: Complete

Manage:Access - Paid GMAU - MAU Using Paid SAML

The number of unique users logging into GitLab using SAML SSO for Groups.

SaaS/SM: SaaS

Target: 30000. See https://gitlab.com/gitlab-com/Product/-/issues/1832 for further detail.

Is Primary PI: true

Metric Name: user_auth_by_provider (Group_SAML)

Chart (Sisense↗)

Health: Okay

Improvements - In 13.10, Auditor users will have read-only access to groups with SSO enabled. This will allow the support organization to reduce the number of Administrators for GitLab.com and meet their OKR!
Improvements - In 13.10, GitLab will begin enforcing SSO for Git activity. This will provide the same level of protection to Git as the Web UI and been requested by several large customers and prospects.
Improvements - In 13.10, we will increase security for GitLab.com by lowering the timeout for SAML SSO sessions from 7 days to 1 day.
Insights - We continue to see a healthy upward trend for group SAML logins per month! This is a combination of decreasing the SSO timeout and enteprise customer growth for GitLab.com . My analysis shows that decreasing the SSO timeout increased Group SAML usage by 10%.

Implementation Status: Complete

Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features

The number of unique users viewing Audit Events, Compliance Dashboard, or Credential Inventory.

SaaS/SM: Both

Target: TBD (expected [April 2021](https://gitlab.com/gitlab-org/manage/compliance/general/-/issues/13))

Is Primary PI: true

Metric Name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory, usage_data_g_compliance_dashboard, usage_data_a_compliance_audit_events_api

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Attention

We're still building out the compliance features that are tracked by this metric. This improvement work will take longer than desired to complete since we do not have enough people to ship any faster. We expect iterations to ship on a continuous basis across each of these features.
Setting a target is premature since we're still collecting initial data. We would like to collect a total of 6 months worth of data before setting a target.
Result - Increased GMAU by ~25% in December by making Audit Events available to Developers and making the feature more discoverable.
Adding a Change Management view to the Compliance Dashboard. Planned for Q2 2021.
Improving PAT & SSH credential management. Ongoing with expected epic completion in Q2 2021.
Improving discoverability of Compliance features. Planned for Q2 2021. - Adding additional credentials to the credential inventory. Planned for Q3 2021.

Implementation Status: Complete

Manage:Import - GMAU - MAU importing

A monthly count of the unique number of users performing an import

SaaS/SM: Both

Target: TBD - See Health section below for details

Is Primary PI: true

Metric Name: unique_users_all_imports

Health: Unknown

No target defined yet.
Target will be defined after the metric instrumentation is implemented and we get a baseline of data.
Improvement - Reducing friction for GitLab-to-GitLab migrations - The MVC for the Group Migration feature was enabled in production in 13.9. This feature will enable a much easier migration of an entire GitLab group from one instance to another.

Implementation Status: Dashboard

Instrumentation has been implemented in 13.9.
Dashboard update is in progress.

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 484,510 for All, 148,176 for Paid

Is Primary PI: true

URL(s)

Metric Name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Achieved - FY21 Q4 targets of 437,490 for All, 101,835 for Paid. New targets set for FY22 Q1.
Driver - For Paid, continue high level progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible, and continuing to make incremental progress on iterations.
Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
Improvement - For All, continue making progress on quality of life improvements for existing features.
Improvement - For Paid, creating multi-level Epics spanning a Group hierarchy is clunky and non-intuitive. Improving this, along with making the Roadmap more interactive and intuitive would be a huge catalyst to drive Product Planning adoption.
Improvement - For Paid, continue making incremental, iterative progress towards Plan's core JTBD as outlined in the Issue Tracking direction.

Implementation Status: Complete

xMAU data models are now updated with the new counter that aggregates unique users across many different issue actions.
New Plan xMAU dashboard is live.

Lessons Learned (February):

Only 28% of instances are currently on a version with the new Plan xMAU metric counters.
The ratio of net new Paid GMAU to churned Paid GMAU improved from 3.75 in December to 3.99 in January, which indicates net growth and retention for Plan Paid users is healthy.
MAU interacting with Iterations increased by 52% from December to January. The next step to drive feature adoption is more holistically integrating Iterations into Boards.

Focus This Month (March):

Driver - Determine technical feasibility for allowing an issue to be assigned to multiple groups, which would likely solve Plan adoption blockers (priorities 1 and 3).
Continue making progress on the 13.9 release plan and roll into 13.10 goals.
Open > 3 documentation MRs in support of better articulating Plan use cases.
Complete the validation track for saved filters/searches, customizable issue types, and interacting with a full issue from within a Board
Merge architectural blueprint for consolidating Groups and Projects on the backend.

Plan:Product Planning - Paid GMAU - MAU creating Epics each Month

The number of unique users creating Epics each month.

SaaS/SM: Both

Target: 3,900 Combined Unique Users for Self Managed and SaaS

Is Primary PI: true

Metric Name: epics

Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Achieved - Target of 3,900 (3,943 actual GMAU for epic creation)
Insight - Epic creation for SaaS continues to grow. Self-Managed was stable, but that could be due to the Janauary tracking bug.
Improvement - Epic Boards MVC 1 is planned to ship in 13.10 and should continue the trend into the end of FY21 by enabling Portfolio-level planning.
Improvement - Epic Swimlanes shipped in 13.6
Closed - original PI dashboard issue for epic creation

Implementation Status: Complete

We consider our Epic Creation GMAU as "Complete". However, our next target will be set after a second instrumentation.
Instrument "all epic interactions" during 13.9 - 13.10. Instrumentation issue and dasboard issue.
A new target will be selected when we have a baseline.

Create:Source Code - GMAU - Unique users writing to a project Git repository

A rolling count of the number of unique users who performed any Git write operation within the last 28 days

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.

Is Primary PI: true

URL(s)

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Starting in GitLab 13.6 AMAU for this metric takes into consideration all repository types.
Improvement - work towards providing users more insights while navigating their code with Code Intelligence
Improvement - work towards providing more control over managing repository size

Implementation Status: Complete

Enabled on GitLab.com
Released for self-managed in GitLab 13.3

Create:Code Review - GMAU - Unique users using Merge Requests

This is defined as users who perform one or more of the following actions in a merge request - create, merge, or close a Merge Request.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.

Is Primary PI: true

URL(s)

https://app.periscopedata.com/app/gitlab/526749/Create---Code-Review-category-dashboard

Metric Name: merge_requests_users

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

Target - This aligns with our current understanding that Code Review is about 50% of the MAU of the stage metric and continuing to maintain that difference.
Insight - Paid users of Code Review make up about one-third of total Code Review users - this provides an opportunity to convert to paid and increase users.
Improvement - Support for custom commit messages when applying a suggested change will allow more teams with strict commit message guidelines to adopt this feature and maintain conformity with the project.
Improvement - Mark changes as viewed in merge requests is a foundational step towards tracking and communicating progress on merge requests to other participants.
Improvement - SCM <-> Code Review <-> Verify correlation charts now defined in new charts

Implementation Status: Complete

Enabled on GitLab.com 2020-07-04
Enabled for self-managed on GitLab 13.2
Working on more comprehensive MAU tracking in &4948

Create:Editor - GMAU - Unique users committing via any web editor

A rolling count of the number of unique users who performed a Git write operation to any respository using the Web IDE, Web Editor or Snippet within a stage in the last 28 days

SaaS/SM: Both

Target: 4% growth month over month for All

Is Primary PI: true

URL(s)

https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU

Metric Name: action_monthly_active_users_ide_edit

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Attention

New target GMAU to be set once we get cumulative, de-duped data from newly aligned categories.
Insight - Rebound from the holiday season to nearly November levels. We'll be monitoring to ensure consistent organic growth in subsequent months.
Improvement - No targeted efforts in the next few releases to specifically grow GMAU, but currently working to migrate the Web IDE to use Editor Lite which will bring consistency across editors and a moderate performance improvement.

Implementation Status: Complete

Instrumentation of Static Site Editor and Wiki to roll up into GMAU completed in 13.7, waiting for data to be reflected in the charts.

Create:Ecosystem - GMAU - MAU of any integration

Monthly Unique Users of any GitLab integration.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

Metric Name: TBD

Health: Unknown

No target defined
Improvement - Adding Group and Instance integration will allow larger sets of projects to be integrated easily
Improvement - Improvements and new functionality in the Jira integration should increase adoption, expanding on the success of our most popular integration

Implementation Status: Instrumentation

Awaiting implementation of data in Usage Ping
Improvement - (February 2021) First individual tracking counter (Jira) has shipped and is live on GitLab.com. Team will observe and ensure stability/performance, then enable the feature flag for the 13.10 release.
Awaiting implementation of Sisense dashboard (blocked by Usage Ping implementation)

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Number of unique users utilizing a Geo secondary. This adoption metric helps us understand whether end users are actually seeing value in, and are using, geo secondaries.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

Metric Name: geo_nodes

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

We've merged a first MR to enable gathering usage ping data from Geo secondaries. We plan to finish instrumentation in 13.10
We’ve only instrumented access to the Geo secondary Web interface yet. This is important because it measures users logging into the secondary UI; however, it does not account for Git access, which is much more common. This will be instrumented in Geo Usage Ping Epic. We also know that the UX of the secondary web interface is not good and we want to remove it, see Opportunity Canvas(internal). In order to assess the impact of our planned changes, having this (low) GMAU is really important.
For Disaster Recovery we are measuring potential Geo users - the number of active licenses. In an ideal world, no regular user would ever need to rely on Geo because there is no disaster, but if one occurs everyone benefits. Based on the node number distribution ~60% of our customers use Geo mainly for DR. I think this is worth measuring because setting up Geo is always a conscious decision by the customer - it must be configured.
We are working to add support for replicating all data types, so that Geo is on a solid foundation for both DR and Geo Replication.~80% of data types are now replicated. Group wikis are tentatively scheduled for 13.10 to be implemented by the Editor Group.

Implementation Status: Instrumentation

Geo is not available on GitLab.com today, so cannot use Snowplow or the .com database.

Lessons Learned (February):

Successfull failover to a Geo secondary on a 50k reference architecture for a large customer.
The number is still very low right now. Two potential reasons a) data is incomplete b) WebUI is read-only and requires a different UI. We are planning to change this, see this opportunity canvas.
Year over Year growth of 120% for new Geo customers.

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

The number of unique active users and unique paid users interacting with either Basic Search or Advanced Search per month.

SaaS/SM: Both

Target: 10% month over month (SaaS and self-managed combined)

Is Primary PI: true

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

January 2020
Significant decrease in self-managed GMAU, which seems to be due to some kind of charting or usage ping error as negative GMAU was being reported which should be impossible.
Decrease in SaaS could be attributed to performance issues impacting search this month. We are working to address and have made significant progress.

Implementation Status: Complete

Data collected is incomplete for Sept and Oct 2020

Lessons Learned (February):

Search for files by name and path is surfacing as one of our largest Global Search use cases. 13.8 Has made file searching easier for users
Scaling Elasticsearch will continue to require engineering as long as we see growth in usage and document count in SaaS

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

The sum of all monthly active users on gitlab.com that roll up to paid accounts in a 28 day rolling period.

SaaS/SM: SaaS

Target: 111750

Is Primary PI: true

Metric Name: COUNT_EVENTS_ACTIVE_USERS_LAST_28_DAYS_BY_PLAN_WAS_PAID

Chart (Sisense↗)

Health: Attention

Paid Monthly Active Users on GitLab.com decreased from 105K in December to 103K in January.
Insight - This dip appears to be seasonal as we saw a similar decrease in January 2020. We would expect churn to be accentuated in January as the result of contracts expiring before the new year.
Improvement - To influence Paid GMAU, we are continuing to focus on delivering SaaS Availability Improvements, working with the Manage stage to eliminate key feature gaps that exist for Adminstrators on .com compared to self-managed, and extending our compliance posture on Gitlab.com

Implementation Status: Complete

Instrumentation complete.

Lessons Learned (February):

Need to account for seasonal fluctuations into target projections.

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

Funnel Description: Handbook Link

SaaS/SM: Both

Is Primary PI: true

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

Metric is instrumented for both SaaS and Self-managed

Implementation Status: Complete

With ability to trigger a pipeline dependent on successful attempts to author CI yaml files, we have an issue to track number of new gitlab-ci.yml files created.
We are also working to instrument other performance indicators for Verify:CI
Updating dashboard to have both SM / SaaS and Paid / All breakouts

Lessons Learned (February):

Insight - In January 2021 we saw a slight uptick in users again, mostly concentrated in the Paid, self-managed user base for continuous integration. Although, we have not delivered on our ambitious 1.14M unique users triggering pipelines, we are close at 917K users meaning only ~220K users shy of attaining our goal.
Improvement - With the plateau in monthly active users, we proceeding with an evaluation of maturity of our core Jobs To Be Done in Continuous Integration to offer a challenge or affirmation for the "Loveable" status.
Improvement - We are finalizing the research in adoption hurdles for SCM to Verify to further expand opportunities for SpO as well as increase MAU. We will be prioritizing the next generation of features in CI that target these gaps once we invest in addressing technical debt, bugs, and UI polish in the 1H of FY22.
Improvement - Lastly, we are beginning to explore the idea of NextGen CI, which is related to the SCM to Verify adoption challenges, while tackling other interesting problem sets like ML/AI in CI.

Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file

Count the number of users that are pushing MRs to gitlab-ci.yml files. Since we'd like to count the on-going usage of our users with pipelines.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: N/A

Chart

Health: Unknown

Not instrumented

Implementation Status: Implementation

Instrumentation for .com is undergoing and currently is in-review, we should have some of the data ready on our next PI meeting.
Still waiting on instrumentation for usage ping data which is scheduled for 13.10. It appears that the data exists; however, the data team was facing challenges extracting it.
Evaluated for FY21-Q4 OKRs but not part of FY21-Q3 KRs

Lessons Learned (February):

Insights - We delivered the first MVC for Pipeline Editor, which has recieved great traction with preliminary signals on social such as the LinkedIn post got 149 likes & 1800+ views in 24 hours in addition to it receiving the most clicks from the "What's New" in 13.8 so far.
Improvements - Secondary improvements that will continue to drive this metric, will include the empty state for pipeline editor, which will allow new users to easily start using pipeline editor. Creating an versatile empty template, which will empower users to interact with the YAML file in any number of ways that are currently not supported today, and Branch selector for pipeline editor to allow users to work with the pipeline editor from any non-master branch.

Verify:Testing - Paid GMAU - Count of active paid testing feature users

A rolling count of unique users who have interacted with a paid testing feature in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: We are targeting X unique paid testing users by end of FY22 Q1.

Is Primary PI: true

Metric Name: NA

Chart (Sisense↗)

Health: Unknown

Individual Metrics are implemented and in the Dashboard.
An aggregate metric for unique users interacting with any paid Testing Feature is on track to review after 13.9 release.

Implementation Status: Data Availability

The target is missing in the chart.

Lessons Learned (February):

Insights - Considering growth trends in the previous chart, between December 2020 and January 2021 there was a 13% growth of snowplow sessions, which is showing a level of retention in addition to further engagement from either the existing user or more users.
Insights - Comparing overall clicks on the Code Quality tab against Unique Users of the feature, even with a small data set, implies strong repeat usage.
Improvements - Instrumentation of unique user tracking for all existing paid features on track to complete in 13.9. This has moved up since the last review.
Improvements - There are several bug fixes in progress and scheduled for Browser Performance Testing and the Group Code Coverage page that should increase engagement in those features and overall unique users.

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

Is Primary PI: true

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Health: Okay

Improvements - As of March 1st, 2021, we now have user level data for SaaS and Self-Managed.

Implementation Status: Data available

Lessons Learned (February):

Insights - We only have data for January and February, but we saw 45% growth in the number of SaaS users that published a package to GitLab.com.
Insights - npm and Maven are the most commonly used formats
Improvements - In 13.10, we will release a key improvement for npm that will relax GitLab's stric naming conventions.
Improvements - In 13.10, we will resolve a bug that has been creating some confusion for heavy users of Maven.
Improvements - We do expect the number of Self-Managed users to rise as more pinging instances upgrade to 13.8.

Release:Release - SMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 301,000 for SMAU, 124,000 for paid SMAU

Is Primary PI: true

URL(s)

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

Insight - Our FY21 Q4 is 30% and this month, and we did not meet our target. The increase in deployments was only at 11% in this quarter.
Insight - We have exceeded our yearly target for paid SMAU. We are currently at 134k users. Target was 124k.
Improvements - Even with the end of year deployment freeze, there was still an increase of 6K users this month.
Improvements - Release has recently shipped some of the DORA4 metrics and specifically deployment frequency which gives the user insights into their deployment efficiency and should encourage them to increase deployments.
Improvements - In the next few milestones, we will add in-product guidance for users that have pipelines that are not triggering deployments.

Implementation Status: Complete

We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
We need to Expand Revenue Attribution to Release SMAU
Improvements - Instrumentation was changed to the formal metric recommended by the Product Intelligence team, making the data more accurate.
Improvements - Counter was instrumented for 28-days
Improvements - User and unique count of deployments are instrumented
Improvements - SaaS and self-managed data are both available on a single chart
Updating dashboard to have Paid / All breakouts
Improvements - Target has been fixed and reflected in the charts.

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.

SaaS/SM: Both

Target: 18000

Is Primary PI: true

Metric Name: project_clusters_enabled, group_clusters_enabled, instance_clusters_enabled

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

Improvement - We are working towards moving cluster creation over to Terraform in order to provide customizable, IaC based cluster provisioning. This requires a production ready GitLab Managed Terraform State.

Implementation Status: Complete

The target was fixed, and there is a simple pattern how it can be adjusted in the widget
Proxy SMAU that we are looking to update to be based on the below IaC and Kubernetes Management AMAUs
Insight - With our IaC developments, we've started to collect reliable, usage-oriented metrics
Improvement - The GitLab Kubernetes Agent has been released, and it's first usage pings are coming in. See below.
Updating dashboard to have both SM / SaaS and Paid / All breakouts
The Data Team strongly advises against estimating this value given that it is already an estimate. An estimate of an estimate yields very inaccurate results.

Lessons Learned (February):

Insight - As the metrics got fixed in August, we are still trying to get an idea about the natural variation versus trends in the data. As a result, we reach the target from month to month even though the target is for the quarter. As the most recent iteration in this quest, the target was modified to 18000 SMAU.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 12,000 Estimated total SMAU

Is Primary PI: true

URL(s)

https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

We met and surpassed our FY21Q4 usage target of 12k SMAU

Implementation Status: Complete

Instrumentation complete - issue.

Lessons Learned (February):

Insight - There is a data issue for January 2021. Insights for Monitor SMAU/GMAU remain the same as those for December 2020.
Insight - SMAU for December has decreased from November 2020. We believe this to be due to the holiday season.
Insight - Monthly interactions on Alerts are far lower than those on Incidents. As an example, we can compare the AMAU for users changing the status on alerts from December (38) to the number of users who manually created an incident in December, which was ~2,300. Our hypothesis is two-fold. First, we believe configuring alerts is challenging. We addressing this by making it easier to integrate alerting sources into GitLab by enabling users to customize the endpoints. Second, we know that it is very common for teams to create incidents manually. They do this for two reasons - 1. Many teams do not have performance alerting automation set up and therefore rely on a human reporting a problem and 2. Teams that do have automatic performance alerting still want to create incident manually because they want a human to triage the alert first.
Improvements - We are making it easier to integrate alerting sources into GitLab by enabling users to customize the endpoints.
Improvements - We are adding on-call schedule management, which is a required feature set for Incident Management to compete in market.

Secure:Static Analysis - GMAU - Users running Static Analysis jobs

The highest of the number of unique users who have run SAST or Secret Detection jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10%

Is Primary PI: true

URL(s)

https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_sast_jobs, user_secret_detection_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

On track

Implementation Status: Complete

Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (February):

Despite Januaray data issues, we have recovered from the dip in December usage.
Thrilled to see increase in % of total successful SAST jobs increasing. This suggests that stability improvements and analyzer improvements we've been focusing on are indeed making meaningful impact.
We continue to see [healthy transition to the new Secret Detection template]](https://app.periscopedata.com/app/gitlab/718481/Static-Analysis-Metrics—@tmccaslin?widget=10205713&udv=1090066) and are nearing the end of the transition to the standalone Secret Detection template suggesting we can safely remove in 14.0.

Focus This Month (March):

Now that SAST & Secret Detection is available to all plan types, creating a better experience for non-Ultimate users to help them discover value and become interested in upgrading. Improved MR experience is now complete & Improved Configuration Experience is a focus of 13.10. Both of these have been slowed by complexities with licensing specific features and differences between code structure in EE and CE. However, this should help us maintain our growth rate from SAST to core and create an upgrade path.
Moving forward we are focused strongly on data accuracy and reduction of false positives. We have a new exploration to validate a new vulnerability tracking improvement from Vulnerability Research which could materially reduce false positives. This also opens a new metrics problem that we need a way to track accuracy with real world data. We intend to measure the improvements of this change via the new vulnerabilities data in sisence to ensure this change results in material positive impact before rolling it out.
To increase efficiency and reduce maintenance costs of our 10 linter analyzers we are trialing a transition to SemGrep for our linter analyzers. If successful we will reduce 10 open source analyzers into 1 while also improving the detection rules and proving a more consistent rule management experience across these analyzers, this will be materially impactful as we seek to build our proprietary SAST engine on top of these results.

Secure:Dynamic Analysis - GMAU - Users running DAST

Number of unique users who have run one or more DAST jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10% per-month

Is Primary PI: true

URL(s)

https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_dast_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

On track, despite a drop in usage for January that can be explained as the typical DAST testing cycle.

Implementation Status: Complete

Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (February):

The drop in usage in January is not altogether unexpected after December usage increase. While other scanners had a drop in December, many organizations run DAST tests when active development is slow and not as many people are in the office. Once a DAST test is run, the results can stay valid for an entire development cycle, until there is a workable, deployable app to test. Lower usage after a bump when development is slow is inline with many organizations' DAST testing cycle.
Customers have stopped using DAST in the past because of the huge number of vulnerabilities produced in the dashboard, vulnerability aggregation is the first step in addressing this.
More profile options (specifically authentication) need to be introduced for on-demand DAST scans to have broad usage or to be considered GA.

Focus This Month (March):

Aggregating noisy vulnerabilities to stop DAST from overwhelming the Vulnerability Report (https://gitlab.com/gitlab-org/gitlab/-/issues/254043)
Add more options to Site profile (https://gitlab.com/groups/gitlab-org/-/epics/3771)

Secure:Composition Analysis - GMAU - Users running any SCA scanners

The highest of the number of unique users who have run one or more Dependency Scanning jobs, or number of unique users who have run one or more License Scanning jobs.

SaaS/SM: Both

Target: 2% higher than current month (SaaS and self-managed combined) for All and Paid

Is Primary PI: true

URL(s)

https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_license_management_jobs, user_dependency_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

On track

Implementation Status: Complete

Currently our data is MVC and a lagging indicator, trends are accurate but due to extrapolation of self-hosted, low opt-in of self-hosted, overall missing granularity, and 28 day cycles actual numbers are imprecise, in addition I believe there was another data event (-1's) this month uncertain the impact, but the data is at a point it allows high level trending comparisons across sections and stages.

Lessons Learned (February):

Nothing through use of xMAU Metrics.

Focus This Month (March):

Same as last month, nothing directly related to these lagging indicators, but continue work on auto remediation as part of Dependency Scanning to complete however are blocked by a token issue. We are also still planning to address enabling our containers to work on OpenShift in 13.10 however we are running into some items which may cause ongoing extra maintenance requirements.

Secure:Fuzz Testing - GMAU - Users running fuzz testing

The highest of the number of unique users who have run one or more Fuzz Testing jobs.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >10% per-month

Is Primary PI: true

URL(s)

https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: user_coverage_fuzzing_jobs on self-managed, (options like '%gitlab-cov-fuzz%' OR options like '%FUZZAPI_REPORT%') on .com

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

We have a good growth rate and need more users to use fuzz testing.

Implementation Status: Definition

We now have reporting on .com for both coverage and API fuzz testing.
The issue to more accurately track users was merged recently so we will update the charts to use the new data as it starts coming in.
Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.

Lessons Learned (February):

As part of category maturity scorecard testing and digging into the metrics about success/fail of jobs, we learned our users struggle with documentation. This contributes to the percentage of users who try fuzz testing and are not successful.
More materials are needed to help field staff and customers understand our fuzz testing, the unique value it brings compared to competitors, and how to use it.

Focus This Month (March):

Corpus management for coverage-guided fuzz testing (https://gitlab.com/groups/gitlab-org/-/epics/2915)
Add support for AFL fuzz targets (https://gitlab.com/gitlab-org/gitlab/-/issues/321075)
Improve our documentation based on known pain points (https://gitlab.com/gitlab-org/ux-research/-/issues/1273)
Continue analysis of new data to understand how often and how many users successfully use fuzz testing.
Monitor inflow of new data and update performance indicator charts as needed.
Create additional collateral (slides, recorded videos) to describe our fuzz testing to field staff and customers

Secure:Threat Insights - GMAU - Users interacting with Secure UI

Number of unique sessions viewing a security dashboard, pipeline security report, or expanding MR security report. Same as Paid until OSS projects can be separated.

SaaS/SM: SaaS

Target: Progressively increasing month-over-month, >2% per-month

Is Primary PI: true

URL(s)

https://gitlab.com/gitlab-data/analytics/-/issues/5670

Metric Name: n/a

Chart (Sisense↗)

Health: Okay

Highest monthly GMAU to date, about 10% MoM increase

Implementation Status: Definition

Collection is complete for .com only
Snowplow data is session only, not unique users
Can't currently differentiate between free OSS Gold/Ultimate users and paid Gold/Ultimate users.
Cannot separate out GitLab employee usage from customers

Lessons Learned (February):

Group-level features and MR continue to see highest increases
Comparing Pipeline security total views to those with MR as referrer indicate most traffic from the latter
Opportunity to test if pipeline page has standalone value by moving vulnerability report to MR

Focus This Month (March):

Continue executing on roadmap based on input from other sensing mechanisms
Evaluate shifting focus to more Group-level features based on usage patterns
Gather input on future design vision via Kano analysis and WSJF
Complete Dismissal types / reasons
Complete Bulk updates
Prepare steps 2-4 of MR security widget UX improvement

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

The number of users who have run a container scanning job or interacted with the Protect UI in the last 28 days of the month.

SaaS/SM: Both

Target: Progressively increasing month-over-month, >2%

Is Primary PI: true

Metric Name: user_container_scanning_jobs

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

Usage decreased slightly from December -> January. This was anticipated and expected. See the lessons learned section for more details.
It is probable that the company-wide data collection challenges in January had at least some impact on these numbers as well.
Usage growth is projected to continue to decline in the short-term. See lessons learned section for more details.

Implementation Status: Complete

Limitations in data (see SCA limitations above)
Data is not available yet for the Security Orchestration category as it has not yet reached Minimal maturity

Lessons Learned (February):

As Container Security has not been resourced for some time, we have a pent-up list of feature requests and on-going maintenance work that is presenting a strong headwind to increasing usage.
Usage growth is anticipated to be minimal or negative until we can make a dent in the outstanding usability challenges. Work is anticipated to be completed by March, with usage numbers recovering across the March-June timeframe.
The release of the Security Orchestration category (planned for February) will also bolster the Group's usage numbers once we instrument metrics tracking for the category.

Focus This Month (March):

Alert Dashboard MVC (https://gitlab.com/groups/gitlab-org/-/epics/3438) will be released this month
Project-level DAST Scan Schedule Policies (https://gitlab.com/groups/gitlab-org/-/epics/4598) is being worked on for a March release
Replacing Clair with Trivy (https://gitlab.com/groups/gitlab-org/-/epics/5398) will set the foundation for us to be able to scan containers in production environments

Other Stage PIs Details

Enablement - Section PPI, Stage PPI - Median End User Page Load Time

Median end user page performance collected by Real User Monitoring. This metric captures how well we are doing in serving our primary customers, our end users, by measuring their end user experience.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: performanceTiming

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

We have improved, and are about as fast as our primary competition for the most common pages. (2 faster, 2 slower, 1 effective tie)
We are continuing to improve our performance, with an additional 12 pages under 2.5s LCP since the start of the quarter.

Implementation Status: Target

We are still working to improve our per-geo views.
Need to identify a target.

Lessons Learned (February):

We have made improvements to LCP, however TTI/TBT are still significant gaps vs. our primary competition

Other Group PIs Details

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Of the total number of self-managed installs, what percentage are on one of the three most recent versions.

SaaS/SM: Both

Target: 40%

Is Primary PI: true

URL(s)

Metric Name: versions_behind_latest

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

A target of 40% has been set. 40% is the upgrade rate that was reached in early 2019.
Insights - Continuing to fluctate in the high 20's, the overall trend of increasing age continues.
In order to achieve our target, we will need to reverse the broader trend.

Implementation Status: Complete

Primary PPI is complete, and target is set.

Lessons Learned (February):

There is a continuing trend towards more out of date instances, which has been continuing since 2017. The Postgres 11 upgrade in 13.0 was a brief bump, but did not significantly alter the long term trajectory.
GitLab Paid instances are significently more up to date, but still slowly trending older. The Helm chart is well above target, and when controlling for official Distribution artifacts the metric improves into the 30's.
We have started running a post-install and post-upgrade survey on users experience which has gotten over 500 responses with great feedback. Right now we are at a 6.5/7 rating on upgrade experience, and 4/7 on install. We are working to anaylze the initial feedback now.
We are also working to better highlight to administrators that their instances are out of date.

Enablement:Memory - Group PPI - Memory Consumed

Average memory consumed by all invididual GitLab processes

SaaS/SM: Both

Target: 1.5GB

Is Primary PI: true

Metric Name: topology.nodes[0].node_services

Chart (Sisense↗)

Health: Attention

We are targeting to get below 2GB of total RAM usage, so we need to get these services further below that target.
We've identified high-impact issues to reduce the overall memory footprint. 13.10 will finish up these chanages and benchmark memory consumption in a memory constrained environment.
Memory consumption has stabilized for the last four versions (13.7-13.9)

Implementation Status: Complete

Working to adjust chart to include how many of each process are started by default to better represent a default configuration.
Puma can now run in single-mode in 13.9, which can save up to 200MB of RAM in memory constrained environments.
GitLab exporter settings were reworked in 13.9 which reduced the memory consumption by ~60% (~67-71MB).

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Database query Apdex, with 100ms the target and 250ms tolerable. This measures the ratio of queries which complete within the satisfactory time, informing how well the database is scaling and performing.

SaaS/SM: Both

Target: 0.99

Is Primary PI: true

URL(s)

https://gitlab.com/gitlab-org/gitlab/-/issues/227305

Metric Name: query_apdex_weekly_average

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

Apdex is continuing to exceed the target.
Improvement - We are focusing on addressing a Primary Key overflow risk for tables with an integer PK. This must be completed before the tables run out of space to avoid a hard-stop of GitLab.com (outage of unknown length).

Implementation Status: Complete

The PPI has been instrumented in 13.4.

Lessons Learned (February):

Apdex has improved significantly in 13.7 and 13.8.
Preliminary analysis suggests that caching database queries when using a loadbalancer may be a main reason for improved EE performance.
Apdex on GitLab.com exceeds our group PPI (see 100ms - Tolerable 250ms and 50ms - Tolerable 100ms)

Verify:Runner - Other - Total downloads of runner binaries and docker images per month

Count of downloads of runner binaries grouped by platform, gitlab-runner and gitlab-runner helper container images. Measuring runner binaries' downloads will provide insight into whether we are making the right investments in core runner features and capabilities by platform (operating system and computing architecture). The assumption is that if self-managed users continue to maintain and upgrade their GitLab Runner build infrastructure, this may suggest continued long-term use of GitLab as their DevOps platform.

Funnel Description: Handbook Link

SaaS/SM: SM

Target: TBD

Is Primary PI: true

Metric Name: NA

Health: Unknown

Not instrumented

Implementation Status: Definition

We need to start collecting the AWS S3 bucket metrics and integrate them into the data warehouse. Issue for tracking is in the Infra Project pending scheduling.

Lessons Learned (February):

Insight - Total downloads of 19,400,000 of the gitLab-runner docker image in January 2021 reflects a 4% increase from December 2020, with an average growth in downloads per month from November 2020 to January 2021 at 3.42%. From the current data, we see a continued upward trajectory in the adoption and use of GitLab CI. In thinking about the usefulness of this data, we can posit that a material downward trajectory in downloads could be a leading indicator that we are losing mindshare in the market.

Other PI Pages

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
To update the way each individual (K)PI displays:
- make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
- make changes to the Performance Indicators ERB file for any other properties and its format
When adding a new PI page,
- create the new PI page in the business division's handbook section (like the Engineering Division page)
- include the relative link to the PI page in the Performance Indicators List ERB file
- add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
`name`	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
`base_path`	Required	Relative path to the performance indicator page that this (K)PI should live on
`definition`	Required	refer to Parts of a KPI
`parent`	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
`target`	Required	The target or cap for the (K)PI. Please use `Unknown until we reach maturity level 2` if this is not yet defined. For GMAU, the target should be quarterly.
`org`	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
`section`	Optional	the product section (Ex: dev) as defined in sections.yml
`stage`	Optional	the product stage (Ex: release) as defined in stages.yml
`group`	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
`category`	Optional	the product group (Ex: feature_flags) as defined in categories.yml
`is_key`	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
`health`	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
`health.level`	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI.
`health.reasons`	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
`urls`	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
`funnel`	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
`sisense_data`	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
`sisense_data.chart`	Optional	indicates the numeric Sisense chart/widget ID. For example: `9090628`
`sisense_data.dashboard`	Optional	indicates the numeric Sisense dashboard ID. For example: `634200`
`sisense_data.shared_dashboard`	Optional	indicates the numeric Sisense shared_dashboard ID. For example: `185b8e19-a99e-4718-9aba-96cc5d3ea88b`
`sisense_data.embed`	Optional	indicates the Sisense embed version. For example: `v2`
`sisense_data_secondary`	Optional	allows a second Sisense dashboard to be embeded. Same as `sisense data`
`sisense_data_secondary.chart`	Optional	Same as `sisense_data.chart`
`sisense_data_secondary.dashboard`	Optional	Same as `sisense_data.dashboard`
`sisense_data_secondary.shared_dashboard`	Optional	Same as `sisense_data.shared_dashboard`
`sisense_data_secondary.embed`	Optional	Same as `sisense_data.embed`
`public`	Optional	boolean flag that can be set to `false` where a (K)PI does not meet the public guidelines.
`pi_type`	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
`product_analytics_type`	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
`is_primary`	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
`implementation`	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
`implementation.status`	Optional	indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI.
`implementation.reasons`	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
`lessons`	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
`lessons.learned`	Optional	`learned` is an attribute that can be nested under `lessons`and indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
`monthly_focus`	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
`monthly_focus.goals`	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
`metric_name`	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Each KPI chart should be a timeseries chart.
- Use Purple bars to denote values.
- Use a Red stepped-line for timeseries target.
- Directional targets will be used:
  - Above ...
  - Below ...
  - At ...
  - At or above ...
  - At or below ...
- Optional: Use a single Green bar for the current month (in progress). Preferred for metrics which accumulate through a month but optional due to the performance impact to render the chart.
- Optional: Use a Black line for rolling average.
Each Sisense dashboard for KPIs should consider the following settings to ensure timely updates:
- Setting up auto-refresh for a frequency that fits the KPI
- Excluding Dashboards from Auto Archive
Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
- In Sisense, create a shared dashboard link to get the shared dashboard ID.
- In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
- Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
Multi-series performance indicators should consider the following guidelines:
- If series are mutually exclusive, use stacked bars for each series with a monthly time series
- If series are not mutually exclusive, use grouped bars for each series with a monthly time series
- Do not graph any targets in the chart.
- Current month styling guidelines will not apply
Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :

Stage and Group Performance Indicators

Maintained by:

On this page

SMAU Summary

GMAU Summary

Other Stage PIs Summary

Other Group PIs Summary

SMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Create - SMAU - Unique users writing to a Git repository

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

Package:Package - SMAU, GMAU - Count of users publishing packages

Release:Release - SMAU - Count of users triggering deployments using GitLab

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Sec, Secure - Section MAU, SMAU - Unique users who have used a Secure scanner

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

GMAU PI Details

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

Manage:Access - Paid GMAU - MAU Using Paid SAML

Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features

Manage:Import - GMAU - MAU importing

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Plan:Product Planning - Paid GMAU - MAU creating Epics each Month

Create:Source Code - GMAU - Unique users writing to a project Git repository

Create:Code Review - GMAU - Unique users using Merge Requests

Create:Editor - GMAU - Unique users committing via any web editor

Create:Ecosystem - GMAU - MAU of any integration

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

Verify, Verify:CI - GMAU, SMAU - Unique users triggering pipelines

Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file

Verify:Testing - Paid GMAU - Count of active paid testing feature users

Package:Package - SMAU, GMAU - Count of users publishing packages

Release:Release - SMAU - Count of users triggering deployments using GitLab

Configure:Configure - SMAU, GMAU - Proxy user count based on attached clusters

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Secure:Static Analysis - GMAU - Users running Static Analysis jobs

Secure:Dynamic Analysis - GMAU - Users running DAST

Secure:Composition Analysis - GMAU - Users running any SCA scanners

Secure:Fuzz Testing - GMAU - Users running fuzz testing

Secure:Threat Insights - GMAU - Users interacting with Secure UI

Protect:Container Security - SMAU, GMAU - Users running Container Scanning or interacting with Protect UI

Other Stage PIs Details

Enablement - Section PPI, Stage PPI - Median End User Page Load Time

Other Group PIs Details

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Enablement:Memory - Group PPI - Memory Consumed

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Verify:Runner - Other - Total downloads of runner binaries and docker images per month

Other PI Pages

Legends

Health

How to work with pages like this

Pages

Helper Functions

Data

Guidelines