Dev Section Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Dev Section Performance Indicators

On this page

• Section PI Overview
• Regular Performance Indicators
  • Dev - Section CMAU - Estimated & Recorded Sum of Dev Section SMAU
  • Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features
  • Manage, Manage:Optimize - Error Budget for GitLab.com
  • Manage:Access - Paid GMAU - MAU Using Paid SAML
  • Manage:Access - Error Budget for GitLab.com
  • Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features
  • Manage:Compliance - Other - Projects Marked with a Compliance Framework
  • Manage:Compliance - Error Budget for GitLab.com
  • Manage:Import - GMAU - MAU importing
  • Manage:Import - Other - Share of New Projects Imported
  • Manage:Import - Error Budget for GitLab.com
  • Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues
  • Plan, Plan:Project Management - Error Budget for GitLab.com
  • Plan:Product Planning - Paid GMAU - MAU creating Epics each Month
  • Plan:Product Planning - Error Budget for GitLab.com
  • Plan:Certify - Paid GMAU - MAU Interacting with Requirements
  • Plan:Certify - Other - Number of Requirements Created Monthly
  • Plan:Certify - Error Budget for GitLab.com
  • Create - SMAU - Unique users writing to a Git repository
  • Create:Source Code - GMAU - Unique users writing to a project Git repository
  • Create:Source Code Secondary - Other - GitLab.com performance scores
  • Create:Source Code - Error Budget for GitLab.com
  • Create:Code Review - Category MAU - Unique users using Merge Requests
  • Create:Code Review - Other - Code Review/SCM Ratio by UMAU and Subscription Tier
  • Create:Code Review - Editor Extension (VS Code) - Category MAU
  • Create:Code Review - Error Budget for GitLab.com
  • Create:Gitaly - Other - Availability
  • Create:Gitaly - Other - Unique instances with Gitaly Cluster enabled
  • Create:Gitaly - Error Budget for GitLab.com
  • Create:Editor - GMAU - Unique users committing via any web editor
  • Create:Editor - AMAU - Unique users committing by web editor
  • Create:Editor - Other - Number of Commits per Month from the Web IDE
  • Create:Editor - Other - Number of new Snippets per Month
  • Create:Editor - AMAU - Unique users committing via the Static Site Editor
  • Create:Editor - AMAU - Uniques users of Wiki
  • Create:Editor - Error Budget for GitLab.com
  • Create:Ecosystem - GMAU - MAU of any integration
  • Create:Ecosystem - Other - Total active project integrations
  • Create:Ecosystem - Other - Integrations managed at Group or Instance level
  • Create:Ecosystem - Error Budget for GitLab.com
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Note: This page is dynamically generated via the Dev Section performance indicators .yml file.

Section PI Overview

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Dev	Section CMAU	Estimated & Recorded Sum of Dev Section SMAU	Okay	Both	Complete
Manage, Manage:Optimize	SMAU, GMAU	MAU Viewing Analytics Features	Okay	Both	Complete
Manage:Access	Paid GMAU	MAU Using Paid SAML	Okay	SaaS	Complete
Manage:Compliance	Paid GMAU	MAU Viewing Compliance Features	Attention	Both	Complete
Manage:Import	GMAU	MAU importing	Okay	Both	Complete
Plan, Plan:Project Management	SMAU, Paid GMAU	MAU Interacting With Issues	Okay	Both	Complete
Plan:Product Planning	Paid GMAU	MAU creating Epics each Month	Okay	Both	Complete
Create	SMAU	Unique users writing to a Git repository	Okay	Both	Complete
Create:Source Code	GMAU	Unique users writing to a project Git repository	Okay	Both	Complete
Create:Code Review	Category MAU	Unique users using Merge Requests	Okay	Both	Complete
Create:Gitaly	Other	Availability	Unknown	Both	Instrumentation
Create:Editor	GMAU	Unique users committing via any web editor	Okay	Both	Complete
Create:Ecosystem	GMAU	MAU of any integration	Unknown	Both	Instrumentation

Regular Performance Indicators

Dev - Section CMAU - Estimated & Recorded Sum of Dev Section SMAU

A sum of all SMAUs across the three stages in the Dev Section (Manage, Plan, and Create) for All users and Paid users.

SaaS/SM: Both

Target: FY22Q1 -- Section CMAU growth 4% MoM with a goal of 3.13M for Estimated CMAU & a goal of 775K for Estimated Paid CMAU

Is Primary PI: true

Metric Name: issues, action_monthly_active_users_project_repo, analytics_unique_visits_for_any_target

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• CMAU is fully instrumented
• Dev Section recorded its highest Estimated CMAU at 3.04M!
• Dev Section recorded its highest Estimated Paid CMAU at 754K!
• All Dev Stages have now rebounded from holiday season drop

Implementation Status: Complete

Focus This Month (May):

• Work with Data team to better calculate "Estimated Self-Managed Uplift" to help balance out calculations

Manage, Manage:Optimize - SMAU, GMAU - MAU Viewing Analytics Features

The number of unique users viewing any Analytics page.

SaaS/SM: Both

Target: 116050

Is Primary PI: true

Metric Name: analytics_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• Insights - Growth in page views by SaaS users was relatively flat. This is not surprising because feature announcements for February were focused on the user availability feature. However, SaaS views of our key feature, Value Stream Analytics, continued to grow steadily with 12% growth at the group level, possibly attributable to the 13.10 announcement of UX improvements.
• Insights - There continues to be strong interest in Value Stream Analytics. Usage of the group-level feature is still low compared to the project-level feature, presumably because of the tier difference (available in Core at the project level and Premium at the group level). However, usage of the group-level feature continues to grow with an 11% increase in views by self-managed Premium and Ultimate customers from February to March.
• Insights - MR Analytics page views continued to climb after the 13.9 announcement of the Mean Time to Merge metric with another 17% increase in views from self-managed Premium customers and 30% from Ultimate. In the previous PI review we announced that views of this page by Premium customers (self managed) increased 78% from December to February.
• Improvements - The DevOps Adoption feature that was previously only available to self-managed admins is being added at the group level in 13.11, expanding availability to all Ultimate users.
• SaaS data is not available for the paid SMAU chart due to limitations of the redis counter that is used.

Implementation Status: Complete

Manage, Manage:Optimize - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Manage:Access - Paid GMAU - MAU Using Paid SAML

The number of unique users logging into GitLab using SAML SSO for Groups.

SaaS/SM: SaaS

Target: 61000. See https://gitlab.com/gitlab-com/Product/-/issues/2258 for further detail.

Is Primary PI: true

Metric Name: user_auth_by_provider (Group_SAML)

Chart (Sisense↗)

Health: Okay

• Insights - We introduced the Minimal Access role in 13.5 and option to configure the default role for users added through SAML SSO or SCIM in 13.3. Our data shows that half of SAML SSO group owners are changing the default role and Minimal Access is the second most commonly used role! This is further confirmation of customer apetite for improvements to Group SAML SSO and features that allow for more fine-grained authorization.
• Insights - I found that less than half of the groups that have SAML SSO enabled are enforcing it. This means that while you can access a group through SSO, you are not required to have an active SAML SSO session and you can access the group through other authentication methods. Our documentation has been updated and I sent a message in Slack to the Customer Success team to reinforce that SAML SSO enforcement is a security best practice. In 13.12, we will start enforcing SAML SSO by default for new groups that configure SAML. Enforcing SAML SSO was intially not the default behavior due to risk of accidental lockout due to a misconfiguration. In 13.8, we made a change to always allow group owners into the group to mitigate this risk. After the default behavior is changed, I will re-measure to determine if not enforcing SAML SSO was a deliberate choice or customers were simply keeping the default option.
• Improvements - We now enforce Group SAML SSO for Git activity. This will provide the same level of protection to Git as the Web UI and been requested by several large customers and prospects.
• Improvements - Enterprise customers using SAML want to continue down the path of their IdP being the source of truth for GitLab authentication and authorization. Our long-term roadmap includes enforcing SSO for API activity, disabling password authentication for users created by an external IdP and iterating on capabilities to set user attributes using SAML to increase group isolation.

Implementation Status: Complete

Manage:Access - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Manage:Compliance - Paid GMAU - MAU Viewing Compliance Features

The number of unique users viewing Audit Events, Compliance Dashboard, or Credential Inventory.

SaaS/SM: Both

Target: 10% quarter-over-quarter

Is Primary PI: true

Metric Name: i_compliance_audit_events, g_compliance_audit_events, g_compliance_dashboard, i_compliance_credential_inventory, usage_data_g_compliance_dashboard, usage_data_a_compliance_audit_events_api

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Attention

• We're still building out the compliance features that are tracked by this metric so we expect trends may change in the future.
• We are biasing for action and setting a target with the information we have, rather than waiting. We expect this may need to be adjusted in the future.

Implementation Status: Complete

• Completed

Lessons Learned (April):

• We have seen across-the-board growth this month in all our individual metrics, which indicates teams are starting to evaluate or adopt our capabilities more.
• We have repeatedly heard that actionability is important from users. They don't just want info, they want info they could use to make decisions.

Focus This Month (May):

• Re-factoring of audit events to prepare for adding new events in the future
• We saw a large growth in self-managed users compared to .com users. This could be because more of our instance-specific features are getting adoption, from .com data integrity issues, or from something else. We will need to keep investigating.

Manage:Compliance - Other - Projects Marked with a Compliance Framework

New and cumulative number of projects using compliance frameworks.

SaaS/SM: Both

Target: 10% quarter-over-quarter growth in new projects using compliance frameworks (SaaS and self-managed combined)

Metric Name: projects_with_compliance_framework

Chart (Sisense↗)

Health: Attention

• In process

Implementation Status: Complete

Lessons Learned (April):

• We saw a 17% month-over-month growth in compliance framework labels usage.
• From customer conversations we have learned that compliance framework labels are very appealing to users but they need functionality driven off of those labels, not just labeling.

Focus This Month (May):

• We're continuing iteration on compliance frameworks to give them more actionability and solve several standardization and enforcement problems for customers.
• Since we will be releasing custom compliance frameworks as well as compliance pipelines, we expect to see an even greater growth rate in the next few months.

Manage:Compliance - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Manage:Import - GMAU - MAU importing

A monthly count of the unique number of users performing an import

SaaS/SM: Both

Target: 96,000 MAU importing projects

Is Primary PI: true

Metric Name: usage_activity_by_stage_monthly.manage.unique_users_all_imports

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• Target - Initial target set to 96,000, which is a 20% increase from the current user engagement level.
• Insights - A slight drop in the Saas numbers from March to April was offset by a slight increase in the self-managed numbers.
• Improvement - Reducing friction for GitLab-to-GitLab migrations - Our progress toward achieving feature parity with the existing Group Export/Import has been significantly slowed down, due to a focus on infradev performance improvements and support for migrating new customers.
• Note - SaaS data is not available for the paid GMAU chart due to limitations of the redis counter used.

Implementation Status: Complete

• Instrumentation has been implemented in 13.9.
• The charts are completed and data is available starting with February 2021.

Manage:Import - Other - Share of New Projects Imported

Share of projects created via import within 90 days of the namespace's creation.

SaaS/SM: Both

Target: TBD - See Health section below for details

Metric Name: N/A

Chart (Sisense↗)

Health: Unknown

• No target defined yet.
• Target will be defined once the dashboard shows both Saas and self-managed.

Implementation Status: Instrumentation

• Instrumented for Saas.
• Dashboard shows only Saas at this time.
• Instrumentation for self-managed pending. Tentative target is 14.0.
• Dashboard will be updated with Self-Managed data, once instrumented.
• Insight - The sharp increase in the number of new Projects created in March, likely from a large customer migration, is what caused the GMAU "drop" in April, as we settled back into more normal activity levels.

Manage:Import - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Plan, Plan:Project Management - SMAU, Paid GMAU - MAU Interacting With Issues

Number of unique users interacting with an issue (open, close, reopen, adjust label, adjust milestone, adjust weight, comment)

SaaS/SM: Both

Target: 484,510 for All, 148,176 for Paid

Is Primary PI: true

URL(s)

• https://about.gitlab.com/direction/plan/issue_tracking/
• https://app.periscopedata.com/app/gitlab/775807/Plan-xMAU-(monthly)

Metric Name: redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• FY22 Q1 Goals - Currently at 103% attainment for All and 96% for Paid with one reporting period remaining in FY22 Q1. New targets will be set for FY22 Q2.
• Driver - For Paid, continue high level progress on achieving feature parity between an Instance, Group, and Project, making issues more extensible, and continuing to make incremental progress on iterations.
• Improvement - For All, work towards addressing the problem areas that emerged out of the Category Maturity Scorecard exercise for Issue Tracking.
• Improvement - For All, continue making progress on quality of life improvements for existing features.

Implementation Status: Complete

Lessons Learned (April):

• Only 39% of instances are currently on a version with the new Plan xMAU metric counters.
• March saw 2.2 net new Paid GMAU for every 1 churned Paid GMAU, which is a decline of a full point from February. (Self-Managed Only)
• Starter EoL Impact - Net increase of 239 MAU (1.3%) from February to March for Starter on Self-Managed due to instances upgrading to > 13.5. This is a significant decline in growth which saw 10.7% increases from January to February. For SaaS, loss of 507 Starter MAU (-5.6%) from February to March. The blended growth rate for Premium and Ultimate across SaaS and Self-Managed was 11.6% from February to March, which was the lowest in the trailing three months with February averaging 15.2% and January 12.4%.
• March saw a decline of 7.6% in overall instances reporting usage pings (39% down to 36%), which may correlate to perceived growth slowing down a bit in Plan.

Focus This Month (May):

• Shift focus from 13.11 feature goals to meet immediate performance needs for SaaS. This will likely carry on into 13.12 and beyond given sheer size of internal requests in our stage (infradev 5, availability 2, performance-refinement 10, security 52). The bulk of the Plan capabilities roadmap to drive xMAU now falls within the 8th, 9th and 11th positions on our prioritization matrix. This is likely to significantly impact future MAU performance for Plan.
• Validate the feasibility of a "feature freeze" starting in 14.1 in order to fix long running instabilities and technical/product debt within our span of control, as well as increase our future throughput on executing on our longer term product strategy.
• Merge architectural blueprint for consolidating Groups and Projects on the backend.

Plan, Plan:Project Management - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Plan:Product Planning - Paid GMAU - MAU creating Epics each Month

The number of unique users creating Epics each month.

SaaS/SM: Both

Target: 3,900 Combined Unique Users for Self Managed and SaaS

Is Primary PI: true

Metric Name: epics

Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• March 2021 - 5333 GMAU goes well above the current target, and we continue to instrument all epic board interactions to set a new target
• Insight - Crossed 100k cumulative epics created, with back-to-back months of 40% growth in February and March
• Improvement - Make it easier to add epics by defaulting project in 13.11
• Improvement - Epic Boards MVC 2 is planned to ship in 13.12 with a full roll out in 14.10 and should continue the trend into the end of FY21 by enabling Portfolio-level planning.

Implementation Status: Complete

• We consider our Epic Creation GMAU as "Complete". However, our next target will be set after a second instrumentation after 14.0 and epic boards release.
• Currently working on instrumenting "all epic interactions" during 13.9 - 13.12. Instrumentation issue and dasboard issue.
• We have added instrumentation for issue and epic boards
• A new target will be selected when we have a baseline.

Plan:Product Planning - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Plan:Certify - Paid GMAU - MAU Interacting with Requirements

The number of unique users interacting (create, view, reference) each month

SaaS/SM: Both

Target: 2000 interactions

Metric Name: requirements_interactions

Chart (Sisense↗)

Health: Attention

• Note - Data is currently showing those requirements with test reports (both manual and automated). Will work with the data team to sum with the navigation interactions.
• Improvement - Requirements in the process of being converted to issue type, which will have the added benefit of improved tracking using redis counters.
• Insight - 90 Day rolling average page views for requirements increased from 45k to 57k, showing an increased interest in requirements management.

Implementation Status: Dashboard

• Dashboarding - Requirement Creation instrumented is complete. Tracking issue for gmau dashboard here

Plan:Certify - Other - Number of Requirements Created Monthly

A monthly count of the number of requirements created.

SaaS/SM: Both

Target: 1000 by end of March 2021

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/5270

Metric Name: requirements_created

Chart (Sisense↗)

Health: Okay

• Improvement - Link Test Cases to Requirements planned for 14.0
• Insight - Self-managed instances creating requirements continues to grow.

Implementation Status: Dashboard

• Dashboarding - accurate instrumentation on Self Managed, still needs verification for SaaS - issue

Plan:Certify - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Okay

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Create - SMAU - Unique users writing to a Git repository

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• Observation - December 2020 SMAU is less than November 2020 SMAU. A similar drop can be observed in GMAU as well, suggesting a common cause. The cause is likely seasonality consistent with the end of year. For this reason January SMAU and GMAU is expected to return to November 2020 levels and continue growing.
• Insight - on the median GitLab instance, 33% of users perform a Git write operation each month (source).
• Insight - on the median Paid GitLab instance (self-managed), 50% of users perform a Git write operation each month (source).

Implementation Status: Complete

• Improvement - AMAU is currently over counted. A deduplicated counter was implemented gitlab#234027 in GitLab 13.6. The charts do not yet reflect the new counter gitlab#291943.

Create:Source Code - GMAU - Unique users writing to a project Git repository

A rolling count of the number of unique users who performed any Git write operation within the last 28 days.
Why: We want to ensure all the features in our group provide a great experience that ultimately will allow everyone to contribute more often. A great experience in SCM is a critical starting point for this.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.
- Q1FY22 targets - 2.44M GMAU, 590k paid GMAU
- Q2FY22 targets - 2.67M GMAU, 644k paid GMAU
- Q3FY22 targets - 2.92M GMAU, 704k paid GMAU
- Q4FY22 targets - 3.19M GMAU, 769k paid GMAU

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/728380/WIP-Create-Stage-Dashboard?widget=9520571&udv=0

Metric Name: action_monthly_active_users_project_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• Starting in GitLab 13.6 AMAU for this metric takes into consideration all repository types.
• Insight - GMAU has increased avg of 7% MoM in FY22Q1 (against goal of 3%) compared to 3% MoM in FY20Q4.
• Insight - Paid GMAU has increased avg of 5% MoM in FY22Q1 (against goal of 4%) compared to 2% in FY20Q4.
• We continue to use this as the group's primary indicator. In order to align with our SaaS first and product depth direction, during FY22 we will also have an increased focus around performance as our secondary PI.

Implementation Status: Complete

• Enabled on GitLab.com
• Released for self-managed in GitLab 13.3

Create:Source Code Secondary - Other - GitLab.com performance scores

Performance of the top 3 backend services for Source Code (apdex), and its related frontend counterparts (LCP) showing improvememnts over time.
Why: Aligning with our SaaS first and product depth direction, we recognize the performance of GitLab.com is a large factor considered when evaluating GitLab. We firmly believe speed is the killer feature and as such will work to provide a speedy experience to set a great stage for new and existing users.

Target: TBD

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/327659
• https://dashboards.gitlab.net/d/stage-groups-source_code/stage-groups-group-dashboard-create-source-code?orgId=1&from=1618343400000&to=1618365059999&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-controller=All&var-action=All&viewPanel=12
• https://dashboards.gitlab.net/d/stage-groups-source_code/stage-groups-group-dashboard-create-source-code?orgId=1

Health: Unknown

• Insight - total time taken to process pull mirror operations has steadily increased. We will re-evaluate its implementation to enhance or refactor.
• Insight - recent performance improvements have decreased LCP of commits view by ~20% 💪
• We continue to investigate which are the most relevant measures to track and how to aggregate sensibly. We will measure apdex for the top 3 services used (BE) and LCP for its corresponding application pages (FE). This is currently WIP, see investigation issue.

Implementation Status: Instrumentation

• The Create:Source Code group is investigating which of the many measures currently instrumented for GitLab.com are suitable for dual apdex and LCP Tracking. Once selection is complete we will work to bring into a single dashboard and set targets.

Create:Source Code - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Create:Code Review - Category MAU - Unique users using Merge Requests

This is defined as users who perform one or more of the following actions in a merge request - create, merge, or close a Merge Request.

SaaS/SM: Both

Target: Our target growth rate is 3% higher than the previous month for SMAU, and 4% for Paid SMAU.

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/526749/Create---Code-Review-category-dashboard

Metric Name: merge_requests_users

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• Target - This aligns with our current understanding that Code Review is about 50% of the MAU of the stage metric and continuing to maintain that difference.
• Insight - Paid users of Code Review make up about one-third of total Code Review users - this provides an opportunity to convert to paid and increase users.

Implementation Status: Complete

• Enabled on GitLab.com 2020-07-04
• Enabled for self-managed on GitLab 13.2
• Aggregated metrics for new AMAU tracking implemented in %13.10, expect to start reporting around 14.0

Create:Code Review - Other - Code Review/SCM Ratio by UMAU and Subscription Tier

SaaS/SM: Both

Target: TBD - Primarily this metric is used for monitoring the relationship between SCM GMAU and Code Review GMAU to make sure it doesn't not fall out of alignment.

URL(s)

• https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify
• https://app.periscopedata.com/app/gitlab/801102/SCM-%3C%3E-Code-Review-%3C%3E-Verify?widget=10728688&udv=1218043

Metric Name: merge_requests_users

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Unknown

• Target - this metric is primarily monitored to remain at a consistent level.
• Insight - Paid Instances with 10+ users more closely track to 75% of SCM Users using Code Review
• Insight - Free Instances and those with less than 10 users drive down the SCM<->Code Review Ratio as most have less than 50% adoption

Implementation Status: Complete

Create:Code Review - Editor Extension (VS Code) - Category MAU

This is defined as users who use [GitLab Workflow](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) for VS Code and make at least one API request to a GitLab instance.

SaaS/SM: Both

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/825329/Editor-Extension-(VS-Code)-MAU-Metrics

Chart (Sisense↗)

Health: Unknown

• Improvement - GitLab Workflow 3.17.0 released with responding to threads in MR diffs and change type for files
• Improvement - Averaging about 3 releases per month with 2 of those usually containing new features
• Improvement - Cloning directly to VS Code released in main GitLab application
• Insight - Early self-managed reporting trending upwards. Expect to see continued growth on the self-managed number as instances upgrade to >=13.9.
• Insight - Lots of opportunity to continue to increase adoption for GitLab.com users based on current MAU compared to total SCM/Code Review users.

Implementation Status: Data Availability

• Instrumentation in 13.9, comprehensive Category MAU not charts not available yet

Create:Code Review - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Create:Gitaly - Other - Availability

Percentage of time during which Gitaly is fully operational and providing service to users within SLO parameters. Availability is measured according by Apdex (latency) and error rate. Gitaly is considered available when at least 50% of users are experiencing satisfactory latency, and 50% of requests are completing successfully. If either of these conditions is not met, the service is experiencing an outage.

SaaS/SM: Both

Target: 99.95%, consistent with GitLab.com availability service level objectives

Is Primary PI: true

URL(s)

• https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview
• https://gitlab.com/gitlab-org/gitlab/-/issues/216683

Metric Name: gitaly_apdex

Health: Unknown

• Instrumentation for this metric started in 13.4 and is still ongoing. Currently utilizing SaaS metrics via Grafana.

Implementation Status: Instrumentation

• Rolling 7 day metrics available in Grafana for GitLab.com. Current availability per Adpex is 99.98%.
• Improvement - self-managed availability metrics are planned, but not currently in the backlog due to technical limitations.

Create:Gitaly - Other - Unique instances with Gitaly Cluster enabled

Number of recorded GitLab instances with Gitaly Cluster enabled

SaaS/SM: Self-Managed

Target: 75 Paid Customers using Gitaly Cluster

URL(s)

• https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview

Metric Name: gitaly_clusters

Chart (Sisense↗)

Health: Okay

• Insight - Exceeded target of 50 paid instances in April with a total of 54! Raising target to 75 paid customers using Gitaly Cluster.
• Insight - Ultimate usage tripled in April
• Seeing rising interest as NFS Support is deprecated in 14.0
• Improvement - Updated instructions for migration to Gitaly Cluster
• Improvement - introduce Variable Replication Factor and auto-reblancing within a cluster to enable more enterprise-level adoption.

Implementation Status: Complete

• Instrumentation for reporting on instances with Gitaly Cluster enabled was introduced in 13.0 with the GA of Gitaly Cluster.

Create:Gitaly - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Okay

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Create:Editor - GMAU - Unique users committing via any web editor

A rolling count of the number of unique users who performed a Git write operation to any repository using the Web IDE, Web Editor or Snippet within a stage in the last 28 days

SaaS/SM: Both

Target: 4% growth month over month for All

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU

Metric Name: action_monthly_active_users_ide_edit

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

Health: Okay

• Insight - Healthy growth, particularly in self managed, likely due to adoption of versions containing analytics and organic growth.
• Improvement - The Web IDE was updated in 13.9 to use Editor Lite which brings consistency to the underlying editors powering Snippets, Single-file Editor, and Web IDE as well as a moderate performance improvement.

Implementation Status: Complete

• Instrumentation of Static Site Editor and Wiki to roll up into GMAU completed in 13.7, waiting for data to be reflected in the charts.

Create:Editor - AMAU - Unique users committing by web editor

A rolling count of the number of unique users who performed a Git write operation to any respository using the Web IDE, Web Editor or Snippet within a stage in the last 28 days - users grouped by editor used

SaaS/SM: Both

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/741875/Create:Editor-xMAU

Metric Name: action_monthly_active_users_web_ide_edit, action_monthly_active_users_sfe_edit, action_monthly_active_users_snippet_editor_edit

Chart (Sisense↗)

Health: Unknown

• Insight - Editing through the Web Editor (single file editor) is 2-to-1 for users of the Web IDE though the gap is narrowing slightly over time. This confirms data recognized previously based on navigation paths that the single file editor is more widely used. We need to continue monitoring this to determine future investments in the Web IDE.
• Note - Important to call out that these are not unique users, so the same user can be reported as using multiple web editors.

Implementation Status: Completed Dashboard

• Instrumentation is in place for both self-managed and SaaS.
• Updating dashboards to for Paid / All breakouts

Create:Editor - Other - Number of Commits per Month from the Web IDE

The total number of commits made via the Web IDE.

SaaS/SM: Both

Target: Maintain organic growth, 650K by end of Q1 2021

URL(s)

• https://app.periscopedata.com/app/gitlab/723366/Create:Editor-Expanded-for-Embed

Metric Name: web_ide_commits

Chart (Sisense↗)

Health: Okay

• Insight - Investigation into the February spike is ongoing but it doesn't seem to be explained entirely by the data incident with SaaS reporting in January/February.
• Insight - Ignoring the February outlier, growth is still strong and we have exceeded our goal for Q1 2021.

Implementation Status: Complete

• Instrumentation working and charts updated to include both self-managed and SaaS data.
• Updating dashboard for Paid / All breakout.

Create:Editor - Other - Number of new Snippets per Month

The number of new Snippets created each month.

SaaS/SM: Both

Target: 4% Growth month-over-month to 82k Snippets per Month by Q2 2021

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/4400

Metric Name: snippets_change

Chart (Sisense↗)

Health: Okay

• Insight - Growth is back on track and we'll monitor it to make sure it doesn't drop again.
• Insight - Self-Managed continues to account for about 8x the number of snippets. Investigation into the user behavior here is warranted but it's likely due to the nature of snippets visibility and permissions. Self-Managed users may be more comfortable creating snippets for their instance.
• Improvements - In 13.10 we are increasing the visibility of the multi-file snippet feature by showing the count of files for each snippet in the list view.
• Improvement - snippets.gitlab.com now redirects to your personal snippets page for easier access and improved visibility.

Implementation Status: Complete

• Instrumentation for number of snippets is complete for self-managed and SaaS.

Create:Editor - AMAU - Unique users committing via the Static Site Editor

Monthly unique users that have made at least one commit via the static site editor.

SaaS/SM: Both

Target: 150 AMAU by June 30, 2021

URL(s)

• https://app.periscopedata.com/app/gitlab/684213/WIP-Static-Site-Editor-Usage-Metrics
• https://about.gitlab.com/direction/create/static_site_editor/

Metric Name: static_site_editor_create_commit

Secondary Chart (Sisense↗)

Health: Attention

• Insight - Decline in AMAU consistent with the deprioritization of feature development.

Implementation Status: Complete

• Improvement - Instrumentation for Usage Ping completed in 13.7

Create:Editor - AMAU - Uniques users of Wiki

Monthly unique users that have viewed or edited a wiki

SaaS/SM: Both

Target: 10% higher than current month (SaaS and self-managed combined)

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/230481

Metric Name: action_monthly_active_users_wiki_repo

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Okay

• No target defined

Implementation Status: Complete

• Improvement - Slightly lower growth month over month but that's consistent with the lack of feature work. We'll be monitoring it in the coming months to see if it starts to drop.
• Improvement - Planning to slice the data to show how much group-level Wikis are contributing to growth.
• Insight - Wiki growth continues to be strong, stronger than anticipated actually. We're investigating to make sure the data is accurate.
• Improvement - Beginning work on a WYSIWYG/visual markdown editor for Wiki, targeting a MVC available in 14.0.

Create:Editor - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Create:Ecosystem - GMAU - MAU of any integration

Monthly Unique Users of any GitLab integration.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/5181
• https://gitlab.com/gitlab-org/gitlab/-/issues/297401
• https://gitlab.com/gitlab-data/analytics/-/issues/7974

Metric Name: TBD

Health: Unknown

• No target defined
• Improvement - Adding Group and Instance integration will allow larger sets of projects to be integrated easily
• Improvement - Improvements and new functionality in the Jira integration should increase adoption, expanding on the success of our most popular integration

Implementation Status: Instrumentation

• Awaiting implementation of data in Usage Ping
• Improvement - (March 2021) Updates are in progress, GMAU tracking is implemented and data is starting to flow in, will be on the dashboard once updates complete
• Awaiting implementation of Sisense dashboard (blocked by Usage Ping implementation)

Create:Ecosystem - Other - Total active project integrations

The total number of active project service integrations across all usage ping data (which includes GitLab.com)

SaaS/SM: Both

Target: Maintain flat adoption rate relative to overall GitLab growth

Metric Name: projects_asana_active, projects_assembla_active, projects_bamboo_active, projects_bugzilla_active, projects_buildkite_active, projects_campfire_active, projects_confluence_active, projects_custom_issue_tracker_active, projects_discord_active, projects_drone_ci_active, projects_emails_on_push_active, projects_external_wiki_active, projects_flowdock_active, projects_github_active, projects_gitlab_slack_application_active, projects_hangouts_chat_active, projects_hipchat_active, projects_irker_active, projects_jenkins_active, projects_jira_active, projects_jira_cloud_active, projects_jira_dvcs_cloud_active, projects_jira_dvcs_server_active, projects_jira_issuelist_active, projects_jira_server_active, projects_mattermost_active, projects_mattermost_slash_commands_active, projects_microsoft_teams_active, projects_packagist_active, projects_pipelines_email_active, projects_pivotaltracker_active, projects_prometheus_active, projects_pushover_active, projects_redmine_active, projects_slack_active, projects_slack_notifications_active, projects_slack_slash_active, projects_slack_slash_commands_active, projects_teamcity_active, projects_unify_circuit_active, projects_webex_teams_active

Chart (Sisense↗)

Health: Okay

• Observation - Growth tracks fairly evenly against overall usage growth (total number of projects created), demonstrating flat relative adoption rate of integrations
• Observation - GitLab.com users integrate at ~1/3 the rate that self-managed users do

Implementation Status: Dashboard

• Good overall tracking of project integrations, detailed breakdown available per-integration
• Updating dashboards to for Paid / All breakouts

Create:Ecosystem - Other - Integrations managed at Group or Instance level

The total number of active project integrations that are being defined by a Group or Instance integration

SaaS/SM: Both

Target: Not defined

URL(s)

• https://gitlab.com/gitlab-data/analytics/-/issues/7974

Metric Name: TBD

Health: Unknown

• No data so far

Implementation Status: Dashboard

• Improvement - (March 2021) Dashboard update in progress that will add this data to the dashboard.

Create:Ecosystem - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Protect Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Talent Acquisition Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.