Enablement Section Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Enablement Section Performance Indicators

On this page

• Section PI Overview
• Regular Performance Indicators
  • Enablement - Section PPI, Stage PPI - Median End User Page Load Time
  • Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab
  • Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary
  • Enablement:Geo - Paid GMAU - Number of potential Geo users
  • Enablement:Geo - Error Budget for GitLab.com
  • Enablement:Memory - Group PPI - Memory Consumed
  • Enablement:Global Search - Paid GMAU - The number of unique paid users per month
  • Enablement:Global Search - Error Budget for GitLab.com
  • Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)
  • Enablement:Database - Error Budget for GitLab.com
  • Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Section PI Overview

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement	Section PPI, Stage PPI	Median End User Page Load Time	Attention	Both	Complete
Enablement:Distribution	Group PPI	Percentage of installations on the 3 most recent versions of GitLab	Attention	Both	Complete
Enablement:Geo	Paid GMAU	Number of unique users utilizing a Geo secondary	Attention	Both	Instrumentation
Enablement:Geo	Paid GMAU	Number of potential Geo users	Attention	Both	Instrumentation
Enablement:Memory	Group PPI	Memory Consumed	Attention	Both	Complete
Enablement:Global Search	Paid GMAU	The number of unique paid users per month	Okay	Both	Complete
Enablement:Database	Group PPI	Database Query Apdex (100ms target, 250ms tolerable)	Attention	Both	Complete
Enablement:Infrastructure	Paid GMAU	Number of unique paid users that perform an action on GitLab.com	Okay	SaaS	Complete

Regular Performance Indicators

Enablement - Section PPI, Stage PPI - Median End User Page Load Time

Median end user page performance collected by Real User Monitoring. This metric captures how well we are doing in serving our primary customers, our end users, by measuring their end user experience.

SaaS/SM: Both

Target: 1.3

Is Primary PI: true

Health: Attention

• Median page load times have not changed since April, however the outliers (p90 and p95) have slightly decreased.
• Now Median and p90 load times by country by week are available.

Metric Name: performanceTiming

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• We are working to directly tie page load data to routes.

Lessons Learned (July):

• Page load times are not significantly affected by geography, with median times of 1.1s for US, 1.2 for most of Europe, and 1.3 for Australia. It seems internet latency is not a significant driver of overall performance for .com, however pages with many async requests will see higher impact.
• We are working to directly tie page load data to routes, with some of this content already sent but not processed in our data pipeline.
• We are also working to include RUM data (frontend performance) in error budgets, but pulling it into Grafana has proven more difficult than originally thought. We will need to update our lambda to push to Prometheus.
• Also exploring a CI test suite to catch regressions before they make it to production.

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Of the total number of self-managed installs, what percentage are on one of the three most recent versions.

SaaS/SM: Both

Target: 40%

Is Primary PI: true

Health: Attention

• A target of 40% has been set. 40% is the upgrade rate that was reached in early 2019.
• Insights - % on latest 3 versions has been stable, median age of version has returned to increasing trend.
• More research is required to understand reasoning for jump in age across all versions and editions.

Metric Name: versions_behind_latest

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10278985&udv=1102166
• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10279895&udv=1102166

Implementation Status: Complete

• Primary PPI is complete, and target is set.

Lessons Learned (July):

• The most recent update to the median age of license shows an increase across the board. This will continue to be investigated and discussed to see how we can trend downward again. A new research topic to explore is, how can we prompt community edition users to upgrade more regularly, since they are notably the largest median age.
• Security was a central focus for admins to upgrade their version from the ease of upgrading study. The inital MVC has been merged, to prompt users to sign up for the security newsletter in the next steps docs after installation. The next iteration will be to prompt current users to sign up for the security newsletter in the admin console.
• Another important aspect for admins to upgrade was "when developers are asleep", or less active within an organization. The MVC will be to display this activity to admins, and showcase the time of least activity.
• Work has begun on better highlighting when instances are out of date to administrators. This will impact results once it has been completed, estimated for this milestone.
• Based on the ease of upgrading study admins also were open to being notified more prominently when an upgrade was available. Work has begun on creating MVC's for a few new ways to notify admins in their preferred communication channel, that an upgrade is available.

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Number of unique users utilizing a Geo secondary. This adoption metric helps us understand whether end users are actually seeing value in, and are using, geo secondaries.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Attention

• We've enabled gathering usage data from Geo secondaries and have added git fetch metrics to the usage data. These metrics are not yet available; we are working on getting them plotted in Sisense. We also intend to add tracking for git push operations on a Geo secondary. This work is currently paused due to capacity constraints.
• GMAU is currently based on logging into the Secondary web interface. Git access is more common and we will begin tracking Git pulls in 13.11 with tracking for Git pushes planned as part of the Geo Usage Ping Epic. We also know that the UX of the secondary web interface is not good and we want to remove it, see Opportunity Canvas(internal). In order to assess the impact of our planned changes, having this (low) GMAU is really important. We've recently completed a PoC to make the secondary UI indistinguishable from the primary. Implementation work is in progress and can be tracked in the secondary mimicry epic.

Metric Name: geo_nodes

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Implementation Status: Instrumentation

• Geo is not available on GitLab.com today, so cannot use Snowplow or the .com database.

Lessons Learned (July):

• The number of users is still very low, likely because the WebUI is insufficient. We are planning to change this, see this opportunity canvas and in-progress implementation work in the secondary mimicry epic.

Enablement:Geo - Paid GMAU - Number of potential Geo users

Sum of all users who are using instances with Geo enabled. All these users are positively impacted due to Geo in case of a failover or restore from backup.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Attention

• For Disaster Recovery we are measuring potential Geo users - the number of active licenses. In an ideal world, no regular user would ever need to rely on Geo because there is no disaster, but if one occurs everyone benefits. Based on the node number distribution ~60% of our customers use Geo mainly for DR. I think this is worth measuring because setting up Geo is always a conscious decision by the customer - it must be configured.
• We are working to add support for replicating all data types, so that Geo is on a solid foundation for both DR and Geo Replication.
• In 13.12 we replicate ~86% of all data types (25 out of 29 in total) and verify ~48%. Shipped Terraform State File verification

Metric Name: geo_nodes

Chart (Sisense↗)

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Implementation Status: Instrumentation

• Geo customers and user data available but we can't distinguish DR use case from replication yet.

Enablement:Geo - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Attention

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana
• Geo is not deployed on .com - This currently monitors one sidekiq worker that checks whether Geo is enabled.\

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Enablement:Memory - Group PPI - Memory Consumed

Average memory consumed by all invididual GitLab processes

SaaS/SM: Both

Target: 1.5GB

Is Primary PI: true

Health: Attention

• Memory consumption remains stable; 14.0 data to early to analyze.
• In collaboration with the Secure stage the team reduced the memory impact for the worst 1% of calls to the security report endpoint from 7GB down to 21MB,a 99.7% reduction. This shows that increasing visibility of problems plus collaboration can make a huge difference!
• By moving a subset of SideKiq workers from the primary database to read-only secondaries, the team removed about 10% of transactions from the primary PostgreSQL database - even on busy days. This is important because it increases the stability and performance of GitLab.com
• To further reduce memory consumption, we are planning to work on splitting the application into functional parts to ensure that only needed code is loaded and on reducing the memory consumption for Puma and Sidekiq endpoints.

Metric Name: topology.nodes[0].node_services

Chart (Sisense↗)

Implementation Status: Complete

• Working to adjust chart to include how many of each process are started by default to better represent a default configuration.
• Plot displays a 13.13 release for unknown reasons. Investigating

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

The number of unique active users and unique paid users interacting with either Basic Search or Advanced Search per month.

SaaS/SM: Both

Target: 10% month over month (SaaS and self-managed combined)

Is Primary PI: true

Health: Okay

• In May, Paid GMAU is up by 9%. SaaS up 10%, Self-Manged up 9%
• There is contiuning growth of more than 10% MoM since Oct 2020
• Advanced Search performance remains stable since scaling up to 12 nodes.
• UI enhancements planned for the next month to address top SUS feedback will help accelerate growth

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Data collected is incomplete for September and October 2020 and January 2021

Lessons Learned (July):

• The Recently Searched/Used Projects and Groups will be added to the dropdowns on the search results page. Our customers have been requesting this feature for over 5 years.

Enablement:Global Search - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana
• Error Budget was Exceeded for Global Search in May 2021. It has been discovered that the priamry reason causing this is Search Count calls.

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Database query Apdex, with 100ms the target and 250ms tolerable. This measures the ratio of queries which complete within the satisfactory time, informing how well the database is scaling and performing.

SaaS/SM: Both

Target: 0.99

Is Primary PI: true

Health: Attention

• The query Apdex has decreased in all charts after the release of 13.10, including the weekly one, which has a pretty constant number of instances reporting stats. No similar trend on the GitLab.com charts, nor any other clear signal indicating what may be affecting the Apdex. This is still present in 13.12 but the team has not focused on an investigation due to capacity constraints. Current focus is mitigating the Primary Key Overflow risk.
• We expect the updates performed in the scope of Research Database queries for performance and frequency to further improve the ratio of queries which complete within 250ms and lower the variance of the Apdex even further.
• We expect our work on automated database migration testing to indirectly improve the database performance by minimising the number of database related incidents during deployments.

Metric Name: query_apdex_weekly_average

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/227305

Implementation Status: Complete

• The PPI has been instrumented in 13.4.

Lessons Learned (July):

• Apdex on GitLab.com exceeds our group PPI (see 100ms - Tolerable 250ms and 50ms - Tolerable 100ms for master) but reflects with sharp drops the production incidents that related to the database.

Enablement:Database - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

The sum of all monthly active users on gitlab.com that roll up to paid accounts in a 28 day rolling period.

SaaS/SM: SaaS

Target: 134000

Is Primary PI: true

Health: Okay

• Paid Monthly Active Users on GitLab.com increased by 4.3% in May to 130K, on track for our Q2 target.
• Net Retention on GitLab.com remains above the 130% target while Gross Retention is below the 90% target due to a dip starting in January 2021. We created a dashbord to investigate further and found that the drop was driven by a Large customer whose spend contracted this year citing uncertainty in their industry due to the pandemic and an SMB customer who decided to downgrade to Premium because they weren't using the Ultimate feature set. We expect Gross Retention to stay in the 85% range through the end of the calendar year since the metric is pulling in last year's spend from these two customers and comparing it to their current spend on a monthly basis. We will continue to monitor retention as a supporting metric for Paid GMAU moving forward.

Metric Name: COUNT_EVENTS_ACTIVE_USERS_LAST_28_DAYS_BY_PLAN_WAS_PAID

Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation complete.

Lessons Learned (July):

• Need to account for seasonal fluctuations into target projections.

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Centralized Dashboards
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Talent Acquisition Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
-1	Confidential	Metric & metric health are confidential
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.