Enablement Section Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Enablement Section Performance Indicators

On this page

• Section PI Overview
• Regular Performance Indicators
  • Enablement - Section PPI, Stage PPI - Median End User Page Load Time
  • Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab
  • Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary
  • Enablement:Memory - Group PPI - Memory Consumed
  • Enablement:Global Search - Paid GMAU - The number of unique paid users per month
  • Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)
  • Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Section PI Overview

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement	Section PPI, Stage PPI	Median End User Page Load Time	Attention	Both	Target
Enablement:Distribution	Group PPI	Percentage of installations on the 3 most recent versions of GitLab	Attention	Both	Complete
Enablement:Geo	Paid GMAU	Number of unique users utilizing a Geo secondary	Attention	Both	Instrumentation
Enablement:Memory	Group PPI	Memory Consumed	Attention	Both	Complete
Enablement:Global Search	Paid GMAU	The number of unique paid users per month	Attention	Both	Complete
Enablement:Database	Group PPI	Database Query Apdex (100ms target, 250ms tolerable)	Attention	Both	Complete
Enablement:Infrastructure	Paid GMAU	Number of unique paid users that perform an action on GitLab.com	Attention	SaaS	Complete

Regular Performance Indicators

Enablement - Section PPI, Stage PPI - Median End User Page Load Time

Median end user page performance collected by Real User Monitoring. This metric captures how well we are doing in serving our primary customers, our end users, by measuring their end user experience.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: performanceTiming

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• We seem to have had a performance regression, which is showing up in our page load times. This is corroborated by our Sitespeed testing, with 13% fewer pages <2.5s and average LCP growing by 0.5s. We need to investigate. This does not seem related to our stability challenges recently.

Implementation Status: Target

• We are still working to improve our per-geo views.
• Need to identify a target.

Lessons Learned (March):

• We have made improvements to LCP, however TTI/TBT are still significant gaps vs. our primary competition

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Of the total number of self-managed installs, what percentage are on one of the three most recent versions.

SaaS/SM: Both

Target: 40%

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10278985&udv=1102166
• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10279895&udv=1102166

Metric Name: versions_behind_latest

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• A target of 40% has been set. 40% is the upgrade rate that was reached in early 2019.
• Insights - Continuing to fluctate in the high 20's, the overall trend of increasing age continues.
• In order to achieve our target, we will need to reverse the broader trend.

Implementation Status: Complete

• Primary PPI is complete, and target is set.

Lessons Learned (March):

• Overall the trend of the age of instances from 2017 still continues to climb, however it appears that the trend is is beginning to slow. This will need to be monitored next month to see if this remains true or if the trend reverts back to a higher rate.
• The Helm chart is still well above target, and when controlling for official Distribution artifacts the metric improves into the 30's.
• We are still in the process of analyzing the initial feedback now. We have also recieved some results from this study, which was focused on the choice of CE vs EE and respondants left some comments related to openess to receive more notifications about upgrading. We will conduct more research on the best way to notify users of upgrading and how to automate the upgrade process. The initial issue can be found here
• We will continue to iterate on how to better highlight when instances are out of date to administrators.

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Number of unique users utilizing a Geo secondary. This adoption metric helps us understand whether end users are actually seeing value in, and are using, geo secondaries.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Metric Name: geo_nodes

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• We've enabled gathering usage data from Geo secondaries and have added git fetch metrics to the usage data. These metrics will be available starting with the 13.11 release. We also intend to add tracking for git push operations on a Geo secondary.
• GMAU is currently based on logging into the Secondary web interface. Git access is more common and we will begin tracking Git pulls in 13.11 with tracking for Git pushes planned as part of the Geo Usage Ping Epic. We also know that the UX of the secondary web interface is not good and we want to remove it, see Opportunity Canvas(internal). In order to assess the impact of our planned changes, having this (low) GMAU is really important. We've recently completed a PoC to make the secondary UI indistinguishable from the primary. Implementation work is in progress and can be tracked in the secondary mimicry epic.
• For Disaster Recovery we are measuring potential Geo users - the number of active licenses. In an ideal world, no regular user would ever need to rely on Geo because there is no disaster, but if one occurs everyone benefits. Based on the node number distribution ~60% of our customers use Geo mainly for DR. I think this is worth measuring because setting up Geo is always a conscious decision by the customer - it must be configured.
• We are working to add support for replicating all data types, so that Geo is on a solid foundation for both DR and Geo Replication.Over 80% of data types are now replicated. Replication for group wikis was shipped in 13.10 and the Testing group plans to add replication for Pipeline Artifacts in 13.11.

Implementation Status: Instrumentation

• Geo is not available on GitLab.com today, so cannot use Snowplow or the .com database.

Lessons Learned (March):

• Read-only maintenance mode released in 13.9, a highly requested feature which makes it easier for system administrators to perform maintenance operations such as failovers.
• While there was a large jump from January to February, the number is still very low right now. Two potential reasons a) data is incomplete b) WebUI is read-only and requires a different UI. We are planning to change this, see this opportunity canvas and in-progress implementation work in the secondary mimicry epic.
• Steady month over month growth with 4.7% increase in new Geo customers from Jan to Feb.

Enablement:Memory - Group PPI - Memory Consumed

Average memory consumed by all invididual GitLab processes

SaaS/SM: Both

Target: 1.5GB

Is Primary PI: true

Metric Name: topology.nodes[0].node_services

Chart (Sisense↗)

Health: Attention

• Memory consumption has stabilized for the last four versions (13.7-13.10).
• The high impact issues in Running GitLab in a memory constrained environment have been completed. As a followup, we have completed benchmarking the memory consumption in a memory constrained environment, and we have added documentation on Running GitLab in a memory-constrained environment.
• To further reduce memory consumption, we are planning to work on splitting the application into functional parts to ensure that only needed code is loaded and on reducing the memory consumption for Puma and Sidekiq endpoints.

Implementation Status: Complete

• Working to adjust chart to include how many of each process are started by default to better represent a default configuration.

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

The number of unique active users and unique paid users interacting with either Basic Search or Advanced Search per month.

SaaS/SM: Both

Target: 10% month over month (SaaS and self-managed combined)

Is Primary PI: true

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• Self-Manged and SaaS both continues to see significant increase in Paid GMAU, Up +103K since December 2020.
• We continue to see steady growth in the number of customers that enabled Advanced Search. Usage Ping for Advanced Search

Implementation Status: Complete

• Data collected is incomplete for September and October 2020 and January 2021

Lessons Learned (March):

• Improvments to Global Search Documentation was identifed as the top request from customers. Documentation got major revisions as part of 13.10.

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Database query Apdex, with 100ms the target and 250ms tolerable. This measures the ratio of queries which complete within the satisfactory time, informing how well the database is scaling and performing.

SaaS/SM: Both

Target: 0.99

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/227305

Metric Name: query_apdex_weekly_average

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• Apdex is stable and continuing to exceed the target, with the Apdex for EE having fully caught up with CE.
• We expect the updates performed in the scope of Research Database queries for performance and frequency to further improve the ratio of queries which complete within 250ms and lower the variance of the Apdex even further.
• We expect our work on automated database migration testing to indirectly improve the database performance by minimising the number of database related incidents during deployments.

Implementation Status: Complete

• The PPI has been instrumented in 13.4.

Lessons Learned (March):

• Apdex on GitLab.com exceeds our group PPI (see 100ms - Tolerable 250ms and 50ms - Tolerable 100ms) but reflects with sharp drops the production incidents that related to the database.
• The vast majority of GitLab 13.9 instances use PostgreSQL 12.5. Only 2.4% of EE instances are still in PostgreSQL 11.x.

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

The sum of all monthly active users on gitlab.com that roll up to paid accounts in a 28 day rolling period.

SaaS/SM: SaaS

Target: 111750

Is Primary PI: true

Metric Name: COUNT_EVENTS_ACTIVE_USERS_LAST_28_DAYS_BY_PLAN_WAS_PAID

Chart (Sisense↗)

Health: Attention

• Paid Monthly Active Users on GitLab.com increased by 10% in February to 113K, surpassing our target of 111K.
• Insight - The increase was driven by several expansion deals in the Premium tier. We saw a similar jump in paid GMAU from January to February last year.
• Improvement - To influence Paid GMAU, we are focusing on improving uptime and stability of GitLab.com via the Rapid Action process, .com daily standup, and prioiritizing InfraDev related issues above new customer requests. Additionally, we are working with the Manage stage to eliminate key feature gaps that exist for Adminstrators on .com compared to self-managed, and extending our compliance posture on Gitlab.com.

Implementation Status: Complete

• Instrumentation complete.

Lessons Learned (March):

• Need to account for seasonal fluctuations into target projections.

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Threat Management Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Talent Acquisition Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a Red stepped-line for timeseries target.
  • Directional targets will be used:
    • Above ...
    • Below ...
    • At ...
    • At or above ...
    • At or below ...
  • Optional: Use a single Green bar for the current month (in progress). Preferred for metrics which accumulate through a month but optional due to the performance impact to render the chart.
  • Optional: Use a Black line for rolling average.
• Each Sisense dashboard for KPIs should consider the following settings to ensure timely updates:
  • Setting up auto-refresh for a frequency that fits the KPI
  • Excluding Dashboards from Auto Archive
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
• Multi-series performance indicators should consider the following guidelines:
  • If series are mutually exclusive, use stacked bars for each series with a monthly time series
  • If series are not mutually exclusive, use grouped bars for each series with a monthly time series
  • Do not graph any targets in the chart.
  • Current month styling guidelines will not apply
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :