Enablement Section Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Enablement Section Performance Indicators

On this page

• Section PI Overview
• Regular Performance Indicators
  • Enablement - Section PPI, Stage PPI - Median End User Page Load Time
  • Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab
  • Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary
  • Enablement:Memory - Group PPI - Memory Consumed
  • Enablement:Global Search - Paid GMAU - The number of unique paid users per month
  • Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)
  • Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Section PI Overview

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Enablement	Section PPI, Stage PPI	Median End User Page Load Time	Attention	Both	Dashboard
Enablement:Distribution	Group PPI	Percentage of installations on the 3 most recent versions of GitLab	Attention	Both	Complete
Enablement:Geo	Paid GMAU	Number of unique users utilizing a Geo secondary	Attention	Both	Instrumentation
Enablement:Memory	Group PPI	Memory Consumed	Attention	Both	Complete
Enablement:Global Search	Paid GMAU	The number of unique paid users per month	Attention	Both	Complete
Enablement:Database	Group PPI	Database Query Apdex (100ms target, 250ms tolerable)	Attention	Both	Complete
Enablement:Infrastructure	Paid GMAU	Number of unique paid users that perform an action on GitLab.com	Attention	SaaS	Complete

Regular Performance Indicators

Enablement - Section PPI, Stage PPI - Median End User Page Load Time

Median end user page performance collected by Real User Monitoring. This metric captures how well we are doing in serving our primary customers, our end users, by measuring their end user experience.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Metric Name: performanceTiming

Health: Attention

• We have improved, but are still slower than our primary competition
• We are continuing to focus on LCP as an OKR in Q4 for Q4 to continue to improve.

Implementation Status: Dashboard

• Working with data team to calculate and dashboard the metric. Using Page Load Time as best metric available from Snowplow without additional instrumentation.
• No target defined, as metric is not dashboarded yet.

Lessons Learned (December):

• We have made improvements to LCP, however TTI/TBT are still significant gaps vs. our primary competition

Enablement:Distribution - Group PPI - Percentage of installations on the 3 most recent versions of GitLab

Of the total number of self-managed installs, what percentage are on one of the three most recent versions.

SaaS/SM: Both

Target: 40%

Is Primary PI: true

URL(s)

• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10278985&udv=1102166
• https://app.periscopedata.com/app/gitlab/441909/Active-Instances?widget=10279895&udv=1102166

Metric Name: versions_behind_latest

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• A target of 40% has been set. 40% is the upgrade rate that was reached in early 2019.
• Insights - We have trended down slightly in November, from 29% to 27%. This is largely being driven by our Core instances, although we are seeing a more gradual decrease in Paid as well.
• In order to achieve our target, we will need to reverse the broader trend.

Implementation Status: Complete

• Primary PPI is complete, and target is set.

Lessons Learned (December):

• There is a continuing trend towards more out of date instances, which has been continuing since 2017. The Postgres 11 upgrade in 13.0 was a brief bump, but did not significantly alter the long term trajectory.
• Even when controlling for dormant instances with no activity, the trend persists.
• Before executing potential solutions, we should try to better understand the problem. We are running a survey to collect installation feedback, and will follow up with updates.

Enablement:Geo - Paid GMAU - Number of unique users utilizing a Geo secondary

Number of unique users utilizing a Geo secondary. This adoption metric helps us understand whether end users are actually seeing value in, and are using, geo secondaries.

SaaS/SM: Both

Target: TBD

Is Primary PI: true

URL(s)

• https://gitlab.com/groups/gitlab-org/-/epics/4660
• https://app.periscopedata.com/app/gitlab/500159/Enablement::Geo-Metrics

Metric Name: geo_nodes

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• First batch of data for users authenticating on a secondary via the WebUI is available; likely incomplete
• We are working to add support for replicating all data types, so that Geo is on a solid foundation for both DR and Geo Replication. Snippet replication shipped in 13.7. ~80% of data types are now replicated.

Implementation Status: Instrumentation

• Geo is not available on GitLab.com today, so cannot use Snowplow or the .com database.
• Working towards tracking Git traffic on secondaries and replication events for other data via Geo Usage Ping Epic. We anticipate that most Geo users interact with Geo in this way.

Lessons Learned (December):

• The number is very low right now. Two potential reasons a) data is incomplete b) WebUI is read-only and requires a different UI. We are planning to change this, see this opportunity canvas
• Year over Year growth of 232% with regards to potential Geo users(17k to 58.5k). We know that Geo has low penetration as a percentage of total deployments, but skews heavily toward the large enterprise with a 25% percentage of Premium+ user share.

Enablement:Memory - Group PPI - Memory Consumed

Average memory consumed by all invididual GitLab processes

SaaS/SM: Both

Target: 1.5GB

Is Primary PI: true

Metric Name: topology.nodes[0].node_services

Chart (Sisense↗)

Health: Attention

• We are targeting to get below 2GB of total RAM usage, so we need to get these services further below that target.
• We've identified high-impact issues to reduce the overall memory footprint
• Memory consumption is rising as a function of adding additional features

Implementation Status: Complete

• Working to better support topology metrics for multi-node instances and Kubernetes
• Working to adjust chart to include how many of each process are started by default to better represent a default configuration.

Enablement:Global Search - Paid GMAU - The number of unique paid users per month

The number of unique active users and unique paid users interacting with either Basic Search or Advanced Search per month.

SaaS/SM: Both

Target: 10% month over month (SaaS and self-managed combined)

Is Primary PI: true

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• December 2020 - Paid GMAU had a 9% increse in Recorded Self-Managed and 9% decrease in SaaS - GMAU had a 9% increse in Recorded Self-Managed and 9% decrease in SaaS - Decrease in SaaS could be attributed to performance issues impcating search this month.

Implementation Status: Complete

• Data collected is incomplete for Sept and Oct 2020

Lessons Learned (December):

• Search for files by name and path is surfacing as one of our largest Global Search use cases.
• Scaling Elasticsearch will continue to require engineering as long as we see growth in usage and document count in SaaS

Enablement:Database - Group PPI - Database Query Apdex (100ms target, 250ms tolerable)

Database query Apdex, with 100ms the target and 250ms tolerable. This measures the ratio of queries which complete within the satisfactory time, informing how well the database is scaling and performing.

SaaS/SM: Both

Target: 0.99

Is Primary PI: true

URL(s)

• https://gitlab.com/gitlab-org/gitlab/-/issues/227305

Metric Name: query_apdex_weekly_average

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Health: Attention

• Apdex is currently trending in the wrong direction, although it is so close and only two datapoints, it could be noise.
• Improvement - We are focusing on partitioning the largest tables to improve the performance and scalability of the database.

Implementation Status: Complete

• The PPI has been instrumented in 13.4.

Lessons Learned (December):

• Apdex is relatively stable over the last three releases.
• Apdex does not seem to trend down based on scale of instance.
• Apdex is notably lower on EE instances than CE, see secondary chart. We are investigating the reasons for this. There is no meaningful correlation with memory or PostgreSQL version. EE performs worse if the amount of queries is low (~1000 requests per hours) but overtakes CE for very large request volumes (100k+ per hours). As we are dealing with averages this may indicate that some background queries dominate the average for lower request volumes but are drowned out at high volumes.
• Apdex on GitLab.com exceeds our group PPI (see 100ms - Tolerable 250ms and 50ms - Tolerable 100ms)

Enablement:Infrastructure - Paid GMAU - Number of unique paid users that perform an action on GitLab.com

The sum of all monthly active users on gitlab.com that roll up to paid accounts in a 28 day rolling period.

SaaS/SM: SaaS

Target: 111750

Is Primary PI: true

Metric Name: COUNT_EVENTS_ACTIVE_USERS_LAST_28_DAYS_BY_PLAN_WAS_PAID

Chart (Sisense↗)

Health: Attention

• Surpassed 100K Paid Monthly Active Users on GitLab.com in November
• Insight - Growth rate decreased from 7.3% to 2.8% in November putting our Q4 target at risk. The decrease was largely driven by a drop in the Gold tier growth rate, which was higher than normal over the previous two months thanks to several new deal closures.
• Improvement - Continued focus on improving overall availability of GitLab.com by resolving corrective actions with stage groups, reducing complexity of deployments by creating a centralized release controller, and rolling out Gitaly Cluster on .com to achieve increased redundancy of repositories.

Implementation Status: Complete

• Instrumentation complete.

Lessons Learned (December):

• Paid GMAU is a lagging indicator and changes month over month are best understood by looking at upstream activity. For example, the growth rate for the Gold tier can fluctuate significantly based on the number of signed deals for that tier in a given month. To obtain more visibility into upstream activity, we are planning to add a supporting metric, .com win rate, which will provide more visibility into our sales funnel. This effort is being tracked here

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Dev Sub-Department Performance Indicators
    • Ops Sub-Department Performance Indicators
    • Secure Sub-Department Performance Indicators
    • Threat Management Sub-Department Performance Indicators
    • Growth Sub-Department Performance Indicators
    • Enablement Sub-Department Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• Data Team KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Meetings by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Meetings by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Meetings by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Meetings by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Meetings by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a Red stepped-line for timeseries target.
  • Directional targets will be used:
    • Above ...
    • Below ...
    • At ...
    • At or above ...
    • At or below ...
  • Optional: Use a single Green bar for the current month (in progress). Preferred for metrics which accumulate through a month but optional due to the performance impact to render the chart.
  • Optional: Use a Black line for rolling average.
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property
• Multi-series performance indicators should consider the following guidelines:
  • If series are mutually exclusive, use stacked bars for each series with a monthly time series
  • If series are not mutually exclusive, use grouped bars for each series with a monthly time series
  • Do not graph any targets in the chart.
  • Current month styling guidelines will not apply
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :