Ops Section Product Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Ops Section Product Performance Indicators

On this page

• Regular Performance Indicators
• Other PI Pages
• Legends
  • Health
  • Instrumentation
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Regular Performance Indicators

Ops Section - TMAU for the Ops section

A sum of all SMAUs across the five stages in the Ops Section (Verify, Package, Release, Configure & Monitor)

Target: TBD

URL(s)

• https://about.gitlab.com/direction/ops/#north-star-metric

Chart (Sisense↗)

Status: Unknown

• No target defined yet
• Improvement - CI Adoption Journey
• Improvement - Runner Self-Guided Install

Instrumentation: Attention

• No specific single-metric dashboard available yet
• Hoping to improve our not yet instrumented Monitor and Configure SMAU

Verify:CI & Verify:Runner - Count of pipelines triggered by unique users

A rolling count of pipelines triggered by unique users in the last 30 days

Target: TBD

Status: Unknown

• No target defined yet

Instrumentation: Attention

• Research in progress for instrumentation.

Verify:Testing - Count of active paid testing feature users

A rolling count of users who have interacted with a paid testing feature

Target: TBD

Status: Unknown

• No target defined yet

Instrumentation: Attention

• The Performance Indicator recently changed.
• Research for new data is in progress.

Package:Package - Packages published/installed using the Package Registry

A count of events in which a package is published to or installed from the Package Registry on GitLab.com.

Target: The FY21 Q3 target 55k events per day, a 37% increase from Q2

Chart (Sisense↗)

Status: Okay

• We hit our Q1 target of increasing the rolling 7-day average from 20k to 22k.
• We hit our Q2 target of increasing the rolling 7-day average to 40k events. This 81% increase was due to increased adoption of NPM, Maven, Conan and NuGet repositories.
• We intend to hit our Q3 target by moving the Package Registry to the GitLab Starter edition and by starting to measure adoption for the new PyPI and Composer repositories.

Instrumentation: Attention

• Adoption of the PyPI and Composer repositories are not yet being measured.
• We are currently tracking this data for GitLab.com via Snowplow. We have an issue open to track this data via the Usage ping. But it is not currently prioritied. We are hoping to work with the telemetry team on an example implementation.
• Here is the issue defining the PI

Release:Progressive Delivery & Release:Release Management - Count of deployments using GitLab

A rolling count of deployments with Gitlab in the last 30 days.

Target: Our FY21 Q2 target was to see an increase in deployments by at least 10%.

URL(s)

• https://app.periscopedata.com/app/gitlab/640196/New-WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU
• https://app.periscopedata.com/app/gitlab/661492/Temp---Eli-Release-Stage-Examples?widget=8578558&udv=0

Chart (Sisense↗)

Status: Attention

• We are uncertain if the counter is the correct dimension, for example, 28-day counters instead of the all-time counters in usage ping, we have a time-bound on everything so we can be sure about when things happened.

Instrumentation: Okay

• We are currently tracking this data for GitLab.com via Snowplow and as a usage ping for self-managed instances.
• We need to Expand Revenue Attribution to Release SMAU

Configure:Configure - Number of CI/CD pipeline runs going through our infrastructure management deployment oriented features

A rolling count of pipelines calling infrastructure management deployment commands in the last 30 days

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/511813/Configure-team-business-metrics

Chart (Sisense↗)

Status: Unknown

• No target defined yet
• Auto DevOps - need to figure out a way to measure templates usage
• IaC - minor increase in usage - to early to reject noise hypothesis. We just started to collect data on our new features usage
• Kubernetes - had a design sprint, working on fixing the biggest shortcomings to increase adoption, currently working to ship new approach

Instrumentation: Attention

• We're still missing reliable metrics for self-managed instances.
• Improving chart to single metric across all infrastructure deployment types

Monitor:APM - Total number of metric and log views

Total number of metrics and log dashboard views on GitLab.com

Chart (Sisense↗)

Status: Unknown

• We don't have a set threshold to measure against.

Instrumentation: Attention

• Does not include Self-Managed via Usage Ping
• Log view tracking available in 13.1
• Move to single chart for both metrics and logs views

Monitor:Health - Count of incident issues created

The number of incidents created in a month for GitLab SaaS.

Target: TBD

URL(s)

• https://app.periscopedata.com/app/gitlab/668012/Health-Group-Dashboard

Chart (Sisense↗)

Status: Unknown

• No target defined yet

Instrumentation: Problem

• Both SaaS and usage ping data are being captured currently for a generic number of incident issues
• This data includes internal Gitlab projects - we need to filter this out
• We need to instrument time periods for the self-managed data to understand month over month usage
• We'd like to capture count of issues promoted from alerts and errors

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Growth Section Performance Indicators
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Secure & Defend Section Performance Indicators
  • Enablement Section Performance Indicators
• Corporate Finance Performance Indicators
• Recruiting Performance Indicators
• People Success Indicators
• Data Team KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
0	Unknown	Unknown

Instrumentation

Value	Level	Meaning
3	Okay	We are correctly capturing and measuring data for GitLab.com and self-managed instances.
2	Attention	Missing one of: GitLab.com or self-managed data.
1	Problem	Missing both: GitLab.com and self-managed data.
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To update the options for the maturity table, make changes to the Performance Indicators Maturities ERB file
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicator_target(name) helper function returns the target value from the target property for a specific PI from the Performance Indicators data file.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from the Performance Indicators data file and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this is the Performance Indicators data file which is in YAML file. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department)
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI.
urls	Optional	list of urls associated with the (K)PI
sisense_data	Optional	contains elements/properties related to Sisense, including chart (numeric Sisense widget ID), dashboard (numeric Sisense dashboard ID), shared_dashboard (Sisense shared dashboard ID), embed (v2)
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
instrumentation	Optional	has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI.

Guidelines

• Each KPI chart should be a timeseries chart.
  • Use Purple bars to denote values.
  • Use a single Green bar for the current month (in progress).
  • Use a Red stepped-line for timeseries target.
  • Optional: Use a Black dashed-line for rolling average.
• Each KPI should have a standalone dashboard with a single chart representing the KPI and a text box with a link back to the handbook definition.
  • In Sisense, create a shared dashboard link to get the shared dashboard ID.
  • In Sisense, use the Share Link action of the chart to get the chart (widget_id) and the dashboard ID.
  • Add the shared_dashboard, chart , and the dashboard key-value pairs to the Performance Indicators data file under the sisense_data property
• Avoid : in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :
• urls: should be an array (indented lines starting with dashes) even if you only have one url
• Both maturity.level and health.level display a value between 0 and 3 (inclusive). The health value is a manual input while the maturity is an automated value based on other discrete (K)PI data properties.