We are organizing for and regularly reviewing Performance Indicators (PIs) in order to enable a dialog between ourselves (all of R&D) about most effective use of our R&D efforts towards the most impactful improvements. As a result we should:
As a communication tool, Performance Indicators are only as useful as the range of the audience. As a result we should:
The section PI overview shows Primary PIs for each Product Section, Stage, and Group.
| Stage/Group | PI Type | PI | Health | SaaS / SM | PI Workflow |
|---|---|---|---|---|---|
| Ops | Section TMAU | Sum of Ops Section SMAU | Okay | Both | Complete |
| Verify, Verify:CI, Verify:Pipeline Authoring, Verify:Runner | SMAU, GMAU | Unique users triggering pipelines | Okay | Both | Complete |
| Verify:Testing | Paid GMAU | Count of active paid testing feature users | Unknown | SaaS | Instrument Tracking |
| Package, Package:Package | SMAU, GMAU | Count of users publishing/installing packages | Unknown | Both | Instrument Tracking |
| Release, Release:Progressive Delivery, Release:Release Management | SMAU, GMAU | Count of users triggering deployments using GitLab | Okay | Both | Complete |
| Configure, Configure:Configure | SMAU, GMAU | Proxy user count based on attached clusters | Okay | Both | Complete |
| Monitor | SMAU | Proxy user count based on prometheus integration | Okay | Both | Complete |
| Monitor:Health | GMAU | Unique users that interact with Alerts and Incidents | Unknown | Both | Instrument Tracking |
A sum of all SMAUs across the five stages in the Ops Section (Verify, Package, Release, Configure & Monitor)
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Complete
Target: 466k by End of Q3
Is Primary PI: true
URL(s)
Chart (Sisense↗)
Health: Okay
Instrumentation: Attention
A rolling count of unique users triggering pipelines in the last 28 days
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Complete
Target: Our FY21 Q3 Target is to see 10% growth. We are tracking towards a goal of 500k unique users triggering pipelines by end of FY21 (Jan 31, 2021).
Is Primary PI: true
URL(s)
Chart (Sisense↗)
Health: Okay
Instrumentation: Okay
A rolling count of unique users who have interacted with a paid testing feature in the last 28 days
Free/Paid: Paid
SaaS/SM: SaaS
PI Workflow: Instrument Tracking
Target: TBD
Is Primary PI: true
URL(s)
Chart (Sisense↗)
Health: Unknown
Instrumentation: Attention
A count of events in which a package is published to or installed from the Package Registry on GitLab.com.
Free/Paid: None
SaaS/SM: SaaS
PI Workflow: Complete
Target: The original FY21 Q3 target of 55k events per day, a 37% increase from Q2, has already been reached. 70k events per day is our new target.
URL(s)
Chart (Sisense↗)
Health: Okay
Instrumentation: Okay
The maximum distinct count of users who either published a package or interacted with an installed package from the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: n/a
Is Primary PI: true
URL(s)
Health: Unknown
Instrumentation: Problem
The maximum distinct count of paid users who either published a package or interacted with an installed package from the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.
Free/Paid: Paid
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: n/a
URL(s)
Health: Unknown
Instrumentation: Problem
The distinct count of users who publish a package to the Package Registry.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: n/a
URL(s)
Health: Unknown
Instrumentation: Problem
The distinct count of users who interact with a package that is installed from the Package Registry.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: n/a
URL(s)
Health: Unknown
Instrumentation: Problem
A rolling count of unique users triggering deployments with Gitlab in the last 28 days.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Complete
Target: Our FY21 Q3 Target is to see a growth of 24%.
Is Primary PI: true
URL(s)
Chart (Sisense↗)
Health: Okay
Instrumentation: Okay
A rolling count of environment page and release page view for Gitlab.com in the last 28 days.
Free/Paid: None
SaaS/SM: SaaS
PI Workflow: Complete
Target: Our FY21 Q3 Target is to see a growth of 10%.
Chart (Sisense↗)
Health: Okay
Instrumentation: Attention
A rolling count of users triggering feature flag toggles with Gitlab in the last 28 days.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: TBD
Health: Unknown
Instrumentation: Problem
A rolling count of users triggering automatic deployments (in without any manual activity) with Gitlab in the last 28 days.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: TBD
Health: Unknown
Instrumentation: Unknown
A rolling count of unique users triggering pipelines which created a review app with Gitlab in the last 28 days.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: TBD
Health: Unknown
Instrumentation: Unknown
A rolling count of users using advanced deployments, namely, canary and incremental rollouts with Gitlab in the last 28 days.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: TBD
Health: Unknown
Instrumentation: Problem
A rolling count of users using using AutoDevOps or Ci templates to deploy to AWS based on cloud_service:$CI_CLUSTER_CLOUD_SERVICE_TYPE in the last 28 days.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: TBD
Health: Unknown
Instrumentation: Problem
A count of all users in projects that have an attached cluster because project users receive benefit from the cluster without direct usage.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Complete
Target: 5k Configure SMAU by the end of Q3FY21 (20% growth)
Is Primary PI: true
Chart (Sisense↗)
Health: Okay
Instrumentation: Attention
A rolling count of the number of projects using GitLab Managed Terraform State in the last 28 days
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Complete
Target: 700 by end of Q3FY21
URL(s)
Chart (Sisense↗)
Health: Okay
Instrumentation: Attention
A rolling count of the number of GitLab Kubernetes Agent instances in the last 28 days
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Data Availability
Target: 100
URL(s)
Chart (Sisense↗)
Health: Problem
Instrumentation: Okay
A rolling count of pipelines calling infrastructure management deployment commands in the last 28 days
Free/Paid: None
SaaS/SM: SaaS
PI Workflow: PI Target
Target: TBD
URL(s)
Chart (Sisense↗)
Health: Unknown
Instrumentation: Attention
A count of all users in projects that have a connected Prometheus integration because project users receive benefit from the cluster without direct usage.
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Complete
Target: 11k Monitor SMAU by the end of Q3FY21 (20% growth)
Is Primary PI: true
Chart (Sisense↗)
Health: Okay
Instrumentation: Attention
Unique users that interacted with alerts or incidents
Free/Paid: Both
SaaS/SM: Both
PI Workflow: Instrument Tracking
Target: TBD once we understand current activity
Is Primary PI: true
URL(s)
Health: Unknown
Instrumentation: Problem
Unique users who manually create incidents
Free/Paid: None
SaaS/SM: SaaS
PI Workflow: Complete
Target: TBD
Chart (Sisense↗)
Health: Unknown
Instrumentation: Attention
Count of projects where incidents are being created
Free/Paid: None
SaaS/SM: SaaS
PI Workflow: Complete
Target: 100 projects where incidents are being created for GitLab.com by the end of Q3 - October 2020
Chart (Sisense↗)
Health: Problem
Instrumentation: Attention
Total number of unique users that view metrics and log on GitLab.com
Free/Paid: None
SaaS/SM: SaaS
PI Workflow: PI Target
Chart (Sisense↗)
Health: Okay
Instrumentation: Attention
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| 0 | Unknown | Unknown |
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | We are correctly capturing and measuring data for GitLab.com and self-managed instances. |
| 2 | Attention | Missing one of: GitLab.com or self-managed data. |
| 1 | Problem | Missing both: GitLab.com and self-managed data. |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name} |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section) |
stage |
Optional | the product stage (Ex: release) as defined in stages.yml |
group |
Optional | the product group (Ex: progressive_delivery) as defined in stages.yml |
category |
Optional | the product group (Ex: feature_flags) as defined in categories.yml |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI. |
urls |
Optional | list of urls associated with the (K)PI |
sisense_data |
Optional | contains elements/properties related to Sisense, including chart (numeric Sisense widget ID), dashboard (numeric Sisense dashboard ID), shared_dashboard (Sisense shared dashboard ID), embed (v2) |
sisense_data_secondary |
Optional | same as sisense_data. Allows a second sisense chart to be included. |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
pi_type |
Optional | indicates the Product PI type (Ex: AMAU, GMAU, SMAU) |
plan_type |
Optional | indicates if the metric is Free, Paid, or Both. |
telemetry_type |
Optional | indicates if the metric is available on SaaS, SM (self-managed), or Both. |
is_primary |
Optional | boolean flag that indicates if this is the Primary PI for the Product Group. |
instrumentation |
Optional | has two additional elements/property - level (inclusive value between 0-3) and reasons. This should be updated monthly before Key Meetings by the DRI. |
pi_workflow |
Optional | indicates the PI Workflow status |
Above ...Below ...At ...At or above ...At or below ...shared_dashboard, chart , and the dashboard key-value pairs to the corresponding Performance Indicators data file under the sisense_data property: in strings as it's an important character in YAML and will confuse the data parsing process. Put the string in "quotes" if you really need to use a :urls: should be an array (indented lines starting with dashes) even if you only have one urlmaturity.level and health.level display a value between 0 and 3 (inclusive).