Principles - Processes - Categorization - GitLab the Product - Being a PM - Performance Indicators - Leadership
In order to provide a useful single metric for product groups which maps well to product-wide Key Performance indicators, some product performance indicators cascade upwards. Here is how:
Notes:
xMAU is a single term to capture the various levels at which we capture Monthly Active Usage (MAU), encompassing Action (AMAU), Group (GMAU), Stage (SMAU), and Combined (CMAU). If you can use GMAU instead of xMAU please do so because it is more specific, more actionable, and a leading indicator. We should not give xMAU metrics without clearly indicating one of the three versions, Recorded, Estimated, or Predicted.
Each xMAU metric should have three versions:
| KPI | Health | Status |
|---|---|---|
| Estimated Combined Monthly Active Users (CMAU) | Unknown |
|
| Paid Combined Monthly Active Users (Paid CMAU) | Unknown |
|
| Unique Monthly Active Users (UMAU) | Attention |
|
| New Group Namespace Trial to Paid Conversion Rate | Attention |
|
| New Group Namespace Create Stage Adoption Rate | Attention |
|
| New Group Namespace Verify Stage Adoption Rate | Attention |
|
| New Group Namespace with at least two users added | Attention |
|
| Stages per User (SpU) | Attention |
|
| Stages per Organization (SpO) | Attention |
|
| Category Maturity Advancement | Attention |
|
| Paid Net Promoter Score (PNPS) | Attention |
|
| New hire location factor | Attention |
|
| Acquisition impact | Okay |
|
| Percentage of transactions through self-service purchasing | Okay |
|
The estimated sum of all stage monthly active users (SMAU) in a rolling 28 day period for Self-Managed and SaaS. We extrapolate estimated CMAU based on the percentage of self-managed instances that report this data (not all self-managed instances do). Therefore the recorded number is not the same as the estimated or even the actual number. The chart currently reflects recorded CMAU. We plan to change this to estimated CMAU [soon](https://gitlab.com/gitlab-data/analytics/-/issues/6000). Numbers displayed below are for self-managed instances with usage ping (enabled), including the self-managed instance hosting the SaaS product. Data for (CMAU for SaaS) is available separately using GitLab.com Postgres Database Import. Owner of this KPI is the VP, Product Management.
Target: TBD
Chart (Sisense↗)
Health: Unknown
The sum of all (paid stage monthly active users ) in a 28 day rolling period. The license information used to generate this metric from usage ping is being validated, and the currently displayed data could be incorrect by up to 20%. Numbers displayed below are for self-managed instances with usage ping(enabled). Data for (Paid CMAU for SaaS ) is available separately using GitLab.com Postgres Database Import. Owner of this KPI is the VP, Product Management.
Target: TBD
Chart (Sisense↗)
Health: Unknown
The number of unique users that performed an event within the previous 28 days for Self-Managed and SaaS. Numbers displayed below are for self-managed instances with usage ping (enabled) and GitLab.com. Data for ( Paid Self-Managed (EE) Unique Monthly Active Users (UMAU) (28-day trailing window) ). Data for ( SaaS GitLab.com Unique Monthly Active Users (UMAU) (28-day trailing window) ) is also available using GitLab.com Postgres Database Import. UMAU calculation is the same using both data sources (usage ping and GitLab.com Postgres Database Import). Owner of this KPI is the VP, Product Management.
Target: TBD
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Attention
The conversion rate of a new group namespace on a trial to a paid namespace. This is calculated by dividing the number of eligible new group namespaces that were on a trial and converted to paid within 40 days of creation by the total number of eligible new group namespaces that were on a trial within 40 days of creation. Eligible namespaces are top-level namespaces, excluding personal namespaces. The data source is GitLab.com Postgres Database Import. The owner of this KPI is the Director of PM, Growth.
Target: See notes in Key Review slides
This KPI cannot be public.
URL(s)
Health: Attention
The adoption rate of a new group namespace adopting features on the Create stage . This is calculated as the % of eligible new group namespaces the have at least one Create SMAU action within the first 7 days of creation. Eligible namespaces are top-level namespaces, excluding personal namespaces. The data source is GitLab.com Postgres Database Import. The owner of this KPI is the Director of PM, Growth.
Target: 42%
This KPI cannot be public.
URL(s)
Health: Attention
The adoption rate of a new group namespace adopting features on the Verify stage . This is calculated as the % of eligible new group namespaces the have at least one Verify SMAU action within the first 90 days of creation. Eligible namespaces are top-level namespaces, excluding personal namespaces. The data source is GitLab.com Postgres Database Import. The owner of this KPI is the Director of PM, Growth.
Target: SaaS is TBD (FY21 Q4 focus).
This KPI cannot be public.
URL(s)
Health: Attention
The percentage of a new group namespace with at least two users added within 7 days of namespace creation. Eligible namespaces are top-level namespaces, excluding personal namespaces. User counts for top level namespaces can come from multiple avenues including sub-groups or projects. The data source is GitLab.com Postgres Database Import. The owner of this KPI is the Director of PM, Growth.
Target: See notes in Key Review slides
This KPI cannot be public.
URL(s)
Health: Attention
Stages per user is calculated by dividing Stage Monthly Active User (CMAU) by Monthly Active Users (UMAU). SpU is now calculated using solely Usage Ping Data Source. As Stages add new SMAU counters and instances upgrade to newer versions, this KPI will keep on increasing artificially and will stabilize beginning of Q2 2021. The Stages per User (SpU) KPI is meant to capture the number of DevOps stages the average user is using on a monthly basis. This metric is important, as each stage added triples paid conversion. Owner of this KPI is the VP, Product Management.
Target: Our Target SpU is 1.8.
Chart (Sisense↗)
Health: Attention
Stages per Organization is calculated by dividing the sum of Stage Monthly Active Organizations (with a Stage Monthly Active Organization being defined as an instance for Self-Managed or a Namespace for SaaS with at least one Stage Active User on a given month) by the number of Unique Monthly Active Organizations (with UMAO being defined as an Organization with at least one UMAU). The Stages per Organization (SpO) KPI is meant to capture the number of DevOps stages the average organization is using on a monthly basis. This metric is important, as each stage added triples paid conversion. Owner of this KPI is the VP, Product Management.
Target: undefined
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Tertiary Chart (Sisense↗)
Health: Attention
Number of category maturity advancements per quarter. We have a three year goal of having half of our categories at lovable maturity. It's important that we see steady advancement of maturity each quarter in order to achieve the larger goal. Owner of this KPI is the VP, Product Management.
Target: 5 category maturity advancements per quarter
Chart (Sisense↗)
Health: Attention
PNPS is the primary measurement of customer satisfaction with the GitLab product. Abbreviated as the PNPS acronym, please do not refer to it as NPS to prevent confusion. Measured as the percentage of paid customer "promoters" minus the percentage of paid customer "detractors" from a Net Promoter Score survey. Note that while other teams at GitLab use a satisfaction score, we have chosen to use PNPS in this case so it is easier to benchmark versus other like companies. Also note that the score will likely reflect customer satisfaction beyond the product itself, as customers will grade us on the total customer experience, including support, documentation, billing, etc. Owner of this KPI is Product Operations.
Target: Our Target PNPS is 40.
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Attention
This is a subset of an existing KPI. Please see the definition for the parent KPI.
Target: Our Target is 0.72.
Chart (Sisense↗)
Secondary Chart (Sisense↗)
Health: Attention
Acquisition impact measures the number of maturity levels advanced, as a result of acquisitions. The target is 3 per year. (Advancements by acqusition can be seen here)
Target: 3 per year
Chart (Sisense↗)
Health: Okay
Total transactions processed through the webstore, compared to transactions processed outside the webstore. This measure includes SaaS and self-managed subscriptions, additional CI minutes, storage, upgrading tiers and adding users. Driving adoption of self-service purchasing allows us to redirect manual sales efforts towards higher return activities.
Target: 85%
Chart (Sisense↗)
Health: Okay
The count of active Self Hosts, Core and Paid, plus GitLab.com. Owner of this KPI is VP, Product Management. This is measured by counting the number of unique GitLab instances that send us usage ping. We know from a previous analysis that only ~30% of licensed instances send us usage ping at least. We are planning to update the methodology to track opt-in rate more accurately. once a month.
Chart (Sisense↗)
Health: Okay
For a month take the number of refunds and divide that by the total amount of orders (including self-managed and sales initiated orders). This does not include Internal Refunds which are results from when a web direct opportunity from a closed month is not credited to the correct sales person. Internal Refunds are not refunds to customers, therefore not meaningful when analyzing refunds processed as orders from GitLab's revenue system, Zuora.
URL(s)
Chart (Sisense↗)
Health: Attention
Stage Monthly Active Users is required for all product stages. SMAU is defined as the count of unique users who performed a particular action, or set of actions, within a stage in a 28 day rolling period for Self-Managed and SaaS. Numbers displayed below are for self-managed instances with usage ping (enabled), including the self-managed instance hosting the SaaS product. All (SMAU for SaaS) is available separately using GitLab.com Postgres Database Import. Owner of this KPI is the VP, Product Management.
Target: TBD
Chart (Sisense↗)
Health: Attention
user_unique_users_all_secure_scanners statistic in usage ping. More details on AMAUPaid Stage Monthly Active Users is required for all product stages. Paid SMAU is defined as the count of SMAU (Stage Monthly Active Users) that roll up to paid instance for Self-Managed using Usage Ping data or a paid namespace for SaaS using GitLab.com Postgres Database Imports in a 28 day rolling period. The license information used to generate this metric is being validated, and the currently displayed data could be incorrect by up to 20%. Numbers displayed below for self-managed are for instances with usage ping (enabled). (Paid SMAU for SaaS) is available separately using GitLab.com Postgres Database Import. Owner of this KPI is the VP, Product Management.
Target: TBD
Chart (Sisense↗)
Health: Unknown
In categories where there is not a directly applicable SMAU metric as the category is focused on protecting our customer's customer, other metrics need to be reviewed and applied. An example of a group where this applies is [Protect's Container Security group](/handbook/product/categories/#container-security-group) where their categories are designed to protect our customers and by extension their customers who are actively interacting with our customer's production / operations environment. Metrics such as stage monthly active clusters (SMAC) can be used in place of SMAU until it is possible to achieve an accurate metric of true SMAU which may include measurements of monthly active sessions into the customer's clusters.
Target: TBD
Health: Unknown
GMAU is defined as the count of unique users who performed a particular action, or set of actions, within a group in a 28 day rolling period. Each R&D group is expected to have a primary performance indicator they are using to gauge the impact of their group on customer outcomes. Where possible, groups should use Group Monthly Active Users (GMAU) or Group Monthly Active Paid Users (GMAPU) as their primary performance indicator. GMAU should be used when the group is early in its maturity and its primary goal is new user adoption. Paid GMAU should be the primary performance indicator when groups are more mature and have proven an ability to drive incremental IACV. We prefer measuring monthly active usage over other potential measures like total actions taken (eg total dashboard views), as we believe that measuring active usage is a better measure of true user engagement with GitLab's product. Since R&D groups often contain more than one category, picking one category to base the action, or set of actions on, is a recommended simplification as we do not have the data bandwidth to support measuring every category's usage at this time.
The 36-month Predicted GMAU calculation is explained here.
Target: TBD
Chart (Sisense↗)
Health: Unknown
Paid GMAU is defined as the count of GMAU (Group Monthly Active Users) that roll up to a paid instance for Self-Managed or a paid namespace for SaaS in a 28 day rolling period. When Paid GMAU is the focus area for a group, they should measure actions tied to`Up-tiered features`, which are features that are unavailable in the free tier, as this will ensure we are tracking usage that has a clear tie to incremental IACV.
Target: TBD
Health: Unknown
An active and diligent product development is best evaluated by the number of features not being built. As a central tool in problem validation is the [use of opportunity canvas reviews](https://about.gitlab.com/handbook/product/product-processes/#opportunity-canvas), we can measure the number of product directions that we did not want to follow by looking at the number of the dropped or postponed opportunity canvases.
Target: 2 x number of PMs each quarter
Health: Problem
The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.
Target: 99.95% availability, allowed unavailability window is 21.6 minutes per month
Health: Problem
| Value | Level | Meaning |
|---|---|---|
| 3 | Okay | The KPI is at an acceptable level compared to the threshold |
| 2 | Attention | This is a blip, or we’re going to watch it, or we just need to enact a proven intervention |
| 1 | Problem | We'll prioritize our efforts here |
| 0 | Unknown | Unknown |
Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.
Other PI Pages sectionThese ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.
kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:
| Property | Type | Description |
|---|---|---|
name |
Required | String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name} |
base_path |
Required | Relative path to the performance indicator page that this (K)PI should live on |
definition |
Required | refer to Parts of a KPI |
parent |
Optional | should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag. |
target |
Required | The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly. |
org |
Required | the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section) |
section |
Optional | the product section (Ex: dev) as defined in sections.yml |
stage |
Optional | the product stage (Ex: release) as defined in stages.yml |
group |
Optional | the product group (Ex: progressive_delivery) as defined in stages.yml |
category |
Optional | the product group (Ex: feature_flags) as defined in categories.yml |
is_key |
Required | boolean value (true/false) that indicates if it is a (key) performance indicator |
health |
Required | indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI. |
health.level |
Optional | indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI. |
health.reasons |
Optional | indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
urls |
Optional | list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url |
funnel |
Optional | indicates there is a handbook link for a description of the funnel for this PI. Should be a URL |
sisense_data |
Optional | allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes. |
sisense_data.chart |
Optional | indicates the numeric Sisense chart/widget ID. For example: 9090628 |
sisense_data.dashboard |
Optional | indicates the numeric Sisense dashboard ID. For example: 634200 |
sisense_data.shared_dashboard |
Optional | indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b |
sisense_data.embed |
Optional | indicates the Sisense embed version. For example: v2 |
sisense_data_secondary |
Optional | allows a second Sisense dashboard to be embeded. Same as sisense data |
sisense_data_secondary.chart |
Optional | Same as sisense_data.chart |
sisense_data_secondary.dashboard |
Optional | Same as sisense_data.dashboard |
sisense_data_secondary.shared_dashboard |
Optional | Same as sisense_data.shared_dashboard |
sisense_data_secondary.embed |
Optional | Same as sisense_data.embed |
public |
Optional | boolean flag that can be set to false where a (K)PI does not meet the public guidelines. |
pi_type |
Optional | indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI) |
product_analytics_type |
Optional | indicates if the metric is available on SaaS, SM (self-managed), or Both. |
is_primary |
Optional | boolean flag that indicates if this is the Primary PI for the Product Group. |
implementation |
Optional | indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI. |
implementation.status |
Optional | indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI. |
implementation.reasons |
Optional | indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason. |
lessons |
Optional | indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI. |
lessons.learned |
Optional | learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned |
monthly_focus |
Optional | indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI. |
monthly_focus.goals |
Optional | indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal |
metric_name |
Optional | indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name. |
Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.
Estimated Unique all Monthly Active Users between 2020-Jun and 2020-Jul is driven by the addition of newly available CE MAU data.Please see Data for Product Managers
