Ops Section Product Performance Indicators

1. You are here:
2. Handbook
3. Product Handbook
4. Ops Section Product Performance Indicators

On this page

• Intent
• Section PI Overview
• Regular Performance Indicators
  • Ops - Section CMAU - Sum of Ops Section SMAU
  • Ops - Section SpU - Ops Section Contribution to Stages Per User
  • Verify:Pipeline Execution - GMAU, SMAU - Unique users triggering pipelines
  • Verify:Pipeline Execution - Error Budget for GitLab.com
  • Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file
  • Verify:Pipeline Authoring - Other - % first green project
  • Verify, Verify:Pipeline Authoring - Error Budget for GitLab.com
  • Verify:Runner - GitLab SaaS Count of Shared Runners not Disabled on CI enabled projects
  • Verify:Runner - Other - GitLab SaaS customers use of GitLab SaaS vs. self-hosted Runner Minutes (Premium, Ultimate)
  • Verify:Runner - Other - GitLab SaaS customers use of GitLab SaaS vs. self-hosted Runner Minutes (Free)
  • Verify:Runner - Other - Linux Shared (SaaS)
  • Verify:Runner - Other - Windows Shared Runners (SaaS) Minutes Used
  • Verify:Runner - Other - Total downloads of runner binaries and docker images per month
  • Verify, Verify:Runner - Error Budget for GitLab.com
  • Verify:Runner - Other - CI Runners SLO Trends
  • Verify:Testing - GMAU - Count of active testing users
  • Verify, Verify:Testing - Error Budget for GitLab.com
  • Package:Package - SMAU, GMAU - Count of users publishing packages
  • Package:Package - Weekly count of users publishing packages
  • Package:Package - Other - Packages published using the Package Registry
  • Package:Package - Error Budget for GitLab.com
  • Release:Release - SMAU - Count of users triggering deployments using GitLab
  • Release:Release - AMAU - Unique users creating Releases
  • Release:Release - AMAU - Unique users using Environments
  • Release:Release - AMAU - Unique users of Pages
  • Release:Release - AMAU - Unique users triggering feature flags feature flag toggles
  • Release:Release - AMAU - Unique users triggering review apps pipelines that created a review app
  • Release:Release - AMAU - Count of unique users using canary or incremental rollouts users using Canary or Incremental Rollout
  • Release:Release - AMAU - Count of users deploying to AWS AutoDevOps or Ci templates to deploy to AWS
  • Release:Release - Other - Count of users looking at DORA4 Analytics
  • Release:Release - Other - Count of CLI creating releases
  • Release:Release - Error Budget for GitLab.com
  • Configure:Configure - SMAU, GMAU - MAU of GitLab Managed Terraform State
  • Configure:Configure - Adoption PI - Number of Gitlab Kubernetes Agent instances
  • Configure:Configure - AMAU - Unique users running pipelines that have a HashiCorp Vault Variable
  • Configure:Configure - Error Budget for GitLab.com
  • Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents
  • Monitor:Monitor - Other - Count of projects where incidents are being created
  • Monitor:Monitor - AMAU - Alerts Active Users
  • Monitor:Monitor - AMAU - Incidents Active Users
  • Monitor:Monitor - AMAU - On-call users
  • Monitor:Monitor - Error Budget for GitLab.com
• Other PI Pages
• Legends
  • Health
• How to work with pages like this
  • Pages
    • Helper Functions
  • Data
  • Guidelines

Intent

We are organizing for and regularly reviewing Performance Indicators (PIs) in order to enable a dialog between ourselves (all of R&D) about most effective use of our R&D efforts towards the most impactful improvements. As a result we should:

• focus on singular Performance Indicators
• focus on the art of the possible, what we can measure and react to quickly
• focus on business metrics, particularly SMAU
• prioritize for the intent of representing return on business investment (SMAU/GMAU that covers SaaS and Self-Managed) over the intent of building a rapid experimentation capability. This will have the practical effect of PMs prioritizing instrumentation and funnel building in Usage Ping over Snowplow.
• continue beyond measurement and review of our Performance Indicators to building a Growth Model, a Growth funnel and supporting metrics and segmentation

As a communication tool, Performance Indicators are only as useful as the range of the audience. As a result we should:

• store PI metrics, funnels, growth models, status and next steps handbook first
• expose our product groups to PI metrics regularly
• communicate transparently about adjustments to PIs within our groups
• reference PIs in planning issues and prioritization discussions

Section PI Overview

Stage/Group	PI Type	PI	Health	SaaS / SM	Implementation Status
Ops	Section CMAU	Sum of Ops Section SMAU	Okay	Both	Complete
Ops	Section SpU	Ops Section Contribution to Stages Per User	Okay	Both	Complete
Verify:Pipeline Execution	GMAU, SMAU	Unique users triggering pipelines	Okay	Both	Complete
Verify:Pipeline Authoring	GMAU	Number of unique users interacting with gitlab-ci.yml file	Okay	Both	Implementation
Verify:Runner	Other	Total downloads of runner binaries and docker images per month	Unknown	SM	Definition
Verify:Testing	GMAU	Count of active testing users	Attention	Both	Definition
Package:Package	SMAU, GMAU	Count of users publishing packages	Okay	Both	Complete
Release:Release	SMAU	Count of users triggering deployments using GitLab	Okay	Both	Complete
Configure:Configure	SMAU, GMAU	MAU of GitLab Managed Terraform State	Okay	Both	Complete
Monitor:Monitor	SMAU, GMAU	Unique users that interact with Alerts and Incidents	Problem	Both	Complete

Regular Performance Indicators

Ops - Section CMAU - Sum of Ops Section SMAU

A sum of all SMAUs across the five stages in the Ops Section (Verify, Package, Release, Configure & Monitor)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 1.67M by End of FY22-Q2

Is Primary PI: true

Health: Okay

• Reset Q2 goal to increase above current trend-line at 2.5% per month.

Metric Name: ci_pipelines, incident_management_total_unique_counts_monthly, deployments, packages, Managed Terraform State

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/781120/Ops-Section-Dashboard
• https://docs.google.com/spreadsheets/d/1U4nHovtwqqf9hW4X1b-8eH3l2rAUVWviF7VojKpCiLc/edit#gid=0

Implementation Status: Complete

• No updates

Lessons Learned (June):

• Insight - Trend line is more linear than we would hope especially with more stage contributions coming on-line.
• Improvement - Focus on awareness and adoption amongst Release, Package, Configure and Monitor

Ops - Section SpU - Ops Section Contribution to Stages Per User

In order to get a sense of the increase in overall stages per user attributed to the Ops Section stages we divide Section CMAU / Overall UMAU. This is analagous to Product-wide KPIs of CMAU and SMAU and helps us visualize the relative dispersion of the completeness of adoption across stages. The nuber is not a percent - but is the actual stages count from the overall stages per user that was provided by Ops Section stages.

SaaS/SM: Both

Target: 0.47 end of FY22Q2

Is Primary PI: true

Health: Okay

• Reset for Q2 Target.

Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/781120/Ops-Section-Dashboard

Implementation Status: Complete

• None

Lessons Learned (June):

• Insight - Some increase as stage adoption has increased primarily in Verify

Verify:Pipeline Execution - GMAU, SMAU - Unique users triggering pipelines

A rolling count of unique users triggering pipelines in the last 28 days because CI is the foundation for several downstream product areas, and improving SMAU for people using pipelines is important to help with discovery of these further features.

SaaS/SM: Both

Target: Our projected unique users triggering pipelines by end of FY22-Q2 is 1.32M, so we are going to target just above that at 1.35M unique users triggering pipelines.

Is Primary PI: true

Health: Okay

• On track and healthy, we expect to be progressing to FY22-Q3 target as expected.

Metric Name: ci_pipelines

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Insight - We can see in June a general increase in Verify MAU from 1.18M to 1.20M, which is a less than 2% percent overall increase of users. This growth can be seen coming in users on both SaaS (+20K) and self-managed (+10K). Although, when looking at the Paid Segment, we can see a slight decrease from the 400K MAU in May to 399K MAU, which could be attributed to a data tracking error. We have seen a 5% increase in recorded usage of Paid SaaS of users triggering pipelines on GitLab.com in June, which is consistent with the trend of the past 3 months, and we will continue to monitor this to see how enforcing public projects on new namespaces will impact this usage.
• Insight - The Verify Funnel and Drop off and overtime Funnel view. shows in June, a sustained gap between SCM MAU and Verify MAU despite the slight increase in MAU in Verify. The dropoff remains 55%. Pipeline Authoring MAU continues to increase slightly toward their ~250K Q3 target MAU, sustaining the 86% drop off from last month between Verify MAU. Although, we have not seen an improvement in Testing MAU, which continues to decline. We have restated the focus of the testing group and look to add a new PI focus in the coming milestones to this funnel.

Lessons Learned (June):

• Insight - In June 2021, we are starting to see a retention rate decrease from 89% to 82% for self-managed users, which requires more investigation to see if this is a data integrity reporting concern or actual account attrition.
• Insight - In June 2021, to elaborate on the retention rate concern in comparison to May expanding instances have contracted slightly (by -9%), while new instances are slightly decreasing (by -18%) and shrinking instances are improving significantly (by 24%) overall, there is not much new usage although we are not losing any existing users either. This seems to be slightly contraian to the 82% retention rate indicated in June.

Verify:Pipeline Execution - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Attention

• Insight - In July, we exceeded the budget by 8 hours (2 hours less than June), which can be attributed to GET /api/projects/:id/jobs/:job_id/trace and the improvements made to reduce latency related to POST /api/projects/:id/pipeline as well as Projects::JobsController end points.
• July availability improved from 98.60% to 98.80%, +.20%
• Insight - In our Sentry Bug tracking, we have seen continued issues around some API end points and will continue making improvements - Improvement - We have a spike via gitlab#331656 in 14.3 that will help us optimize these queries further once we reduce the load on them.
• Improvement - We have sceduled gitlab#276192 for 14.2 which will add build log size limits helping to mitigate errors with large log sizes.
• Improvement - Lastly, we have noticed several endpoints related to other groups and will work to get those reassigned in gitlab#334556

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Verify:Pipeline Authoring - GMAU - Number of unique users interacting with gitlab-ci.yml file

Count the number of users that are pushing MRs to gitlab-ci.yml files. This indicates on-going usage of pipeline authoring by continually adjusting pipeline configuration.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The FY22-Q2 target is 250,000 unique users interacting with the gitlab-ci.yml file on self-managed and SaaS.

Is Primary PI: true

Health: Okay

• Based on last quarter growth and growing number of users that are using the pipeline editor

Metric Name: redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Implementation

• Insight - Numbers keep growing, which aligns with the sentiment we receive from our users regarding the pipeline editor. The editor came out early this year and had several feature improvements. As a result, I expect this number to keep growing over the next month. Last week we had a record number of 13.8K unique users that interact with our editor.

Lessons Learned (June):

• Improvement - We were able to release Branch selector to allow users to work with the pipeline editor from any non-master branch. In addition, our research had shown that templates provide an easy way for users to start working with CI we've added the template support to the pipeline editor and instrument tracking to understand the user behavior around CI templates.

Verify:Pipeline Authoring - Other - % first green project

Count of projects that have the first passing pipeline in the 24 hours and the 28 days after project creation as a percentage of all new CI projects that created in the time period. This ratio helps us understand how effectively we are highlighting and enabling the GitLab CI/CD capabilities to new project creators. Our desire is an increasing percentage of users complete in the first 24 hours.

SaaS/SM: SaaS

Target: We'd expect this to increase 1% each month for both metrics (24hr and 28 days).

Health: Okay

• We'd like to improve our onboarding experience since we know that our convertion rate multiplied if users are adopting CI.

Metric Name: green_project

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Target

• The data currently in Sisense is only instrumented for both .com and not self-managed.

Lessons Learned (June):

• Insight - Its very hard to get insights out of this metric, we noticed an increasing number of projects which got created from potential abusers, this impact our percentage for first green pipeline. I've opened an issue for the data team to remove the potential abusers, if they wont be able to unclutter this metric we would need to consider changing our performance indicator.

Verify, Verify:Pipeline Authoring - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes in rolling 28 days.

Health: Attention

• This is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana
• Improvement - Pipeline Authoring is completing the investigation into the over budget in infra#1142.
• Insight - We've noticed error budget reduced to 20. This is attributed to errors and speed issues in GET /api/projects/:id/ci/lint and POST /api/:version/projects/:id/mirror/pull. Those 2 API's eating up our error budget, the first endpoint originated from only 2 customers. The team will try to understand better why those two users are generating slower requests than other users. The second endpoint is fairly distributed.

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Verify:Runner - GitLab SaaS Count of Shared Runners not Disabled on CI enabled projects

This chart displays the GitLab SaaS Runners conversion funnel for projects with a CI pipeline. You should review this funnel in concert with the Verify Funnel. Our point of view is that for GitLab SaaS, Verify monthly active users (MAU) on GitLab.com represents the total addressable universe of users that may consume CI minutes on GitLab SaaS Shared Runners. So, on the one hand, we will continue to develop actionable insights to help reduce the MAU drop-off from SCM MAU to Verify MAU. As depicted in this funnel graph, the other side of the coin is to investigate the use and adoption of GitLab SaaS Shared Runners and as such, the impact to Verify MAU.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: None defined at this time.

Health: Attention

• The chart is still a work in progress.

Metric Name: gitlab-saas-shared-runners-funnel

Chart (Sisense↗)

Implementation Status: Target

Lessons Learned (June):

• Insights - In the current chart, you will first notice the drop-off in users who create projects with CI pipelines as measured from the total user count. The starting user count = 191,158, with 63% of those users creating projects for a total of 121,393 projects created. Projects created that include CI pipelines are significantly lower at 13,076 or approximately 7%. The count of projects created including CI pipelines with Shared Runners not disabled is close to the count of projects created with CI pipelines, and comes in at 6%. Based on this first data set, we can posit that we can help increase Verify MAU by investing in the Shared Runners' capabilities, performance, and reliability on .com.
• Improvement - Finalize the funnel chart visualization and iterate on logic as needed for the next PI review.

Verify:Runner - Other - GitLab SaaS customers use of GitLab SaaS vs. self-hosted Runner Minutes (Premium, Ultimate)

This chart displays the split between CI minutes executed on GitLab SaaS Shared Runners compared to CI minutes on self-hosted runners for GitLab SaaS customers in Ultimate and Premium tiers.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: None defined at this time.

Health: Okay

• Metric is implemented and dashboards are created in Sisense.

Metric Name: shared-runner-vs-self-managed

Chart (Sisense↗)

Implementation Status: Target

Lessons Learned (June):

• Insights - 81% of the total CI minutes were on self-hosted runners, a one percentage point decline from May 2021.
• Insights - Preliminary feedback from customer indicates the following as potentially significant factors affecting the use of self-hosted vs GitLab SaaS Runners. The first is that customers choose not to use GitLab SaaS Shared Runners due to the consumption of CI minutes not being easy to track. Therefore, it is impossible to perform internal chargebacks efficiently. Secondly, customers choose to self-host runners since GitLab.com Shared Runners are unreliable. The third, is that some CI jobs require access to resources within the customer's internal network.
• Improvement - Our goal is to improve the reliability of GitLab SaaS Shared Runners as measured by incidents impacting the CI Runners SLO Apdex by the end of Q2. It is likely then that it will take consistently reliable performance through Q3 and Q4 FY22 to change customers' perspectives.

Verify:Runner - Other - GitLab SaaS customers use of GitLab SaaS vs. self-hosted Runner Minutes (Free)

This chart displays the split between CI minutes executed on GitLab SaaS Shared Runners compared to CI minutes on self-hosted runners for GitLab SaaS customers in the Free tier.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: None defined at this time.

Health: Okay

• Metric is implemented and dashboards are created in Sisense.

Metric Name: shared-runner-vs-self-managed-free

Chart (Sisense↗)

Implementation Status: Target

Lessons Learned (June):

• Insights - In June 2021, 59% of the total CI minutes were on self-hosted runners. This represents a 12% increase compared to May 2021.

Verify:Runner - Other - Linux Shared (SaaS)

The count of CI minutes used per month on GitLab.com Linux Shared Runners with the Docker executor. These are CI minutes as defined by those pipelines that are executed on runner managers with the label shared-runners-manager-X. One of our strategic focus areas is increasing combined monthly active users (CMAU). Providing customers on GitLab SaaS, a performant, and full-featured build infrastructure supports that strategy. Therefore, we believe that measuring CI minutes usage by platform is a leading indicator of future performance. Also, there are several potential future investment areas for Linux Runners on GitLab SaaS, such as offering multiple virtual machine types. We must evaluate these investments with the strategic focus areas in mind.

Funnel Description: Handbook Link

SaaS/SM: SaaS

Target: Growth rate of 5% monthly.

Health: Okay

• Metric is implemented and dashboards are created in Sisense.

Metric Name: NA

Chart (Sisense↗)

Implementation Status: Target

Lessons Learned (June):

• Insights - The overall growth rate in June is lower than in May 2021, which we can attribute to normalized usage due to the impact of the efforts to reduce abuse by crypto miners.The overall growth rate in June is lower than in May 2021, which we can attribute to normalized usage due to the impact of the efforts to reduce abuse by crypto miners.
• Insights - A closer look at the growth rates for the Premium and Ultimate plans shows positive usage growth month over month of 21% and 31%, respectively. We also see a 20% growth rate month over month in the CI minutes on self-hosted runners in the related metric.
• Insights - There are a couple of hypotheses or insights that we can glean from the trends. The first is that organizations that are using GitLab are continuing to mature their DevOps practices. This, of course reinforces the value proposition of GitLab as the single DevOps platform. And as a result, we must continue to provide the feature, capabilities, and reliability that customers need in CI and Runners so that GitLab remains the DevOps platform of choice.
• Insights - The second hypothesis is that if we provide customers on GitLab SaaS with a reliable and performant CI Runners solution, we can make a case for customers to transition more of their CI workloads to GitLab SaaS. By doing so, we further reduce the maintenance and overhead costs for customers related to operating their own runner fleet so that, again, long-term, GitLab continues to be the compelling DevOps platform that they choose to use in their organization.
• Improvements - A critical goal for Q3 is to continue to improve the reliability and stability of the CI Runners on GitLab SaaS. This is one foundational element needed to convince customers in Premium and Ultimate to run applicable CI jobs on GitLab SaaS Shared Runners.

Verify:Runner - Other - Windows Shared Runners (SaaS) Minutes Used

The count of CI minutes used per month on GitLab.com Windows Shared Runners with the Shell executor. These are CI minutes as defined by those pipelines that are executed on runner managers with the label windows-shared-runners-manager-X. We must have a clear understanding of use and adoption to more effectively forecast the impact of future investments in GitLab SaaS Windows Runners on our strategic focus areas.

Funnel Description: Handbook Link

SaaS/SM: SaaS

Target: Growth rate of 10% monthly.

Health: Okay

• Metric is implemented and dashboards are created in Sisense.

Metric Name: NA

Chart (Sisense↗)

Implementation Status: Target

Lessons Learned (June):

• Insights - The overall growth rate in June, 291% compared to May, is attributed to abuse by crypto miners, a specific attack vector that has since been mitigated. In looking at the usage for the Premium and Ultimate tiers, the growth rates are 18% and -10% respectively.
• Improvements - We need to build out additional reporting to more quickly and efficiently correlate the usage by customer.

Verify:Runner - Other - Total downloads of runner binaries and docker images per month

Count of downloads of runner binaries grouped by platform, gitlab-runner and gitlab-runner helper container images. Measuring runner binaries' downloads will provide insight into whether we are making the right investments in core runner features and capabilities by platform (operating system and computing architecture). The assumption is that if self-managed users continue to maintain and upgrade their GitLab Runner build infrastructure, this may suggest continued long-term use of GitLab as their DevOps platform.

Funnel Description: Handbook Link

SaaS/SM: SM

Target: Target monthly growth rate of 3% given monthly trends to date.

Is Primary PI: true

Health: Unknown

• Not instrumented

Metric Name: NA

Implementation Status: Definition

• We need to start collecting the AWS S3 bucket metrics and integrate them into the data warehouse. Issue for tracking is in the Infra Project pending scheduling.

Lessons Learned (June):

• Insight - Downloads of the gitlab-runner and the gitlab-runner helper container images continue to grow at an average rate of 3% monthly. The cumulative download count is now over 639M. The total downloads for June 2021 came in at 17.6M (gitlab-runner) and 8M (gitlab-runner helper).

Verify, Verify:Runner - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Lessons Learned (June):

• Insights - The error budget spend for June came in at 4.5 hours, a significant improvement compared to the previous months. The spend for the week ending July 2nd is 38.69 mins. These trends indicate that we are improving our service delivery capabilities. However, there continue to be times when there is service degradation. The overall rate of spend and loss of confidence by customers in the reliability of GitLab SaaS CI runners means that in Q3, we will continue to allocate at least 30% of the runner team's engineering budget to work on improving our deployment capabilities and CI Runners on .com stability and performance.
• Insights - For the week ending 2021-07-23, there is a slight uptick in the error budget spend of 0.23 hours. Even with the slight uptick, the trends over the past two weeks still point to the improved scalability and reliability of the CI runners on GitLab SaaS.

Verify:Runner - Other - CI Runners SLO Trends

This chart displays the SLO attainment trends for GitLab SaaS CI Runners. The error-rate and Apdex score are the two metrics (Service Level Indicators) used to calculate the SLO attainment metric.

SaaS/SM: Both

Target: 99.95% SLO attainment monthly.

Health: Okay

• Metric is implemented in Grafana. However there is work in progress to add the Grafana data to the data warehouse in order to create the corresponding Sisense charts. The Current Error Budget Runner dashboard is the SSOT for tracking. The CI Runners SLO trends chart in Grafana is the SSOT for the CI Runners SLO.

Metric Name: gitlab-saas-ci-runners-slo-trends

Chart

Implementation Status: Target

Lessons Learned (June):

• Insights - The average CI runners availability attained this week is 99.81%, which is .0.10% down from the previous week score of 99.91%.
• Improvement - The goal for the second half of this fiscal year is to achieve and maintain a weekly average SLO score of 99.95%.

Verify:Testing - GMAU - Count of active testing users

A rolling count of unique users who have interacted with a testing feature over the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: TBD

Is Primary PI: true

Health: Attention

• There is low testing feature adoption of existing CI users and growth is at or below overall stage growth rates.

Metric Name: NA

Chart (Sisense↗)

Implementation Status: Definition

• We are shifting away from PGMAU to GMAU to include our focus on Build Artifacts and top of funnel related work.

Lessons Learned (June):

• We had over 1700 Unique SaaS users interacting and over 1500 SM users interacting with Testing features in June.
• MAU growth of Test Summary Expansions was just under 5% between May and June indicating low awareness of the features.

Verify, Verify:Testing - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Okay

• Insight - In June, we met our budget and availability improved to 99.98% from 99.97% previously.
• Insight - The Testing group did not create the initial dashboard and has taken on additional functionality since its creation. As such an audit gitlab#331163 of the controllers monitored is being completed in the 14.2 milestone.
• Insight - A rcent incident related to Artifacts exposed that we do not have good tracking on artifact upload end points and will investigate further as part of our audit.

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Package:Package - SMAU, GMAU - Count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry. In the future, this value will the distinct count of users across all Package AMAUs.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 54,000 unique users publishing packages across SaaS and self-managed combined. This is an increase from the original target of 50,000 based on solid growth in May and in weekly user growth for June.

Is Primary PI: true

Health: Okay

• We hit the May target of 46k active users of the Package Registry.

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Implementation Status: Complete

• Generic package usage is not collecting data for usage ping. GitLab-#331455 is scheduled for 14.1 and will address this.

Lessons Learned (June):

• Insights - We hit the May target of 46k users per month!
• Insights - Growth was split evenly between SaaS and Self-Managed.
• Insights - From a funnel perspective, we saw significant growth in the packages pulled using a deploy token or by a Guest. Both are signs that the Package Registry is being integrated into our customers production workflows. The number of packages pulled using a Deploy token grew from 3mm to 4mm April to May 2021. The number of packages pulled by Guest users grew from 2mm to 3mm during the same time period.
• Improvements - In 14.1, we are investing in the container registry cleanup policies which will not have an impact on this metric.
• Improvements - Based on the above, in 14.2 we plan to resolve some known issues with deploy tokens. GitLab-#285495 will resolve an issue in which group deploy tokens don't work with sub-groups. GitLab-#332028 will resolve an issue in which deploy tokens do not work when a project or group has been updated from private to public or private to internal. We expect this will help drive retention.

Package:Package - Weekly count of users publishing packages

The maximum distinct count of users who published a package to the Package Registry in a given week.

SaaS/SM: Both

Target: The target for FY22-Q2 is 20,000 unique users publishing packages across SaaS and self-managed combined.

Health: Okay

• We are seeing steady growth by week.

Metric Name: user_packages_total_unique_counts_monthly

Chart (Sisense↗)

Implementation Status: Complete

• Generic package usage is not collecting data for usage ping. GitLab-#331455 is scheduled for 14.1 and will address this.

Lessons Learned (June):

• Insights - npm has been growing faster than any other format. Weekly we've been growing the number of users by 2% per week.
• Insights - In 14.0 we released a group endpoint for PyPI. Since it's release three weeks ago, the weekly number of users has grown 11% from 2250 to 2600. On GitLab.com the number of PyPI packages being pulled grew by 50%.
• Insights - Although the user base for the Composer repository is smaller, we have seen solid growth in the number of packages being published and installed. A good sign that some of our investments in 13.12 and 14.0 paid off.
• Improvements - In 14.2 we have several improvements planned for npm packages. GitLab-#275992 will give developers the ability to run npm bin, which is a requirement for working with Yarn. GitLab-#37364 will resolve a bug that has been impacting customers. Finally, GitLab-#243780 will investigate how to improve the performance of the various npm endpoints.

Package:Package - Other - Packages published using the Package Registry

A count of events in which a package is published to the Package Registry.

Funnel Description: Handbook Link

SaaS/SM: SaaS

Target: The FY22 Q2 target (July 31, 2021) is 3mm packages published per month amongst pinging instances.

Health: Okay

• We are seeing steady growth in the number of packages published to the registry.

Metric Name: NA

Chart (Sisense↗)

Implementation Status: Target

• As of June 1st 2021, we updated this performance indicator to use usage ping data, which includes GitLab.com, Self-Managed, and estimated uplift from Self-Managed.

Lessons Learned (June):

• Insights - When you try to install an npm package and it's not found in your GitLab registry, we'll look for and, if it's available, download the package from the npm public registry. On GitLab.com we see about thirty thousands events per day for this action, which is about 30% of the number of packages pulled from the GitLab.com registry.
• Improvement - GitLab-#233413 will add the same functionality for Python packages.

Package:Package - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Attention

• We are currently 9 minutes over our remaining 28d budget, which is an improvement of 2 minutes since the last weekly update.

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Target

Lessons Learned (June):

• Insights - The container registry group endpoint for the container registry slowest response time. We found that one particular customer is responsible for the majority of slow requests. Customer outreach is happening in GitLab-#8344.
• Improvement - GitLab-2896 proposes themes and issues to focus on amplifying SaaS reliability and performance.
• Improvement - GitLab-#243780 is scheduled for 14.2 and will investigate slow p95 latency scores for npm. In the past 7 days, 2.4% of requests have been above the tolerated apdex threshold.
• Improvement - GitLab-#6396 is scheduled for 14.3/4 which will move the dependency proxy download logic to workhorse to increase performance and reliability.
• Improvement - GitLab-333166 has been added to 14.3 and will investigate slow p95 latency scores for the Composer endpoint. Although the frequency of these requests occur much less often than npm and NuGet, I'd like to prioritize this issue sooner since 45% of requests our outside of the tolerated threshold.

Release:Release - SMAU - Count of users triggering deployments using GitLab

A rolling count of unique users triggering deployments with Gitlab in the last 28 days.

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 360,000 for unique users triggering deployments

Is Primary PI: true

Health: Okay

• 10% growth between May and April

Metric Name: deployments

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Tertiary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/777879/Release-Stage-Dashboard

Implementation Status: Complete

Lessons Learned (June):

• Improvements - Deployment frequency is available at the API, project, and group level. Lead time for Merge Request is available at the API, and Project level and will be available at the Group level after 14.0.
• Talking to enterprise customers, it is becoming more clear that GitLab doesn't have the release orchestration features needed to coordinate complex releases.
• Customers are also interested in better management of environments

Release:Release - AMAU - Unique users creating Releases

A rolling count of users creating releases in the last 28 days.

SaaS/SM: Both

Target: 50,000 for FY22-Q2

Health: Okay

• Met last quarter's very ambitious goal of increasing unique users by 30%

Metric Name: release_management

Chart (Sisense↗)

Implementation Status: Complete

Lessons Learned (June):

• We should break this metric down by tier to see what is driving the change.
• We have interest in connecting a Release to the pipeline. This will make Releases even more useful.

Release:Release - AMAU - Unique users using Environments

A rolling count of unique users that have at least one project on their environment dashboard in the last 28 days.

SaaS/SM: Both

Target: TBD

Health: Okay

• Steady, no goals set yet

Metric Name: Environment Instances

Chart (Sisense↗)

Lessons Learned (June):

• This will be a focus area going forward.

Release:Release - AMAU - Unique users of Pages

A rolling count of unique users that have a Pages project in the last 28 days.

SaaS/SM: Both

Target: TBD

Health: Okay

• Usage is steady

Metric Name: pages

Chart (Sisense↗)

Implementation Status: Problematic

• We need to update the chart to use the official GitLab snippet
• The chart shown now is custom domains, we need to see actually unique users deploying pages

Lessons Learned (June):

• We completed the big Pages migration. We will deprecate the legacy storage for Pages in 14.0, it will be interesting to see if this impacts usage.

Release:Release - AMAU - Unique users triggering feature flags feature flag toggles

A rolling count of users triggering feature flag toggles with Gitlab in the last 28 days.

SaaS/SM: SaaS

Target: FY22Q1 increase of 10% in Unique users triggering feature flags.

Health: Attention

• May numbers dropped off

Metric Name: feature_flags

Chart (Sisense↗)

Release:Release - AMAU - Unique users triggering review apps pipelines that created a review app

A rolling count of unique users triggering pipelines which created a review app with Gitlab in the last 28 days.

SaaS/SM: Both

Target: TBD

Health: Attention

• Continues to grow, but this chart is not believable.

Metric Name: review_apps

Chart (Sisense↗)

Implementation Status: Problematic

• Improvements needed, use official GitLab snippet

Lessons Learned (June):

• The line continues to be flat. We should look into how we are building this chart.

Release:Release - AMAU - Count of unique users using canary or incremental rollouts users using Canary or Incremental Rollout

A rolling count of users using advanced deployments, namely, canary and incremental rollouts with Gitlab in the last 28 days.

SaaS/SM: SaaS

Target: TBD

Health: Attention

• Seemingly low usage. We don't have a strategy yet on how to drive improvements.

Metric Name: advanced_deployments

Chart (Sisense↗)

Implementation Status: SaaS only - need to create a usage ping

• Improvements - Create the instrumentation Instrument AMAU of Canary Rollouts
• Improvements - Create the instrumentation Instrument AMAU of Incremental Rollouts

Release:Release - AMAU - Count of users deploying to AWS AutoDevOps or Ci templates to deploy to AWS

A rolling count of users using using AutoDevOps or Ci templates to deploy to AWS based on cloud_service:$CI_CLUSTER_CLOUD_SERVICE_TYPE in the last 28 days.

SaaS/SM: SaaS

Target: TBD

Health: Okay

• Snowplow data instrumented

Metric Name: Users deploying to AWS

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Lessons Learned (June):

• AWS image usage continues to increase.
• The jump in ECS target maintained.

Release:Release - Other - Count of users looking at DORA4 Analytics

Views on the DORA4 page

SaaS/SM: SaaS

Target: TBD

Health: Problem

• Usage is low and we are seeing a steady drop since the peak in January, but it is only for gitlab-org

Metric Name: N/A

Chart (Sisense↗)

Release:Release - Other - Count of CLI creating releases

The count of create_release where release_cli is true

SaaS/SM: SaaS

Target: TBD

Health: Unknown

• We just created this chart

Metric Name: N/A

Chart (Sisense↗)

Lessons Learned (June):

• We have some numbers since mid May!

Release:Release - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• 2021-06-19 - 99.57% Budget already spent
• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Configure:Configure - SMAU, GMAU - MAU of GitLab Managed Terraform State

Monthly active users of GitLab Managed Terraform State

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 7000 monthly active users of GitLab Managed Terraform State

Is Primary PI: true

Health: Okay

• Healthy growth continues

Metric Name: redis_hll_counters.terraform.p_terraform_state_api_unique_users_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/511813/Configure-team-business-metrics
• https://app.periscopedata.com/app/gitlab/677628/Infrastructure-as-Code

Implementation Status: Complete

• We have MAU based metrics.

Lessons Learned (June):

• Insight - Self managed usage is very healthy. We plan an up-sell feature to be developed as one of the first issues for our new frontend engineer. Nota bene, we don't have a design for it to drive adoption further on SaaS.
• Insight - We saw a jump of MAU in all tiers in May. As self-managed instances lag behind SaaS by 3 monthy usually, these might be attributed to the release of the SMAU metric on 22th January.
• Insight - Given the MoM growth, we will likely miss our quarterly target.

Configure:Configure - Adoption PI - Number of Gitlab Kubernetes Agent instances

A rolling count of the number of GitLab Kubernetes Agent instances in the last 28 days

Funnel Description: Handbook Link

SaaS/SM: Both

Target: 200 by the end of Q3FY22

Health: Attention

Metric Name: kubernetes_agents

Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/511813/Configure-team-business-metrics
• https://app.periscopedata.com/app/gitlab/656814/Kubernetes-integration

Implementation Status: Target

• We want to switch to MAU based metric once adoption picks up a bit

Lessons Learned (June):

• Insight - Qualitative feedback shows that the direction was long awaited, but we still miss some features for mass adoption.
• Insight - SaaS saw a jump from 7 to 22 counts between April and May. Given the legacy cert-based was especially restricting for SaaS, we expect faster adoption on SaaS.
• Improvements - We wanted to release the CI/CD tunnel in 14.0, but missed the deadline. It's release is planned for 14.1.
• Insight - Given our current stand, we will very likely miss our quarterly target. We expected an increase in adoption from the CI tunnel, but development took 2 months longer than expected.

Configure:Configure - AMAU - Unique users running pipelines that have a HashiCorp Vault Variable

A rolling count of unique users that have triggered pipelines with a HashiCorp Vault Variable in the last 28 days.

SaaS/SM: Both

Target: no target

Health: Okay

• Blocker - We are short on resources, and I'm currently prioritising other areas that might have only indirect effects on this AMAU.
• Insight The growth continues to be steady.

Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/511813/Configure-team-business-metrics
• https://app.periscopedata.com/app/gitlab/788824/Secrets-Management

Implementation Status: Complete

Configure:Configure - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• This is a new PI and is a shared PI with Infrastructure availability targets
• The dashboards are located in grafana

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Lessons Learned (June):

• 2021-06-19 - 98.98% Availability
• Insight - We are looking into the reasons of missing the budget. An identified issue that eats our budget is planned as a stretch item in 14.1.

Monitor:Monitor - SMAU, GMAU - Unique users that interact with Alerts and Incidents

Unique users that interact with alerts or incidents. (This metric represents the GMAU for the Monitor:Monitor group as well as the SMAU for the Monitor stage. It is based on usage of the Incident Management category which is the only category in the group or stage that is being actively developed. The definition for Monitor SMAU will be seperated from Monitor:Monitor GMAU when we are able to expand the definition to include usage of the other categories. That work is currently not a priority.)

Funnel Description: Handbook Link

SaaS/SM: Both

Target: The target for FY22-Q2 is 18,000 estimated total unique users that interact with alerts and incidents.

Is Primary PI: true

Health: Problem

• 2 months of declining users in SaaS

Metric Name: incident_management_total_unique_counts_monthly

Chart (Sisense↗)

Secondary Chart (Sisense↗)

URL(s)

• https://app.periscopedata.com/app/gitlab/769975/Monitor-Stage-Dashboard

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (June):

• The Monitor SMAU is still largely driven by people creating incidents as an issue types.
• The infrastructure team is looking to adopt Incendent types in their incident response process
• To facilitate the dogfooding, we will prioritize some work for the infra team
• There are some emerging ideas to drive overall usage, such as Alerging on CI/CD pipeline event or deeper integration with Slack. The new Monitor PM will determine path forward.

Monitor:Monitor - Other - Count of projects where incidents are being created

Total count of projects that have created incidents

SaaS/SM: Both

Health: Okay

• Target removed

Metric Name: projects_creating_incidents

Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation complete - issue

Lessons Learned (June):

• 2021-06-19 Ultimate - 53.6k, Core - 34.5k, Premium - 6,112, Starter - 4,742
• Updated the chart from a cumulative chart to just a current count of total projects that have had incidents
• The target no longer make sense. Will choose target in Q3

Monitor:Monitor - AMAU - Alerts Active Users

Count of users performing an action that involves an alert which includes creating an alert, editing an alert, or interacting with features and actions in the side bar of the alert detail page

SaaS/SM: Both

Target: TBD

Health: Attention

• There was a 30% jump in active users, though a relatively small number.

Metric Name: incident_management_alerts_total_unique_user_counts

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (June):

• The custom mapping work for HTTP endpoints is released. Though the fact that we continue to see more users creating incidents from alerts did not appear to influence new alert integration.

Monitor:Monitor - AMAU - Incidents Active Users

Count of users performing an action that involves an incident which includes creating an incident, editing an incident, or interacting with features and actions in the side bar of the incident detail page

SaaS/SM: Both

Target: TBD

Health: Attention

• The growth appears to be flattening.

Metric Name: incident_management_incidents_total_unique_user_counts

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (June):

• Improvements - we are planning improvements to the incident type. Timeline, right hand side features.

Monitor:Monitor - AMAU - On-call users

Count of users who received a notification while on-call

SaaS/SM: Both

Target: TBD

Health: Attention

• Just got instrumented.

Metric Name: incident_management_incidents_total_unique_user_counts

Chart (Sisense↗)

Implementation Status: Complete

• Instrumentation complete - issue.

Lessons Learned (June):

• We have an ultimate user receiving a notification!

Monitor:Monitor - Error Budget for GitLab.com

The error budget provides a clear, objective metric that determines how unreliable the service is allowed to be within a single quarter. Spent budget is the time (in minutes) during which user facing services have experienced a percentage of errors below the specified threshold and latency is above the specified objectives for the service.

Target: 99.95% availability, allowed unavailability window is 20 minutes per month

Health: Problem

• 2021-06-19 - 99.92% availability - overbudget
• The dashboards are located in grafana

Chart (Sisense↗)

Secondary Chart (Sisense↗)

Other PI Pages

• Performance Indicator Maturity Table
• Engineering Function Performance Indicators
  • Customer Support Department Performance Indicators
  • Development Department Performance Indicators
    • Centralized Dashboards
  • Infrastructure Department Performance Indicators
  • Quality Department Performance Indicators
  • Security Department Performance Indicators
  • UX Department Performance Indicators
• Product Performance Indicators
  • Dev Section Performance Indicators
  • Ops Section Performance Indicators
  • Sec Section Performance Indicators
  • Enablement Section Performance Indicators
  • Stage and Group Performance Indicators
• Corporate Finance Performance Indicators
• Talent Acquisition Performance Indicators
• People Success Indicators
• KPI Index

Legends

Health

Value	Level	Meaning
3	Okay	The KPI is at an acceptable level compared to the threshold
2	Attention	This is a blip, or we’re going to watch it, or we just need to enact a proven intervention
1	Problem	We'll prioritize our efforts here
-1	Confidential	Metric & metric health are confidential
0	Unknown	Unknown

How to work with pages like this

Pages

Pages, such as the Engineering Function Performance Indicators page are rendered by an ERB template that contains HTML code.

• Any changes to the Performance Indicator (PI) ERB template will cascade to all links under the Other PI Pages section
• To add, delete, or update (K)PI information (including title, description, etc), make changes to the corresponding Performance Indicators data file. See more details in the Data section below.
• To update the way each individual (K)PI displays:
  • make changes to the Performance Indicators List ERB file for how the (K)PI is displayed on the KPI Index page
  • make changes to the Performance Indicators ERB file for any other properties and its format
• When adding a new PI page,
  • create the new PI page in the business division's handbook section (like the Engineering Division page)
  • include the relative link to the PI page in the Performance Indicators List ERB file
  • add all the (K)PI information into the corresponding Performance Indicators data file

Helper Functions

These ERB templates calls custom helper functions that extract and transform data from the Performance Indicators data file.

• The kpi_list_by_org(org) helper function takes a required string argument named org (deparment or division level) that returns all the KPIs (pi.is_key == true) for a specific organization grouping (pi.org == org) from the Performance Indicators data file.
• The pi_maturity_level(performance_indicator) helper function automatically assigns a maturity level based on the availability of certain data properties for a particular PI.
• The pi_maturity_reasons(performance_indicator) helper function returns a reason for a PI maturity based on other data properties.
• The performance_indicators(org) takes a required string argument named org (deparment or division level) that returns two lists - a list of all KPIs and a list of all PIs for a specific organization grouping (department/division).
• The signed_periscope_url(data) takes in the sisense_data property information from Performance Indicators data files and returns a signed chart URL for embedding a Sisense chart into the handbook.

Data

The heart of pages like this are Performance Indicators data files which are YAML files. Each - denotes a dictionary of values for a new (K)PI. The current elements (or data properties) are:

Property	Type	Description
name	Required	String value of the name of the (K)PI. For Product PIs, product hierarchy should be separate from name by " - " (Ex. {Stage Name}:{Group Name} - {PI Type} - {PI Name}
base_path	Required	Relative path to the performance indicator page that this (K)PI should live on
definition	Required	refer to Parts of a KPI
parent	Optional	should be used when a (K)PI is a subset of another PI. For example, we might care about Hiring vs Plan at the company level. The child would be the division and department levels, which would have the parent flag.
target	Required	The target or cap for the (K)PI. Please use Unknown until we reach maturity level 2 if this is not yet defined. For GMAU, the target should be quarterly.
org	Required	the organizational grouping (Ex: Engineering Function or Development Department). For Product Sections, ensure you have the word section (Ex : Dev Section)
section	Optional	the product section (Ex: dev) as defined in sections.yml
stage	Optional	the product stage (Ex: release) as defined in stages.yml
group	Optional	the product group (Ex: progressive_delivery) as defined in stages.yml
category	Optional	the product group (Ex: feature_flags) as defined in categories.yml
is_key	Required	boolean value (true/false) that indicates if it is a (key) performance indicator
health	Required	indicates the (K)PI health and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
health.level	Optional	indicates a value between 0 and 3 (inclusive) to represent the health of the (K)PI. This should be updated monthly before Key Reviews by the DRI.
health.reasons	Optional	indicates the reasons behind the health level. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
urls	Optional	list of urls associated with the (K)PI. Should be an array (indented lines starting with dashes) even if you only have one url
funnel	Optional	indicates there is a handbook link for a description of the funnel for this PI. Should be a URL
sisense_data	Optional	allows a Sisense dashboard to be embeded as part of the (K)PI using chart, dashboard, and embed as neseted attributes.
sisense_data.chart	Optional	indicates the numeric Sisense chart/widget ID. For example: 9090628
sisense_data.dashboard	Optional	indicates the numeric Sisense dashboard ID. For example: 634200
sisense_data.shared_dashboard	Optional	indicates the numeric Sisense shared_dashboard ID. For example: 185b8e19-a99e-4718-9aba-96cc5d3ea88b
sisense_data.embed	Optional	indicates the Sisense embed version. For example: v2
sisense_data_secondary	Optional	allows a second Sisense dashboard to be embeded. Same as sisense data
sisense_data_secondary.chart	Optional	Same as sisense_data.chart
sisense_data_secondary.dashboard	Optional	Same as sisense_data.dashboard
sisense_data_secondary.shared_dashboard	Optional	Same as sisense_data.shared_dashboard
sisense_data_secondary.embed	Optional	Same as sisense_data.embed
public	Optional	boolean flag that can be set to false where a (K)PI does not meet the public guidelines.
pi_type	Optional	indicates the Product PI type (Ex: AMAU, GMAU, SMAU, Group PPI)
product_analytics_type	Optional	indicates if the metric is available on SaaS, SM (self-managed), or Both.
is_primary	Optional	boolean flag that indicates if this is the Primary PI for the Product Group.
implementation	Optional	indicates the implementation status and reasons as nested attributes. This should be updated monthly before Key Reviews by the DRI.
implementation.status	Optional	indicates the Implementation Status status. This should be updated monthly before Key Reviews by the DRI.
implementation.reasons	Optional	indicates the reasons behind the implementation status. This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one reason.
lessons	Optional	indicates lessons learned from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
lessons.learned	Optional	learned is an attribute that can be nested under lessonsand indicates lessons learned from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one lesson learned
monthly_focus	Optional	indicates monthly focus goals from a K(PI) as a nested attribute. This should be updated monthly before Key Reviews by the DRI.
monthly_focus.goals	Optional	indicates monthly focus goals from a K(PI). This should be updated monthly before Key Reviews by the DRI. Should be an array (indented lines starting with dashes) even if you only have one goal
metric_name	Optional	indicates the name of the metric in Self-Managed implemenation. The SaaS representation of the Self-Managed implementation should use the same name.

Guidelines

Please reference the Engineering Metrics Page for guidelines on chart visualization formatting.