Product Intelligence Guide

1. You are here:
2. Handbook
3. Product Handbook
4. Product Intelligence Guide

Maintained by:

On this page

• Product Intelligence Overview
  • 1. Privacy Policy and Settings
  • 2. Data Collection
  • 3. Processing Pipeline
  • 4. Enable Product Teams
  • 5. Enable Sales and Customer Success
  • 6. Enable Full Funnel Analytics
  • 7. Enhance Value Stream Management Feature
  • Teams Involved
• Systems overview
  • GitLab Inc
  • Self-managed
  • Differences between GitLab Inc and Self-managed
• Collection Framework
  • Snowplow
  • Usage Ping
  • Database import
  • Pageview events
  • UI events
  • CRUD or API events
  • Database records
  • Instance settings
  • Integration settings
  • Reporting level
  • Reporting time period
• Event Dictionary
  • Instructions
  • Planned metrics and events
• Implementing Product Performance Indicators
  • 1. Definition
    • Deduplication of Aggregated Metrics
  • 2. Instrumentation
  • 3. Data Availability
  • 4. Dashboard
  • 5. Handbook
  • 6. Target
  • 7. Complete
• Quick Links

Product Intelligence Overview

At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions.

There are several steps involved to go from collecting data to making it useful for our internal teams and customers.

1. Privacy Policy and Settings

2. Data Collection

3. Processing Pipeline

4. Enable Product Teams

5. Enable Sales and Customer Success

6. Enable Full Funnel Analytics

7. Enhance Value Stream Management Feature

Teams Involved

We work closely with other internal teams in each step in the Product Intelligence Process. The teams involved at each step are:

Editable source file

Systems overview

The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances.

Source file

GitLab Inc

For Product Intelligence purposes, GitLab Inc has three major components:

1. Data Infrastructure: This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors GitLab.com's Snowplow Collector and GitLab's Versions Application.
2. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application.
3. Monitoring infrastructure: This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana.

Self-managed

For Product Intelligence purposes, self-managed instances have two major components:

1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector.
2. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above.

Differences between GitLab Inc and Self-managed

As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data infrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure.

As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure.

Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to.

Collection Framework

✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Planned

Source file

1. Plan-level reporting for Usage Ping Redis on SaaS for multi-tenant reporting
2. Plan-level and Group-level reporting for Usage Ping Postgres on SaaS for multi-tenant reporting
3. Usage Ping Snowplow on Self-Managed (using internal collector and database for moderate volume)
4. Group-level, Project-level, User-level tracking for Snowplow on SaaS
5. Usage Ping Snowplow on SaaS (using external collector and database for scaled up volume)

We use three methods to gather product usage data:

• Snowplow
• Usage Ping
• Database import

Snowplow

Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application.

Snowplow consists of two components:

• Snowplow JS tracks client-side events.
• Snowplow Ruby tracks server-side events.

For more details, read the Snowplow guide.

Usage Ping

Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams.

For more details, read the Usage Ping guide.

Database import

Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the data team handbook.

Pageview events

• Number of sessions that visited the /dashboard/groups page

UI events

• Number of sessions that clicked on a button or link
• Number of sessions that closed a modal

UI events are any interface-driven actions from the browser including click data.

CRUD or API events

• Number of Git pushes
• Number of GraphQL queries
• Number of requests to a Rails action or controller

These are backend events that include the creation, read, update, deletion of records, and other events that might be triggered from layers other than those available in the interface.

Database records

These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in structure.sql.

Instance settings

These are settings of your instance such as the instance's Git version and if certain features are enabled such as container_registry_enabled.

Integration settings

These are integrations your GitLab instance interacts with such as an external storage provider or an external container registry. These services must be able to send data back into a GitLab instance for data to be tracked.

Reporting level

Our reporting levels of aggregate or individual reporting varies by segment. For example, on Self-Managed Users, we can report at an aggregate user level using Usage Ping but not on an Individual user level.

Reporting time period

Our reporting time periods varies by segment. For example, on Self-Managed Users, we can report all time counts and 28 day counts in Usage Ping.

Event Dictionary

Note: We've temporarily moved the Event Dictionary to a Google Sheet. The previous Markdown table exceeded 500+ rows making it difficult to manage. In the future, our intention is to move this back into our docs using a YAML file.

The event dictionary is a single source of truth for the metrics and events we collect for product usage data. The Event Dictionary lists all the metrics and events we track, why we're tracking them, and where they are tracked.

This is a living document that is updated any time a new event is planned or implemented. It includes the following information.

• Section, stage, or group
• Description
• Implementation status
• Availability by plan type
• Code path

We're currently focusing our Event Dictionary on Usage Ping. In the future, we will also include Snowplow. We currently have an initiative across the entire product organization to complete the Event Dictionary for Usage Ping.

Instructions

1. Open the Event Dictionary and fill in all the PM to edit columns highlighted in yellow.
2. Check that all the metrics and events are assigned to the correct section, stage, or group. If a metric is used across many groups, assign it to the stage. If a metric is used across many stages, assign it to the section. If a metric is incorrectly assigned to another section, stage, or group, let the PM know you have reassigned it. If your group has no assigned metrics and events, check that your metrics and events are not incorrectly assigned to another PM.
3. Add descriptions of what your metrics and events are tracking. Work with your Engineering team or the Product Intelligence team if you need help understanding this.
4. Add what plans this metric is available on. Work with your Engineering team or the Product Intelligence team if you need help understanding this.

Planned metrics and events

For future metrics and events you plan to track, please add them to the Event Dictionary and note the status as Planned, In Progress, or Implemented. Once you have confirmed the metric has been implemented and have confirmed the metric data is in our data warehouse, change the status to Data Available.

Implementing Product Performance Indicators

We've recently had a large push across the product organization to become more data driven. Part of this push includes getting product metrics in place for each product section, stage, and group. In FY21-Q3 OKRs, we setup a couple of OKRs to help us accomplish this:

• 100% of groups have Future GMAU (or Paid GMAU) with instrumentation, dashboards, and quarterly targets
• 100% of stages have Future SMAU and Paid SMAU with instrumentation, dashboards, and quarterly targets.

To accomplish these OKRs, we setup a seven step process to implement product metrics. This process was originally presented in the Weekly Product Meeting on August 11, 2020 (slide deck and video presentation) and has been refined over time.

1. Definition

Determine what metrics are important for your specific section, stage, or group.

Instructions:

1. Determine the level at which the metric should be measured:
   1. All product level - TMAU, Paid TMAU
   2. Stage level - SMAU, Paid SMAU, Other PI
   3. Group level - GMAU, Paid GMAU, Other PI
2. Define the metric. This is typically done by selecting an initial AMAU that is most representative of overall stage/group use
3. Determine the right collection framework to utilize. Some guidance for considering collection frameworks:
   • Usage Ping - General - There are many frameworks that report via Usage Ping. All of them have various capabilities regarding types of events and available segementation. All Usage Ping collection frameworks have a frequency of monthly for self-managed usage and every-other-week for SaaS usage.
   • Snowplow - Investment in instrumentation should be discouraged - there is already great URL and path tracking from SnowPlow for Gitlab.com. If additional tracking is needed on front-end events consider utilizing Usage Ping Redis HLL. Snowplow instrumentation becomes immediately available on GitLab.com only.
   • Usage Ping - Redis HLL - Redis HLL can track distinct counts, typically in front-end metrics on both the client (Javascript) and server (Ruby) side. Redis HLL does not provide session level segmentationon GitLab.com. As Usage Ping instrumentation it is returned from both SaaS and Self-Managed.
   • Usage Ping - Postgres - Our original Usage Ping implementation this collects distinct database counts.
   • Usage Ping - Prometheus - Note there is no future plan to increase the segmentation of this collection framework and it is presently limited to Instance wide metrics.
4. Plan your future events and metrics with your engineering team and add them to the Event Dictionary.
5. Add the value from column A of the Event Dictionary to the Performance Indicator (PI) yaml file as the metric_name

Deduplication of Aggregated Metrics

Note: We now enable you to deduplicate aggregated metrics implemented via Redis HLL, in order to get distinct counts (ex distinct users count across multiple actions in a stage). Please read our docs on Aggregated Metrics for more information. We are working towards the ability to deduplicate across multiple Database HLL metrics via #288848, and then deduplication across multiple Redis HLL and Database HLL metrics via #421

2. Instrumentation

Work with your engineering team to instrument tracking for your XMAU. Focus on using Usage Ping as your metrics will be available on both SaaS and self-managed.

The Usage Ping Guide outlines the steps required for instrumentation. It includes:

• What is Usage Ping?
• How Usage Ping works
• Implementing Usage Ping
• Developing and Testing Usage Ping
• Example Payload

Also see the Product Intelligence Guide and Snowplow Guide.

Instructions:

1. Read the docs and work with your engineers on instrumentation.
2. Ask for a Product Intelligence Review in your MR.

3. Data Availability

Plan instrumentation with sufficient lead time for data availability. Ensure your metrics make it into the self-managed release as early as possible.

Timeline:

1. Self-managed releases happen on the 22nd of each month (+30 days)
2. Wait one week for customers to upgrade to the new release and for a Usage Ping to be generated (+7 days)
3. Usage Pings are collected in the Versions application. The Versions database is imported into the Snowflake Data Warehouse manually every two weeks (+14 days). Note that we’re working to automate this to daily imports (bringing this down to +1 day instead).

In total, plan for up to 51 day cycle times (Examples 1, 2). Cycle times are slow with monthly releases and weekly pings, so, implement your metrics early.

Instructions:

1. Merge your metrics into the next self-managed release as early as possible.
2. Wait for your metrics to be released onto production GitLab.com. These releases currently happen on a daily basis.
3. Usage Pings are generated on GitLab.com on a weekly basis. Monitor the #g_product_analytics slack channel where the Product Intelligence team will post the latest GitLab.com Usage Ping (example). Verify your new metrics are present in the GitLab.com Usage Ping payload.
4. Wait for the Versions database to be imported into the data warehouse.
5. Check the dbt model version_usage_data_unpacked to ensure the database column for your metric is present.
6. Check Sisense to ensure data is available in the data warehouse.

4. Dashboard

Dashboard the metric. This is done by creating a Sisense dashboard. Avoid cumulative views and instead focus on month-over-month growth. Instructions for creating dashboards are here.

We need PMs to self-serve their own dashboards as data team capacity is limited. The data team will be focused on enabling self-service, advising PMs, and working on the more challenging XMAU dashboards.

To learn how to create your own dashboard, see Data For Product Managers: Creating Charts

To update the SMAU Summary Dashboards: GitLab.com Postgres SMAU Dashboard (SaaS) and Usage Ping SMAU Dashboard (SaaS + Self-Managed), please open a data team issue.

Dashboard Prioritization

For GMAU and SMAU data issues:

• In Q3, due to very limited data team capacity, the data team capacity will be reserved primarily for GMAU and SMAU.
  • Please add XMAU label to the data issues.
  • Please limit the ask to XMAU only, rather than all the supporting metrics.
• If there is still need to prioritize within GMAU issues, we will work with Scott and Section Leaders on the prioritization.
  • Section Leaders are encourage to rank XMAU Issues on the Product Section Board.

For non-GMAU and non-SMAU data issues:

• In Q3, the data team will have very limited capacity to support non-GMAU and non-SMAU data requests.
  • Please continue to create data issues and label Data issues with the appropriate Product Section Label.
• If there are critical or urgent data asks, please @hilaqu and your section leader in the issue, with an explanation of why. We will evaluate them on a case-by-case basis.

Instructions:

1. Read through Data For Product Managers: Creating Charts
2. Self-serve your dashboard
3. If you need help, create a data team issue and ask your Section Leader to prioritize.

5. Handbook

There are five Product PI pages: The Product Team page and section pages for Dev, Ops, Sec, Enablement.

We need all PMs to ensure their PIs are showing on the performance indicator pages. Based off What we're aiming for

To do so, we need a clear way to communicate to PMs exactly which PIs are remaining. We will be adding placeholder PIs for each section into the performance indicator file so that all required PIs show in the handbook. Once a PI is implemented, the actual PI will replace the placeholder PI.For more information about how PIs and XMAUs are related to one another, see PI Structure.

Instructions:

1. @jeromezng @kathleentam will add placeholder PIs for each section.
2. @jeromezng @kathleentam will then meet with each Section, Stage, and Group, to understand the implementation status of each PI and document any exceptions. Exceptions for PIs will then be signed off by Scott, Anoop, and the Section Leaders.
3. Keep your performance indicator up to date as the implementation status changes.

6. Target

As a product organization, we need to get into the habit of understanding our baselines and setting targets for each stage & group. For the PI Target step, you will work with your Section or Group Leader to define targets for each of your XMAUs.

Set a growth target, and embed in the tracking dashboard. Growth targets should be ambitious but achievable.

Instructions:

1. Work with your Section or Group Leader to define a target.
2. Add the Target Line into your dashboard (example).
3. If you need help, book a meeting with @jeromezng @kathleentam.

7. Complete

All of the prior steps have been completed and a PI is successfully implemented.

Instructions

1. Continually monitor your results, and refine your priorities based on results.
2. Line up your roadmap priorities to positively influence the adoption funnel, drive the performance metric, and hit the growth targets.
3. Deeply understand the drivers of your metric through customer interviews and other product usage data. Visualize this through a user adoption funnel that describes how users enjoy the value of the particular product area you are measuring.

Quick Links