Product Intelligence Guide

1. You are here:
2. Handbook
3. Product Handbook
4. Product Intelligence Guide

Maintained by:

On this page

• Product Intelligence Overview
• Systems Overview
  • GitLab Inc
  • Self-managed
  • Differences between GitLab Inc and Self-managed
• SaaS Data Collection Catalog
  • Frontend Events
    • Collected Data
    • Frontend Event Example Data
  • Backend
    • Backend Event Example Data
      • Service Ping Example Data
  • Data used as identifiers
  • Example User Journey
  • Which Tooling To Use
  • Snowplow
  • Service Ping
  • Database import
  • Reporting level
  • Reporting time period
• Metrics Dictionary
• Implementing Product Performance Indicators
  • 1. Definition
    • Deduplication of Aggregated Metrics
  • 2. Instrumentation
  • 3. Data Availability
  • 4. Dashboard
  • 5. Handbook
  • 6. Target
  • 7. Complete
• Our Commitment to Individual User Privacy in relation to Service Usage Data
• Quick Links

Product Intelligence Overview

At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions.

To get an introduction of Product Analytics at GitLab you have a look at the GitLab Product Data Training (internal) deck

There are several stages and teams involved to go from collecting data to making it useful for our internal teams and customers.

Editable source file

Systems Overview

The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances.

Source file

GitLab Inc

For Product Intelligence purposes, GitLab Inc has three major components:

1. Data Infrastructure: This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors GitLab.com's Snowplow Collector and GitLab's Versions Application.
2. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Service Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application.
3. Monitoring infrastructure: This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana.

Self-managed

For Product Intelligence purposes, self-managed instances have two major components:

1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector.
2. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above.

Differences between GitLab Inc and Self-managed

As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Service Ping, and PostgreSQL database imports all flow into GitLab Inc's data infrastructure. However, on self-managed, only Service Ping flows into GitLab Inc's data infrastructure.

As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure.

Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to.

SaaS Data Collection Catalog

Our SaaS data collection catalog spans both the client (frontend) and server (backend), and uses various tools. We pick-up events and data produced when using the application. By utilizing collected identifiers, we can string these backend and frontend events together to illustrate a GitLab journey at the (1) user (pseudonymized), (2) namespace, and project level.

The below table explains the types of data we collect from GitLab.com and examples for what it can be used. Below we show example data for each collection types.o

Frontend Events

On the client side, a Snowplow JS Tracker is used to track events.

The most up to date schema of the events can be found at https://gitlab.com/gitlab-org/iglu/.

Collected Data

By using the Snowplow JS Tracker we're tracking user-event data like page views or user interactions like clicks.

Examples of what we can gather with this data:

• Number of events clicked on a button, modal, link
• Number of page views on a specific page

Frontend Event Example Data

coming soon

Backend

We use Snowplow Ruby Tracker to gather server-side event data. We also utilize Service Ping to gather aggregated metrics from PostgreSQL database and a Redis in-memory data store. Lastly, the server also contains System Logs which are generated from running the GitLab application.

Backend Event Example Data

coming soon

Service Ping Example Data

A more detailed example of the service ping payload can be found in our docs

{
  "uuid": "0000000-0000-0000-0000-000000000000",
  "hostname": "example.com",
  "version": "12.10.0-pre",
  "installation_type": "omnibus-gitlab",
  "active_user_count": 999,
  "recorded_at": "2020-04-17T07:43:54.162+00:00",
  "edition": "EEU",
  "license_md5": "00000000000000000000000000000000",
  "license_id": null,
  "historical_max_users": 999,
  "licensee": {
    "Name": "ABC, Inc.",
    "Email": "email@example.com",
    "Company": "ABC, Inc."
  },
   "license_user_count": 999,
  "license_starts_at": "2020-01-01",
  "license_expires_at": "2021-01-01",
  "license_plan": "ultimate",
  "license_add_ons": {
  },
  "license_trial": false,
  "counts": {
    "assignee_lists": 999,
    "boards": 999,
    "ci_builds": 999,
    ...
  },
  "container_registry_enabled": true,
  "dependency_proxy_enabled": false,
  ...
}

Data used as identifiers

In order to create the SaaS usage journeys documented below, we collect various identifiers in our data collection catalog. Where the identifier can be used to personally identify a user by someone without permissions to view that information, we will pseudonymize the data via hashing at the collection layer. Follow this epic to track our progress on our pseudonymization project.

Example User Journey

A user signups for a free GitLab.com account and creates a group and/or project (Pseudonymized user_id created and associated with group or project ID), they set up their repo and then view CI/CD and decide they want to invite a colleague to set up this functionality. The newly invited user signs up for GitLab (New pseudonymized user_id created and associated with the existing group or project ID) and sets up CI/CD for their team (backend event).

Why this user journey is valuable to GitLab and our users. In this example, by being able to connect pseudonymized user actions with backend actions we're are able to understand how often teams utilize this adoption path and at what rate they're successful. This will us know what work to prioritize to maximize improvements within the user experience and ensure we're able to understand how impactful these future iterations are.

Which Tooling To Use

✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Planned

Source file

1. Plan-level reporting for Service Ping Redis on SaaS for multi-tenant reporting
2. Plan-level and Group-level reporting for Service Ping Postgres on SaaS for multi-tenant reporting
3. Service Ping Snowplow on Self-Managed (using internal collector and database for moderate volume)
4. Group-level, Project-level, User-level tracking for Snowplow on SaaS
5. Service Ping Snowplow on SaaS (using external collector and database for scaled up volume)

We use three methods to gather product usage data:

• Snowplow
• Service Ping
• Database import

Snowplow

Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application.

Snowplow consists of two components:

• Snowplow JS tracks client-side events.
• Snowplow Ruby tracks server-side events.

For more details, read the Snowplow guide.

Service Ping

Service Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Service Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams.

With the exception of name and email of the license contact associated with customer instances, service ping does not collect personally identifiable (PII) data about the instances users. Service ping simply increments a number when a code path is executed in the application. It can function without any understanding of who users are.

For more details, read the Service Ping guide.

Database import

Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the data team handbook.

Reporting level

Our reporting levels of aggregate or individual reporting varies by segment. For example, on Self-Managed Users, we can report at an aggregate user level using Service Ping but not on an Individual user level.

Reporting time period

Our reporting time periods varies by segment. For example, on Self-Managed Users, we can report all time counts and 28 day counts in Service Ping.

Metrics Dictionary

The Metrics Dictionary is a single source of truth for the metrics and events we collect for product usage data. The Metrics Dictionary lists all the metrics we collect in Service Ping, why we're tracking them, and where they are tracked.

The Metrics Dictionary should be updated any time a new metric is implemented, updated, or removed.

We're currently focusing our Metrics Dictionary on Service Ping. In the future, we will also include Snowplow. We currently have an initiative across the entire product organization to complete the Metrics Dictionary for Service Ping.

Implementing Product Performance Indicators

We've recently had a large push across the product organization to become more data driven. Part of this push includes getting product metrics in place for each product section, stage, and group. In FY21-Q3 OKRs, we setup a couple of OKRs to help us accomplish this:

• 100% of groups have Future GMAU (or Paid GMAU) with instrumentation, dashboards, and quarterly targets
• 100% of stages have Future SMAU and Paid SMAU with instrumentation, dashboards, and quarterly targets.

To accomplish these OKRs, we setup a seven step process to implement product metrics. This process was originally presented in the Weekly Product Meeting on August 11, 2020 (slide deck and video presentation) and has been refined over time.

1. Definition

Determine what metrics are important for your specific section, stage, or group.

Instructions:

1. Determine the level at which the metric should be measured:
   1. All product level - TMAU, Paid TMAU
   2. Stage level - SMAU, Paid SMAU, Other PI
   3. Group level - GMAU, Paid GMAU, Other PI
2. Define the metric. This is typically done by selecting an initial AMAU that is most representative of overall stage/group use
3. Determine the right collection framework to utilize. Some guidance for considering collection frameworks:
   • Service Ping - General - There are many frameworks that report via Service Ping. All of them have various capabilities regarding types of events and available segementation. All Service Ping collection frameworks have a frequency of monthly for self-managed usage and every-other-week for SaaS usage.
   • Snowplow - Investment in instrumentation should be discouraged - there is already great URL and path tracking from SnowPlow for Gitlab.com. If additional tracking is needed on front-end events consider utilizing Service Ping Redis HLL. Snowplow instrumentation becomes immediately available on GitLab.com only.
   • Service Ping - Redis HLL - Redis HLL can track distinct counts, typically in front-end metrics on both the client (Javascript) and server (Ruby) side. Redis HLL does not provide session level segmentation on GitLab.com. As Usage Ping instrumentation it is returned from both SaaS and Self-Managed.
   • Service Ping - Postgres - Our original Service Ping implementation this collects distinct database counts.
   • Service Ping - Prometheus - Note there is no future plan to increase the segmentation of this collection framework and it is presently limited to Instance wide metrics.
4. Plan your future events and metrics with your engineering team and add them to the Event Dictionary.
5. Add the value from column A of the Event Dictionary to the Performance Indicator (PI) yaml file as the metric_name

Deduplication of Aggregated Metrics

Note: We now enable you to deduplicate aggregated metrics implemented via Redis HLL, in order to get distinct counts (ex distinct users count across multiple actions in a stage). Please read our docs on Aggregated Metrics for more information. We are working towards the ability to deduplicate across multiple Database HLL metrics via #288848, and then deduplication across multiple Redis HLL and Database HLL metrics via #421

2. Instrumentation

Work with your engineering team to instrument tracking for your XMAU. Focus on using Service Ping as your metrics will be available on both SaaS and self-managed.

The Service Ping Guide outlines the steps required for instrumentation. It includes:

• What is Service Ping?
• How Service Ping works
• Implementing Service Ping
• Example Payload

Also see the Product Intelligence Guide and Snowplow Guide.

Instructions:

1. Read the docs and work with your engineers on instrumentation.
2. Ask for a Product Intelligence Review in your MR.

3. Data Availability

Plan instrumentation with sufficient lead time for data availability. Ensure your metrics make it into the self-managed release as early as possible.

Timeline:

1. Self-managed releases happen on the 22nd of each month (+30 days)
2. Wait one week for customers to upgrade to the new release and for a Service Ping to be generated (+7 days)
3. Service Pings are collected in the Versions application. The Versions application's database is automatically imported into the Snowflake Data Warehouse every day (+1 day).

In total, plan for up to 38 day cycle times (Examples 1, 2). Cycle times are slow with monthly releases and weekly pings, so, implement your metrics early.

Instructions:

1. Merge your metrics into the next self-managed release as early as possible.
2. Wait for your metrics to be released onto production GitLab.com. These releases currently happen on a daily basis.
3. Usage Pings are generated on GitLab.com on a weekly basis. Monitor the #g_product_intelligence slack channel where the Product Intelligence team will post the latest GitLab.com Usage Ping (example). Verify your new metrics are present in the GitLab.com Usage Ping payload.
4. Wait for the Versions database to be imported into the data warehouse.
5. Check the dbt model version_usage_data_unpacked to ensure the database column for your metric is present.
6. Check Sisense to ensure data is available in the data warehouse.

4. Dashboard

Dashboard the metric. This is done by creating a Sisense dashboard. Avoid cumulative views and instead focus on month-over-month growth. Instructions for creating dashboards are here.

We need PMs to self-serve their own dashboards as data team capacity is limited. The data team will be focused on enabling self-service, advising PMs, and working on the more challenging XMAU dashboards.

To learn how to create your own dashboard, see Data For Product Managers: Creating Charts

To update the SMAU Summary Dashboards: GitLab.com Postgres SMAU Dashboard (SaaS) and Service Ping SMAU Dashboard (SaaS + Self-Managed), please open a data team issue.

Dashboard Prioritization

For GMAU and SMAU data issues:

• In Q3, due to very limited data team capacity, the data team capacity will be reserved primarily for GMAU and SMAU.
  • Please add XMAU label to the data issues.
  • Please limit the ask to XMAU only, rather than all the supporting metrics.
• If there is still need to prioritize within GMAU issues, we will work with Scott and Section Leaders on the prioritization.
  • Section Leaders are encourage to rank XMAU Issues on the Product Section Board.

For non-GMAU and non-SMAU data issues:

• In Q3, the data team will have very limited capacity to support non-GMAU and non-SMAU data requests.
  • Please continue to create data issues and label Data issues with the appropriate Product Section Label.
• If there are critical or urgent data asks, please @hilaqu and your section leader in the issue, with an explanation of why. We will evaluate them on a case-by-case basis.

Instructions:

1. Read through Data For Product Managers: Creating Charts
2. Self-serve your dashboard
3. If you need help, create a data team issue and ask your Section Leader to prioritize.

5. Handbook

There are five Product PI pages: The Product Team page and section pages for Dev, Ops, Sec, Enablement.

We need all PMs to ensure their PIs are showing on the performance indicator pages. Based off What we're aiming for

To do so, we need a clear way to communicate to PMs exactly which PIs are remaining. We will be adding placeholder PIs for each section into the performance indicator file so that all required PIs show in the handbook. Once a PI is implemented, the actual PI will replace the placeholder PI.For more information about how PIs and XMAUs are related to one another, see PI Structure.

Instructions:

1. @jeromezng @kathleentam will add placeholder PIs for each section.
2. @jeromezng @kathleentam will then meet with each Section, Stage, and Group, to understand the implementation status of each PI and document any exceptions. Exceptions for PIs will then be signed off by Scott, Anoop, and the Section Leaders.
3. Keep your performance indicator up to date as the implementation status changes.

6. Target

As a product organization, we need to get into the habit of understanding our baselines and setting targets for each stage & group. For the PI Target step, you will work with your Section or Group Leader to define targets for each of your XMAUs.

Set a growth target, and embed in the tracking dashboard. Growth targets should be ambitious but achievable.

Instructions:

1. Work with your Section or Group Leader to define a target.
2. Add the Target Line into your dashboard (example).
3. If you need help, book a meeting with @jeromezng @kathleentam.

7. Complete

All of the prior steps have been completed and a PI is successfully implemented.

Instructions

1. Continually monitor your results, and refine your priorities based on results.
2. Line up your roadmap priorities to positively influence the adoption funnel, drive the performance metric, and hit the growth targets.
3. Deeply understand the drivers of your metric through customer interviews and other product usage data. Visualize this through a user adoption funnel that describes how users enjoy the value of the particular product area you are measuring.

Our Commitment to Individual User Privacy in relation to Service Usage Data

While collecting service usage data is important to our business, GitLab equally values the privacy of its users. Learn more about our data pseudonymization process, commitment to our users, and how we utilize service usage data here.

Quick Links