Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Product Direction - Product Analytics

On this page

Vision

The vision of Product Analytics is to help us build a better Gitlab. We collect data about how GitLab is used to understand what product features to build, to understand how to serve our customers, and to help our customers understand their DevOps lifecycle.

Roadmap

Privacy Policy and Settings - Rollout Updated Privacy Policy

Product Analytics Steps: Privacy Policy, Privacy Settings

Teams Involved: Legal, Product Analytics, Data

Why This Is Important:

Recent Progress:

What's Next:

Data Collection - Maintain and Scale Our Collection Framework

Product Analytics Steps: Collection Framework, Event Dictionary, Instrumentation

Teams Involved: Product Analytics

Why This Is Important:

Recent Progress:

What's Next:

Data Collection - Snowplow Event Tracking on Self Managed

Product Analytics Steps: Collection Framework, Event Dictionary, Instrumentation

Teams Involved: Product Analytics

Why This Is Important:

Recent Progress:

What's Next:

Processing Pipeline - Decrease Cycle Times for Product Analytics

Product Analytics Steps: Release Cycle, Product Usage, Usage Ping Generation, Collectors, Extractors, Loaders, Snowflake Data Warehouse, dbt Data Model

Teams Involved: Product Analytics, Data, Product Managers

Why This Is Important:

Recent Progress:

What's Next:

Processing Pipeline - Plan and Group-level Reporting for SaaS

Product Analytics Steps: Collection Framework, Usage Ping Generation, Collectors, Processors, Snowflake Data Warehouse, dbt Data Model

Teams Involved: Product Analytics, Data

Why This Is Important:

Recent Progress:

What's Next:

Enable Product Teams with Product Analytics - Lead FY21-Q4 Product OKR1 - KR1 100% of DevOps groups have Projected GMAU (or Paid GMAU) with instrumentation, dashboards, and quarterly targets

Product Analytics Steps: Instrumentation, Sisense Dashboard, Performance Indicators, Metrics Reviews, Product Improvements

Teams Involved: Product Managers, Product Analytics, Data

Why This Is Important:

Recent Progress:

What's Next:

Enable Sales and Customer Success Teams with Product Analytics

Product Analytics Steps: Salesforce Data Feed, Salesforce Dashboard, Gainsight Dashboard, Customer Conversations

Teams Involved: Sales, Customer Success, Product Analytics, Data

Why This Is Important:

Recent Progress:

What's Next:

Enable Full Funnel Analytics

Product Analytics Steps: Marketing Data, Product Data, Fulfillment Data, Sales Data, Customer Success Data, Full Funnel Analysis

Teams Involved: Marketing, Fulfillment, Sales, Customer Success, Product Analytics, Data, Enterprise Applications

Why This Is Important:

Recent Progress:

What's Next:

Make Product Analytics Data Useful For Customers

Product Analytics Steps: Value Stream Data, Value Stream Features

Teams Involved: Manage, Product Analytics

Why This Is Important:

Recent Progress:

What's Next:

Customer Product Analytics Feature

Product Analytics Steps: Collection Framework

Teams Involved: Product Analytics

Why This Is Important:

Recent Progress:

What's Next:

Product Analytics Overview

At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions.

There are several steps involved to go from collecting data to making it useful for our internal teams and customers.

1) Privacy Policy and Settings

graph LR privacy_policy[Privacy Policy] --> data_classification_policy[Data Classification Policy] data_classification_policy[Data Classification Policy] --> privacy_settings[Privacy Settings]

2) Data Collection

graph LR collection_framework[Collect Framework] --> event_dictionary[Event Dictionary] event_dictionary[Event Dictionary] --> instrumentation[Instrumentation] instrumentation[Instrumentation] --> release_cycle[Release Cycle] release_cycle[Release Cycle] --> events_fired[Events Fired] events_fired[Events Fired] --> usage_ping_generated[Usage Ping Generated]

3) Processing Pipeline

graph LR analytics_collectors[Analytics Collectors] --> data_imports[Data Imports] data_imports[Data Imports] --> data_warehouse[Data Warehouse] data_warehouse[Data Warehouse] --> data_modelling[Data Modelling]

4) Enable Product Teams with Product Analytics

graph LR sisense_dashboard[Sisense Dashboard] --> performance_indicators[Performance Indicators] performance_indicators[Performance Indicators] --> pi_target[PI Target] pi_target[PI Target] --> pi_reviews[PI Reviews] pi_reviews[PI Reviews] --> improve_product[Improve Product]

5) Enable Sales and Customer Success Teams with Product Analytics

graph LR salesforce_data_feed[Salesforce Data Feed] --> salesforce_dashboard[Salesforce Dashboard] salesforce_dashboard[Salesforce Dashboard] --> gainsight_dashboard[Gainsight Dashboard] gainsight_dashboard[Gainsight Dashboard] --> customer_conversations[Customer Conversations]

6) Enable Full Funnel Analytics

graph LR marketing_data[Marketing Data] --> product_data[Product Data] product_data[Product Data] --> fulfillment_data[Fulfillment Data] fulfillment_data[Fulfillment Data] --> sales_data[Sales Data] sales_data[Sales Data] --> customer_success_data[Customer Success Data] customer_success_data[Customer Success Data] --> full_funnel_analysis[Full Funnel Analysis]

7) Make Product Analytics Data Useful For Customers

graph LR value_stream_data_feed[Value Stream Data Feed] --> value_stream_features[Value Stream Features]

8) Customer Product Analytics Feature

Teams Involved

We work closely with other internal teams in each step in the Product Analytics Process. The teams involved at each step are:

Step Description Teams Involved
Privacy Policy Outlines what product usage data we collect from our users. Legal, Product Analytics, Data
Data Classification Policy Outlines our data classification levels based on sensitivity. Legal, Security, Product Analytics, Data
Privacy Settings In app settings for users to control what data they share with GitLab. Legal, Product Analytics, Data
Collection Framework Outlines our available data collection tools. Product Analytics
Event Dictionary A single source of truth defining all product metrics and events. Product Managers, Product Analytics
Instrumentation Instrumentation of feature tracking done by each product and engineering team. Product Managers, Product Analytics
Release Cycle The release of GitLab code. We have daily releases for SaaS and monthly releases for self-managed. Product Managers, Product Analytics
Product Usage The use of the product by users. Product Managers, Product Analytics
Usage Ping Generation A weekly job that aggregates and sends product usage data to GitLab. Product Analytics
Collectors The collection of Snowplow and Usage Ping data. Product Analytics, Data
Processors The processing of Usage Ping data. Product Analytics, Data
Enrichers The enrichment of Snowplow data. Product Analytics, Data
Extractors The extraction of Usage Ping and Snowplow data sources. Product Analytics, Data
Loaders The loading of Usage Ping and Snowplow data into the data warehouse. Product Analytics, Data
Snowflake Data Warehouse Where our product usage data is stored. Product Analytics, Data
dbt Data Model How we transform raw data into a data structure that's ready for analysis. Product Analytics, Data
Sisense Dashboards The building of Sisense dashboards. Product Managers, Product Analytics, Data
Performance Indicators The product metrics GitLab's product team pays attention to. Product Managers, Product Analytics, Data
Metrics Reviews Monthly reviews of GitLab's product metrics. Product Managers, Product Analytics, Data
Product Improvements Make product improvements based on product usage data. Product Managers
Salesforce Data Feed Data feed of product usage data into sales systems. Sales, Customer Success, Product Analytics, Data
Salesforce Dashboards Dashboard embeded into each Salesforce Customer showing product usage. Sales, Customer Success, Product Analytics, Data
Gainsight Dashboards Dashboard embeded into each Gainsight Customer showing product usage. Sales, Customer Success, Product Analytics, Data
Customer Conversations Better customer conversrations using product usage data. Sales, Customer Success
Marketing Data Marketing data sources such as Google Analytics and Marketo. Marketing, Product Analytics, Data, Enterprise Applications
Product Data Product data sources such as GitLab.com Postgres, Usage Ping data, and Snowplow data. Product Analytics, Data, Enterprise Applications
License and Subscription Data License and subscription data sources such as Customers app, License app, and Zuora data. Fulfillment, Product Analytics, Data, Enterprise Applications
Sales Data Sales data sources such as Salesforce. Sales, Product Analytics, Data, Enterprise Applications
Customer Success Data Customer Success data sources such as Gainsight. Customer Success, Product Analytics, Data, Enterprise Applications
Full Funnel Analysis The integration of all Marketing, Product, License and Subscription, Sales, and Customer Success Data. Marketing, Fulfillment, Sales, Customer Success, Product Analytics, Data, Enterprise Applications
Value Stream Data Feed A data feed for passing benchmark product usage data back to customers. Manage, Product Analytics
Value Stream Features Features to show customers how they're using GitLab. Manage, Product Analytics

Editable source file

Collection Framework

βœ… Available, πŸ”„ In Progress, πŸ“… Planned, βœ–οΈ Not Planned

Source file

  1. Plan-level reporting for Usage Ping Redis on SaaS for multi-tenant reporting
  2. Plan-level and Group-level reporting for Usage Ping Postgres on SaaS for multi-tenant reporting
  3. Usage Ping Snowplow on Self-Managed (using internal collector and database for moderate volume)
  4. Group-level, Project-level, User-level tracking for Snowplow on SaaS
  5. Usage Ping Snowplow on SaaS (using external collector and database for scaled up volume)

Implementing Product Metrics

We've recently had a large push across the product organization to become more data driven. Part of this push includes getting product metrics in place for each product section, stage, and group. In FY21-Q3 OKRs, we setup a couple of OKRs to help us accomplish this:

To accomplish these OKRs, we setup a seven step process to implement product metrics. This process was originally presented in the Weekly Product Meeting on August 11, 2020 (slide deck and video presentation) and has been refined over time.

Implementation Status Description Responsibility
Definition The definition step outlines the process for deciding which product metrics to track. PM Responsibility, Product Analytics Support
Instrumentation The instrumentation step outlines how each product team implements data collection. PM Responsibility, Product Analytics Support
Data Availability The data availability step outlines the timing of a product release to receiving product usage data in the data warehouse. PM Responsibility, Product Analytics Support
Dashboard The dashboarding step outlines how Sisense dashboards are built. PM Responsibility, Product Analytics Support
Handbook The handbook PI page describes how product performance indicators are added for each product section, stage, and group. PM Responsibility, Product Analytics Support
Target The target definition step outlines how targets are defined for each performance indicator. PM Responsibility, Product Analytics Support
Complete All of the prior steps have been completed. PM Responsibility, Product Analytics Support

1. Definition

Determine what metrics are important for your specific section, stage, or group.

Instructions:

  1. Determine the level at which the metric should be measured:
    1. All product level - TMAU, Paid TMAU
    2. Stage level - SMAU, Paid SMAU, Other PI
    3. Group level - GMAU, Paid GMAU, Other PI
  2. Define the metric. This is typically done by selecting an initial AMAU that is most representative of overall stage/group use
  3. Determine the right collection framework to utilize. Some guidance for considering collection frameworks:
    • Usage Ping - General - There are many frameworks that report via Usage Ping. All of them have various capabilities regarding types of events and available segementation. All Usage Ping collection frameworks have a frequency of monthly for self-managed usage and every-other-week for SaaS usage.
    • Snowplow - Investment in instrumentation should be discouraged - there is already great URL and path tracking from SnowPlow for Gitlab.com. If additional tracking is needed on front-end events consider utilizizing Usage Ping Redis HLL. Snowplow instrumentation becomes immediately available on GitLab.com only.
    • Usage Ping - Redis HLL - Redis HLL can track distinct counts, typically in front-end metrics on both the client (Javasciprt) and server (Ruby) side. Redis HLL does not provide session level segmentationon GitLab.com. As Usage Ping instrumentation it is returned from both SaaS and Self-Managed.
    • Usage Ping - Postgres - Our original Usage Ping implementation this collects distinct database counts.
    • Usage Ping - Prometheus - Note there is no future plan to increase the segmentation of this collection framework and it is presently limited to Instance wide metrics.
  4. Plan your future events and metrics with your engineering team and add them to the Event Dictionary.

2. Instrumentation

Work with your engineering team to instrument tracking for your XMAU. Focus on using Usage Ping as your metrics will be available on both SaaS and self-managed.

The Usage Ping Guide outlines the steps required for instrumentation. It includes:

Also see the Product Analytics Guide and Snowplow Guide.

Instructions:

  1. Read the docs and work with your engineers on instrumentation.
  2. Ask for a Product Analytics Review in your MR.

3. Data Availability

Plan instrumentation with sufficient lead time for data availability. Ensure your metrics make it into the self-managed release as early as possible.

Timeline:

  1. Self-managed releases happen on the 22nd of each month (+30 days)
  2. Wait one week for customers to upgrade to the new release and for a Usage Ping to be generated (+7 days)
  3. Usage Pings are collected in the Versions application. The Versions database is imported into the Snowflake Data Warehouse manually every two weeks (+14 days). Note that we’re working to automate this to daily imports (bringing this down to +1 day instead).

In total, plan for up to 51 day cycle times (Examples 1, 2). Cycle times are slow with monthly releases and weekly pings, so, implement your metrics early.

Instructions:

  1. Merge your metrics into the next self-managed release as early as possible.
  2. Wait for your metrics to be released onto production GitLab.com. These releases currently happen on a daily basis.
  3. Usage Pings are generated on GitLab.com on a weekly basis. Monitor the #g_product_analytics slack channel where the Product Analytics team will post the latest GitLab.com Usage Ping (example). Verify your new metrics are present in the GitLab.com Usage Ping payload.
  4. Wait for the Versions database to be imported into the data warehouse.
  5. Check the dbt model version_usage_data_unpacked to ensure the database column for your metric is present.
  6. Check Sisense to ensure data is available in the data warehouse.

4. Dashboard

Dashboard the metric. This is done by creating a Sisense dashboard. Avoid cumulative views and instead focus on month-over-month growth. Instructions for creating dashboards are here.

We need PMs to self-serve their own dashboards as data team capacity is limited. The data team will be focused on enabling self-service, advising PMs, and working on the more challenging XMAU dashboards.

To learn how to create your own dashboard, see Data For Product Managers: Creating Charts

To update the SMAU Summary Dashboards: GitLab.com Postgres SMAU Dashboard (SaaS) and Usage Ping SMAU Dashboard (SaaS + Self-Managed), please open a data team issue.

Dashboard Prioritization

For GMAU and SMAU data issues:

For non-GMAU and non-SMAU data issues:

Instructions:

  1. Read through Data For Product Managers: Creating Charts
  2. Self-serve your dashboard
  3. If you need help, create a data team issue and ask your Section Leader to prioritize.

5. Handbook

There are five Product PI pages: The Product Team page and section pages for Dev, Ops, Secure & Protect, Enablement.

We need all PMs to ensure their PIs are showing on the performance indicator pages. Based off What we're aiming for

To do so, we need a clear way to communicate to PMs exactly which PIs are remaining. We will be adding placeholder PIs for each section into the performance indicator file so that all required PIs show in the handbook. Once a PI is implemented, the actual PI will replace the placeholder PI.For more information about how PIs and XMAUs are related to one another, see PI Structure.

Instructions:

  1. @jeromezng @kathleentam will add placeholder PIs for each section.
  2. @jeromezng @kathleentam will then meet with each Section, Stage, and Group, to understand the implementation status of each PI and document any exceptions. Exceptions for PIs will then be signed off by Scott, Anoop, and the Section Leaders.
  3. Keep your performance indicator up to date as the implementation status changes.

6. Target

As a product organization, we need to get into the habit of understanding our baselines and setting targets for each stage & group. For the PI Target step, you will work with your Section or Group Leader to define targets for each of your XMAUs.

Set a growth target, and embed in the tracking dashboard. Growth targets should be ambitious but achievable.

Instructions:

  1. Work with your Section or Group Leader to define a target.
  2. Add the Target Line into your dashboard (example).
  3. If you need help, book a meeting with @jeromezng @kathleentam.

7. Complete

All of the prior steps have been completed and a PI is successfully implemented.

Instructions

  1. Continually monitor your results, and refine your priorities based on results.
  2. Line up your roadmap priorities to positively influence the adoption funnel, drive the performance metric, and hit the growth targets.
  3. Deeply understand the drivers of your metric through customer interviews and other product usage data. Visualize this through a user adoption funnel that describes how users enjoy the value of the particular product area you are measuring.
Resource Description
Product Analytics Product Direction The roadmap for Product Analytics at GitLab
Acquisition, Conversion, Product Analytics Development Process The development process for the Acquisition, Conversion, and Product Analytics groups
Product Analytics Guide An overview of our collection tools
Usage Ping Guide An implementation guide for Usage Ping
Snowplow Guide An implementation guide for Snowplow
Event Dictionary A SSoT for all collected metrics and events
Privacy Policy Our privacy policy outlining what data we collect and how we handle it
Product Performance Indicators Workflow The workflow for putting product PIs and XMAUs in place
Data Team: Creating Charts How to create your own dashboard
Data Team: Data Warehouse An outline of where our product usage data is stored
Data Team: dbt Guide How we transform raw data into a data structure that's ready for analysis
Growth Product Direction The roadmap for Growth at GitLab
Growth Product Process The product process for the Growth sub-department
Growth Sub-Department Development Process. The development process for the Growth sub-department
Growth Sub-Department Performance Indicators Page The performance indicators for the Growth sub-department
Growth UX Process The UX process for the Growth sub-department
Growth QE Process The QE process for the Growth sub-department
Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license