The Customer Health Score assists GitLab Account Teams in understanding the relative health of customers for the purposes of predicting expansion, retention, and churn. The initial versions will focus more on adoption. Over time, we will iterate to make them more predictive as we validate leading indicators.
This will be leveraged for the Early Warning System (EWS) as it is a work in progress, starting in FY22-Q2 and expected to take multiple quarters to achieve the overall objective. Future state will include LAM for expansion potential.
Product usage data informs three different scores. They each have a distinct and separate purpose, are meant for different audiences, and use different metrics.
Account Health is an aggregation of key metrics for a multi-perspective view of the customer to be used to predict the customer’s likelihood to:
For instance, the customer may have deployed all their subscription licenses but aren’t actively using them; or they may be using them, but all their Support tickets are very negative.
Looking through just one lens provides a limited view. In a happier example, a customer may have deployed most of their licenses, be heavily using all the current tier’s high end features, and achieving positive business outcomes (PBOs). In this case, metrics indicate expansion opportunities. We will need to PROVE value to the customer and ourselves:
P.R.O.V.E.
| Category | Health Measure | Example | Why? | Metrics | Account Type | Maturity |
|---|---|---|---|---|---|---|
| Product | License Activation | The customer has assigned all licenses | Has the customer deployed their licenses? This is an indicator of seat reduction / expansion | License utilization | All | 100% |
| Product | User Engagement | 73% of users are Monthly Active Users | Are users logging in and using the product? | Unique Monthly Active Users / billable_user_count | All | 0% |
| Product | Adoption (Use Case) | Use Case adoption | Is the customer adopting use cases and progressing into “stickier” areas of GitLab? | SCM —> CI —> DevSecOps adoption | All | 100% |
| Risk | TAM Sentiment | The sentiment as determined by the TAM, if applicable | What has the TAM determined from cadence calls? | TAM Sentiment | TAM owned | 100% |
| Outcomes | ROI Success Plan | Ensure the ROI Success Plan is aligned to customer | A missing or poorly constructed Success Plan highlights a lack of alignment between GitLab and customer desired outcomes. | Green Success Plans Delivered EBRs | TAM owned | 100% |
| Outcomes | Positive Business Outcomes (PBOs) | Completed Success Plan Objectives | Failed or missed PBOs can be a sign of distress; successful PBOs can highlight renewal expansion | Successfully completing at least one PBO each year | TAM owned | Not started |
| VoC | Support - Escalations | Emergency support tickets | Emergency support tickets can indicate unhappiness or frustration | Measure if there are Emergency support tickets in the last 90 days | All | 100% |
| VoC | Support - Engagement | Customer sends in tickets | Determining if the customer is engaged with Support | Retain existing methodology, but tweak to allow more tickets as a good thing | All | 70% |
| VoC | Support - CSAT | Customer completes CSAT surveys and provides feedback | Is the customer giving feedback and what are the scores (response + outcomes) | Benchmark a minimum XX% response rate for green health and provide CSAT results to TAM | All | Not started |
| VoC | NPS Surveys | The customer responds to and provides high scores | Because surveys are a good indicator of the customer’s perception of the product and company; this can | Survey responses rates + survey scores | All | Not started |
| Engagement | Engagement | Recency of TAM cadence call | Lack of customer engagement | Date of last TAM cadence call | TAM owned | 100% |
| Engagement | Executive Sponsorship | Are stakeholders aligned and communicating? | Lack of alignment and communication can indicate a disconnect between execs and ROI | Recency of aligned stakeholder communication | TAM owned | Not started |
| Engagement | Events | Is the customer attending GitLab events? | Event attendance indicates customer engagement, dialogues with team members, and face-to-face interactions | TBD | All | Not started |
| Engagement | Certifications | Are users within the account taking certifications? Are they maintaining their certifications? | Obtaining GitLab certifications is a positive for us and the customer; it also indicates their involvement in GitLab, knowledge of using GitLab, and provides an inference as an internal champion | TBC | All | Not started |
The Account Health Score does and will include many factors with different weightings per group and per individual measure with the goal being a multi-perspective approach, measuring what matters to the customer, and measuring the features that they have access to and can utilize.
Note: if data is missing for any health measure, it is counted as NULL instead of a value (i.e., red).
Tier-based Product Usage Data: will evaluate the customer’s usage based on their current tier and feature access. For example, if a customer is on Premium, we will base their health on Premium-level features to understand their level of adoption. If their health is red or yellow, it signifies risk. If green, it can signify expansion or flat renewal.
Leading and lagging indicators: Some metrics are more leading or lagging indicators. While we will lean toward a predictive solution, lagging metrics are incorporated to assess past performance.
The following graph (Early Warning Segmentation Framework) is used to provide a framework for which strategy to use and which resources to leverage. Customers are grouped by their Account Health and growth potential. Renewal Operations Analysts will support the Field in triaging accounts to identify where to spend their time.
For a fuller list of the project roadmap, see Product Usage Data Roadmap.
The first approach was a calculation of multiple metrics to create a “black box” approach. This was neither helpful to the end user (TAMs, SAs, sales reps), it was not easy to understand the calculation, the Gainsight logic was inadequate, and was not action-oriented to know which aspects of the use case were great and which needed improvement.
The next iteration is a model where each use case incorporates X number of metrics and each metric is valued from 0-1.0. Then, the individual scores can then be summed to an aggregated score for that use case. Below is an example of what could be done for transparently measuring health.
Example: CI has ten individual features with one metric per feature and each metric is equally weighted at 10. Each metric can score between 0-10 with some being zero, some being 5, and others being 10. The aggregate score would be 65 out of 100. The TAM could then evaluate each metric to see which features are being adopted and which ones need improvement.
While the product usage health will be summarized, a separate health view will allow users to view each individual component. This allows users to quickly skim overall health and, when applicable, to look into the details to see which features are not being utilized.
When an account has multiple GitLab instances identified as Production, (Instructions on how to Update Self-Managed Instance Type) the Product Usage health measure uses the most recently updated instance. That means the Product Usage health measure lacks precision on which instance it scores. Note: this is less than 5% of the time because the vast majority of accounts have a single production instance.
(Video Instructions)[https://youtu.be/N0JUABX88Hg] on how to update instance data in Gainsight to include only one instance in Product Usage health measure.
Important to Note:
Predictive Analytics is not a silver bullet. It will not cure all that ails you. Instead, this methodology is probabilistic and incorporates health measures to correlate the typical journey of “healthy” customers (expand and renew) with “unhealthy” customers (downgrade and churn). For example, a healthy sales pipeline has few pushes (moving the close date) and progressively moves through stages (not stale). Conversely, an opportunity with multiple pushes and stuck in stages for long periods of time is an indicator of risk.
| Prediction Type | Model Name | Status | Description |
|---|---|---|---|
| Expansion | Propensity To Expand (PTE) | Active | Predicts whether an account is likely to expand (increase ARR) |
| Churn and Contraction | Propensity To Churn or Contract (PTC) | Active | Predicts whether an account is likely to churn or contract (decrease ARR) |
Triggers will be used for different events:
Each of these metrics will be used to guide the account team in knowing when a customer is approaching the next or has met their milestone. The items listed below are examples of what an account team could look at to glean insights for a productive customer conversation.
Indicators from Seat Reduction or Downtier above plus:
Segmentation will primarily follow the level of service (TAM Priority 1, 2, 3), and secondarily other factors as listed below.
