The Customer Health Score assists GitLab Account Teams in understanding the customer’s relative health of GitLab adoption and engagement through their lifecycle journeys. This assists expansion and retention efforts through understanding the customer's product adoption, risks, and engagement with GitLab.
As initially laid out in the Account Health Scoring epic, the intent of Health Scoring is to enable the teams to better understand customer adoption of GitLab.
95% of customers have a health score (alighed to FY24 Yearlies)
Product usage statistics inform three different scores. They each have a distinct and separate purpose, are meant for different audiences, and use different metrics.
| Name | Purpose | Audience | Metrics | Notes |
|---|---|---|---|---|
| Customer Health Score | To enable the team in understanding the relative health of customers, identify risks, manage renewal likelihood, and identify expansion opportunities and why | Internal GitLab Teams | Account Health is an aggregation of key metrics for a multi-perspective view of the customer. For detail, see Account Health Predictor above | |
| Value Stream Management | This customer-facing dashboard hosts value stream analytics metrics including DORA4 | GitLab customers | DORA 4 metricsm month-over-month usage metrics | |
| Platform Adoption Score | Each customer will have a single Platform Adoption Score used to understand the level of use cases being utilized and value the customer is currently receiving from Gitlab application | Internal GitLab Teams | 3–7 product usage metrics per Use Case, using Red/Yellow/Green scoring. The metrics will roll up into a score for each Use Case, and the Use Case scores will roll up to the Platform Adoption Score | Platform Adoption Scoring |
| Use Case Adoption Scorecards | Each customer will have a scorecard per use case (SCM, CI, CD…) to highlight their adoption progress to celebrate wins and identify areas for improvement | Internal GitLab Teams and customers | 3–7 product usage metrics per Use Case, using Red/Yellow/Green scoring. These are then used to create a deck highlighting the level per use case adoption | Use Case Adoption Scoring |
| DevOps Score | For the customer to understand their DevOps status compared to top-performing instances (self-managed only) | GitLab customers | 10 metrics across Use Cases, displayed as a % of users who have utilized a feature in the past month, compared to how top-performing instances utilized that feature (self-managed only - GitLab features) | Handbook Link |
| DevOps Adoption | DevOps Adoption shows you how groups in your organization adopt and use the most essential features of GitLab. | GitLab customers | Specific metrics across Dev, Sec, and Ops to show a customer's overall adoption. requires configuration (group and project level for SaaS, whereas instance for self-managed - GitLab features) | Docs Link |
| DevOps reports | This is being combined with DevOps Adoption and will be eventually replaced by Customer Executive Dashboards | GitLab Customers | Dev, Sec, and Ops feature utilization (docs) |
Account Health is an aggregation of key metrics to provide insights to assist with identifying:
For instance, the customer may have deployed all their subscription licenses but aren’t actively using them; or they may be using them, but all their Support tickets are very negative.
Looking through just one lens provides a limited view. In a happier example, a customer may have deployed most of their licenses, be heavily using all the current tier’s high end features, and achieving positive business outcomes (PBOs). In this case, metrics indicate expansion opportunities. We will need to PROVE value to the customer and ourselves:
P.R.O.V.E.
Health primarily considers the business impact to GitLab by evaluating the delivery of value and outcomes to customers. The following guideline will provide Customer Success Managers (CSMs) guidance to choose the right health assessment for their customer account. The following are the categories to assess and associated risks with each.
There is a delayed, low, or materially reduced usage (i.e., drop in usage) as measured by license usage, features / use cases, product version (i.e., not adopting current versions - self-managed only), and/or GitLab stages. Value and outcome delivery to the customer misses expectations as defined by the customer. This may also be impacted by way the customer is using the product (i.e., processes, operations and/or policies) where the customer may not be leveraging GitLab best practices to maximize the value of the solution.
Customer has enhancements or defect fixes that are necessary for a customer and have not been delivered. The risk is determined according to the criticality of the request, severity of the issues, and/or number of enhancements and defects. Missed expectations for feature release can also impact product experience.
Customer contact(s) are not responsive, miss meetings and/or unwilling to engage in cadence calls or other engagements like EBRs. This could indirectly mean the customer does not see value in the solution or the solution has been deprioritized.
Sponsor or champion leaves the company, moves to a different part of the organization, and/or has reduced scope of influence.
The customer has expressed concerns and/or dissatisfaction with their experiences with GitLab (i.e., sales, professional services, support, product, etc.) through direct conversations, surveys (e.g., NPS), social media or other communication channels.
Prior to or during the renewal process it is learned that competitors are in play and the result could be a downgrade or churn as the customer is considering alternatives.
The customer's business performance is materially impacted and declining. The company is acquired, merging with another company, divested or another structural change to customer's business.
See Customer Health Assessment and Management for details on how and when a CSM tracks a customer's health, including when an account should be set to Yellow, Red, or Will Churn.
The items below serve as guidelines for the CSM to assess and record customer health and should consider where the customer is their lifecycle journey. Account health feeds into open renewal opportunities.
Understanding how Gainsight calculates a measure score to be Red, Yellow, or Green:
Gainsight scoring framework:
Link to Gainsight Calculation of Measure Group Scores and Overall Score
Calculation: ((Score * Weight) + (Score * Weight)… / (Max Potential Score * Weight)
To view the score, hover over the colored circle. The weight is shown in the upper right corner of the metric.
Use the Customer Health (CSM Portfolio Dashboard) report to view the health of every measure for your customers in one single view.
To view Timeline entires where the CSM Sentiment was updated:
Health score criteria is either manually or automatically applied to determine the overall measure. If an individual measure is missing, the weighting is redistributed to the completed measures.
| Group (PROVE) | Measure | Description | Method | Calculation | Measure Weight | Group Weighting | Segmentation |
|---|---|---|---|---|---|---|---|
| Product | Scores the customer based on their depth and breadth of adoption, if Operational Metrics are available | Automatic | See Customer Use Case Adoption | 50% | |||
| License utilization | 30% | All | |||||
| User Engagement | 10% | All | |||||
| SCM adoption | 15% | All | |||||
| CI adoption | 15% | All | |||||
| DevSecOps adoption | 15% | All | |||||
| CD adoption | 15% | All | |||||
| Risk | CSM sentiment | Qualitative measure the CSM updates to indicate their perceived sentiment of the customer | Manual/Automatic | For all CSM-owned accounts, CSM manually determines red/yellow/green |
100% | 25% | N/A for Tech Touch |
| Outcomes | ROI | Does the customer have a Success Plan that has objectives and notes? | Automatic | For All CSM Prioritization = 1 accounts AND all CSM-owned accounts that have an open Success Plan:- Green: Active Success Plan with 1 or more objectives and no Strategy/Highlight information - Yellow: Draft Success Plan OR Active Success Plan with 1 or more objectives and no Strategy/Highlight information - Red: No Success Plan or no objectives |
100% | 10% | N/A for Scale and Tech Touch |
| Voice of the customer | 5% | ||||||
| Support issues | Assess the health of our support interactions. Current version is MVC with v2 coming. | Automatic | - Green: 1-5 tickets/month - Yellow: 5-15 tickets/month - Red: More than 15 tickets/month |
100% | All | ||
| Support emergency tickets | Based on the number of open/closed tickets. Priority: urgent tickets |
Automatic | - Yellow: 1+ closed emergency ticket in the last 7 days - Red: 1+ open emergency ticket |
0% | All | ||
| Engagement | 10% | ||||||
| Meeting cadence | Based on recency of last call or meeting with the customer | Automatic | For CSM Prioritization = 1 accounts:- Green: <= 35 days - Yellow: > 35 days and <= 60 days - Red: > 60 days For CSM Prioritization = 2 accounts:- Green: <= 65 days - Yellow: > 65 days and <= 90 days - Red: > 90 days |
50% | N/A for Scale and Tech Touch | ||
| Persona engagement | Are we meeting with the correct personas in the account? | Automatic | Persona Engagement is based on the roles of External Attendees added on timeline entries - Green: both Dev Lead and Security Lead are listed as external attendees on a timeline entry in the past three months - Yellow: one of the two personas attend - Red: neither personas are listed as having attended a meeting |
50% | N/A for Growth, Scale and Tech Touch |
CSMs update CSM Sentiment in determining overall account health. The guidelines are as follows:
The CSM Sentiment score will be updated each time you log a Timeline activity and select a value from the CSM Sentiment dropdown. Once you have logged the activity to Timeline, Gainsight will update the value of the CSM Sentiment scorecard measure and display the notes from the Timeline activity on the scorecard. The rule that sets the scorecard value runs every 2 hours.
There are a number of enablement videos you can watch to learn how to update customer health assessment and log activities that affect that assessment.
Product If usage data stops being received into Gainsight, the health measures will move to "NA" after 60 days. This is to prevent analysis and actions based on outdated data. In this case, we prefer to show nothing ("NA") over outdated data.
CSM sentiment
CSM Sentiment health scores become stale after 90 days of not being updated; this will be reflected on your health score dashboard by an exclamation point next to the score. If an account is marked as stale, but you've updated the CSM Sentiment within 90 days, please reach out in gainsight-users. Accounts with a stale CSM Sentiment will also be monitored via the CSM Burn-Down Dashboard in Gainsight and discussed in account planning meetings. Sentiment scores will be set to NA if they have not been updated in more than 120 days to remove outdated values.
Support measures Support measures are considered stale if they have not been updated in more than 30 days. They will be automatically set to NA after 30 days without an update.
When an account has multiple GitLab instances identified as Production (Instructions on how to Update Self-Managed Instance Type), the Product Usage health measures the most recently updated instance instead of the primary instance, causing scoring inconsistencies. Note: this is less than 5% of the time because the vast majority of accounts have a single production instance.
Video Instructions to update instance data in Gainsight to include only one instance in Product Usage health measure.
Included in Health Measures to EXCLUDE the instance section. NOTE: Make sure you select “Opt-Out” rather than null, or the system may overwrite your update. Then click UpdateImportant notes:
Included in Health MeasureIncluded in Health Measure unless they are marked Opt-OutOpt-Out rather than null, or the system may overwrite your updateBecause the DevSecOps health measure looks to the account as Ultimate, this step was added to make sure the correct production instance is scored in the case of multiple subscriptions under a given account.
If a CSM has marked a production instance under a Premium subscription, DevSecOps health will appear as NA. Meaning, even if there are two subscriptions with one Premium and another Ultimate, as long as the CSM marked the Premium one for health scoring, you will no longer see a DevSecOps health score (generally red) on the account.
Gainsight Rules:
NEW: Admin: Update Plan Name on Product Usage Instance Metrics
Plan Name from the Customer Subscription object to the Product Usage Instance Metrics objectSet Score: DevSecOps Adoption Individual Measures
Plan Name on the Product Usage Instance Metrics object instead of the Products Purchased on the Company objectSegmentation will primarily follow the level of service (CSM Priority 1, 2, 3), and secondarily other factors as listed below.
The Account Health Score does and will include many factors with different weightings per group and per individual measure with the goal being a multi-perspective approach, measuring what matters to the customer, and measuring the features that they have access to and can utilize.
Note: if stats are missing for any health measure, it is counted as NULL instead of a value (i.e., red).
Evaluates the customer’s usage based on their current tier and feature access.
For example, if a customer is on Premium, we will base their health on Premium-level features to understand their level of adoption. If their health is red or yellow, it signifies risk. If green, it can signify expansion or flat renewal.
Some metrics are more leading or lagging indicators. While we will lean toward a predictive solution, lagging metrics are incorporated to assess past performance.
The following graph (Early Warning Segmentation Framework) is used to provide a framework for which strategy to use and which resources to leverage. Customers are grouped by their Account Health and growth potential. Renewal Operations Analysts will support the Field in triaging accounts to identify where to spend their time.
The first approach was a calculation of multiple metrics to create a “black box” approach. This was neither helpful to the end user (CSMs, SAs, sales reps), it was not easy to understand the calculation, the Gainsight logic was inadequate, and was not action-oriented to know which aspects of the use case were great and which needed improvement.
The next iteration is a model where each use case incorporates X number of metrics and each metric is valued from 0-1.0. Then, the individual scores can then be summed to an aggregated score for that use case. Below is an example of what could be done for transparently measuring health.
Example: CI has ten individual features with one metric per feature and each metric is equally weighted at 10. Each metric can score between 0-10 with some being zero, some being 5, and others being 10. The aggregate score would be 65 out of 100. The CSM could then evaluate each metric to see which features are being adopted and which ones need improvement.
While the product usage health will be summarized, a separate health view will allow users to view each individual component. This allows users to quickly skim overall health and, when applicable, to look into the details to see which features are not being utilized.
| Category | Health Measure | Example | Why? | Metrics | Account Type | Maturity |
|---|---|---|---|---|---|---|
| Product | License Activation | The customer has assigned all licenses | Has the customer deployed their licenses? This is an indicator of seat reduction / expansion | License utilization | All | 100% |
| Product | User Engagement | 73% of users are Monthly Active Users | Are users logging in and using the product? | Unique Monthly Active Users / billable_user_count | All | 100%; in progress |
| Product | Adoption (Use Case) | Use Case adoption | Is the customer adopting use cases and progressing into “stickier” areas of GitLab? | SCM —> CI —> DevSecOps adoption | All | 100% |
| Risk | CSM Sentiment | The sentiment as determined by the CSM, if applicable | What has the CSM determined from cadence calls? | CSM Sentiment | CSM owned | 100% |
| Outcomes | ROI Success Plan | Ensure the ROI Success Plan is aligned to customer | A missing or poorly constructed Success Plan highlights a lack of alignment between GitLab and customer desired outcomes. | Green Success Plans Delivered EBRs | CSM owned | 100% |
| Outcomes | Positive Business Outcomes (PBOs) | Completed Success Plan Objectives | Failed or missed PBOs can be a sign of distress; successful PBOs can highlight renewal expansion | Successfully completing at least one PBO each year | CSM owned | Not started |
| VoC | Support - Escalations | Emergency support tickets | Emergency support tickets can indicate unhappiness or frustration | Measure if there are Emergency support tickets in the last 90 days | All | 100% |
| VoC | Support - Engagement | Customer sends in tickets | Determining if the customer is engaged with Support | Retain existing methodology, but tweak to allow more tickets as a good thing | All | 70% |
| VoC | Support - CSAT | Customer completes CSAT surveys and provides feedback | Is the customer giving feedback and what are the scores (response + outcomes) | Benchmark a minimum XX% response rate for green health and provide CSAT results to CSM | All | Not started |
| VoC | NPS Surveys | The customer responds to and provides high scores | Because surveys are a good indicator of the customer’s perception of the product and company; this can | Survey responses rates + survey scores | All | Not started |
| Engagement | Engagement | Recency of CSM cadence call | Lack of customer engagement | Date of last CSM cadence call | CSM owned | 100% |
| Engagement | Executive Sponsorship | Are stakeholders aligned and communicating? | Lack of alignment and communication can indicate a disconnect between execs and ROI | Recency of aligned stakeholder communication | CSM owned | Not started |
| Engagement | Events | Is the customer attending GitLab events? | Event attendance indicates customer engagement, dialogues with team members, and face-to-face interactions | TBD | All | Not started |
| Engagement | Certifications | Are users within the account taking certifications? Are they maintaining their certifications? | Obtaining GitLab certifications is a positive for us and the customer; it also indicates their involvement in GitLab, knowledge of using GitLab, and provides an inference as an internal champion | TBC | All | Not started |
Predictive Analytics is not a silver bullet. It will not cure all that ails you. Instead, this methodology is probabilistic and incorporates health measures to correlate the typical journey of “healthy” customers (expand and renew) with “unhealthy” customers (downgrade and churn). For example, a healthy sales pipeline has few pushes (moving the close date) and progressively moves through stages (not stale). Conversely, an opportunity with multiple pushes and stuck in stages for long periods of time is an indicator of risk.
| Prediction Type | Model Name | Status | Description |
|---|---|---|---|
| Expansion | Propensity To Expand (PTE) | Active | Predicts whether an account is likely to expand (increase ARR) |
| Churn and Contraction | Propensity To Churn or Contract (PTC) | Active | Predicts whether an account is likely to churn or contract (decrease ARR) |
Triggers will be used for different events:
Each of these metrics will be used to guide the account team in knowing when a customer is approaching the next or has met their milestone. The items listed below are examples of what an account team could look at to glean insights for a productive customer conversation.
Indicators from Seat Reduction or Downtier above plus: