Use Case Adoption Scoring

Defining Adoption & Identifying Lighthouse Metrics

Timeframe: 2-3 Months

The first step in the process of building a Use Case adoption score starts with identifying Lighthouse metrics that can gauge a customer’s level of performance and adoption for a given use case. To capture the breadth of use case adoption, we rely on Leading Indicators that capture weekly/monthly usage of key features within a use case to accurately measure the maturity of a customer’s adoption.

Product Management, Data, and Customer Success teams play a big role in identifying use case adoption specific metrics. Product Marketing works with Product Managers to identify Adoption Recommendations for each use case which can be viewed on the the respective Use Case Resource Pages. After identifying key metrics, the respective Product team taken on the responsibility of instrumenting said metrics (if they do not exist already), after which we move on to identifying a denominator to normalize the metric (i.e. Git Operations - User L28D / by Billable Users = Git Operation Utilization %).

Metric Instrumentation

Timeframe: 8-12 Months

The respective Product Team for the use case takes on the responsibility to help prioritize and triage the work required to collect and measure usage of key features that have been selected as the Lighthouse and/or Leading indicators. This process can take anywhere from 2-6 months depending on bandwidth and priortization.

Instrumentation is the first peice of the puzzle since it requires benchmarks to be able to move onto thresholding which derives from historical usage of the said use case and features. We require at least 6-8 months of historical usage to be able to accurately set thresholds for each measure to calculate an adoption score.

Add Metrics to Ingestion Models (Snowflake Tables)

Timeframe: 1-2 Months

The Data team takes on the responsibility of adding the selected Lighthouse Metric(s)/Leading Indicators into the relevant ingestion models (Snowflake Tables) that feed into our BI systems such as Tableau (previously Gainsight). This process requires a minimum of 4 weeks of work and validation to ensure we’re capturing usage accurately at an instance level.

Threshold and Weighting Analysis

Timeframe: 3-6 Weeks

The threshold percentages are chosen by looking at how many customers fall into certain quartiles as well as looking at the averages/median for a subset of customers that we know have healthy usage in this use case. Combining those two gets us to a great starting point of where those percentages should be for each metric. We then look at how many customers would fall into each color based on that starting point. There are some final tweaks after that, as well as gathering feedback from CSMs and Product on set percentages for each color.

Testing

Timeframe: 2-4 Weeks

This is an important part of validating the selected metrics, thresholds, weighting, and data integrity. The process entails calculating the score for at least 15-20 real accounts, specifically where we have a good understanding of their adoption of the given use case, which we can then share with the CSMs to confirm the resulting score.

Operationalize in Gainsight/Tableau

Timeframe: 2-3 Weeks

Use Case Adoption metrics currently live in Snowflake where they are calculated and then visualized in the Gainsight scorecard. There are corresponding reports within C360 that allow Gainsight users to conduct a deep dive into any specific use case for the purposes of supporting conversations with their customers to be able to advise on best practices and recommendations to improve adoption.

Enablement

Timeframe: 1-2 Weeks

The last piece is enabling CSMs to be able to calculate the score, formulate the customer-facing slide deck and be able to present the score in detail. This requires cross-functional partnership between CSMs, CSM Managers, Enablement and CSOps, where a CSM will usually lead the enablement sessions with a mock presentation, while CSOps handle Q&A and any follow-ups that may come out of those enablement sessions.

Change Management

Timeframe: Varies

With a product that updates monthly while adding new features to the GitLab use cases, we have an ongoing process to iterate and improve the respective adoption scores, aiming for the most accurate depiction of product adoption by our customers. As new features are built and made available, new metrics are also instrumented by the respective product teams to be able to provide insights to customers that can help them improve their experience and attain their DevOps goals with GitLab.