The Category Maturity Scorecard is a Summative Evaluation that takes into account the entire experience as defined by a Job to be Done (JTBD), instead of individual improvement(s), which are often measured through Usability Testing. This process provides data to help us grade the maturity of our product based on user performance.
The goal of this process is to produce data as objectively as possible given time and resource constraints. For this reason, the process is more rigorous than other UX research methods, and it focuses more on quantitative data (numbers and measures) and less on qualitative data (thoughts and verbal feedback).
To produce data in which we have confidence, the data should be as free of subjective judgement as possible, relying on observed metrics and self-reported user sentiment. Our goal is to compare data over time to see how additions and improvements have impacted product maturity in a quantifiable way. To facilitate this, we've made this process prescriptive, so that it can be consistently applied by all Product Designers in all areas of our product.
You should conduct this process after a feature/function has already gone through problem and solution validation, so you have information about the ‘why’ behind any failures to complete the associated task(s).
Note: As with any evaluation, it's always a good idea to run a pilot first, so you can identify any improvements needed in the research approach.
Note: If you have questions, suggestions for improvement, or find this process doesn’t meet the needs of your users or product, reach out to the UX Researcher for your group.
Category Maturity Scorecards are about judging the quality of experiences against a defined and verified JTBD. Each Product Designer works with their Product Manager to identify main JTBD(s) and related JTBD(s) within the category, which are mapped to features. JTBD, along with the features they address, are used to create scenarios for the Category Maturity Scorecard process. The number of scenarios used often depends on the complexity of the features tested.
Refer to the Category Maturity page to understand scoring. It is important to note that:
During the JTBD creation and validation phases, the Product Designer and Product Manager devise a set of user criteria to describe the user(s) you're referencing in your job(s). Using this criteria when recruiting for the Category Maturity Scorecard will ensure you are gathering feedback from the right type of user(s).
To balance expediency with getting a variety of perspectives, we conduct the Category Maturity Scorecard research with five participants from one set of user criteria. If you have multiple user types for your JTBD, it is ideal to recruit 5 from each user type. Example: A JTBD can be completed by a DevOps Engineer and a Release Manager. In this case, you’d recruit a total of 10 participants: 5 DevOps Engineers and 5 Release Managers.
The recruiting criteria can be based on an existing persona, but needs to be specific enough to be turned into a screener survey. Create a screener survey in Qualtrics that your prospective participants will fill out to let you know if they're eligible.
The template survey includes a question asking people if they consent to having their session recorded. Due to the analysis required for Category Maturity Scorecards, participants must answer yes to this question in order to participate. Once your screener survey is complete, open a Recruiting request issue in the UX Research project, and assign it to the relevant Research Coordinator. The Coordinator will review your screener, reach out to you if they have any issues, and begin recruiting users based on the timeline you give them.
Recruiting users takes time, so be sure to open the recruiting issue at least 2-3 weeks before you want to conduct your research.
Testing in a production environment is the best choice because your goal is to evaluate the actual product, not a prototype that may have a slightly different experience.
Once you know what scenario(s) you’ll put your participants through, it’s important to determine the interface you’ll use. Some questions to ask yourself:
It’s important to thoroughly plan how a participant will complete your scenario(s), especially if you answered "yes" to any of the questions above. Involve technical counterparts early in the process if you have any uncertainty about how to enable users to go through your desired workflow(s).
If your JTBD interacts with other stage groups’ areas, reach out to them to ensure their part of our product will support your scenario(s).
Because this is a summative evaluation of the current experience, all of the available options the participant should need access to must be available in the GitLab instance. When you recruit users, keep in mind the tools and features they must access to complete the JTBD scenarios.
After you decide on your scenario(s) and user criteria, it's important to go through the flow(s) yourself.
Typically, there is a primary flow that users will most often use and sometimes secondary flows that are less common or efficient. Document both the primary flow and secondary flow(s), as you will use these to define successful completion of the scenario(s). For each flow, document both the entire path taken and why you chose it.
Sample primary path:
Sample secondary path
Having a detailed and documented scenario will help you determine whether your participant took the primary or a secondary successful path. It will also help indicate the point at which your participant deviated from the successful path and veered into the path of failure. Indicating points of failure will help identify areas of product improvement.
Once you've gone through your scenario(s) and documented the primary and, as needed, secondary path(s), have a co-worker complete the scenario as well. Ideally, this person won't be familiar with the scenario, so they don't have an expert-level understanding of how it works. Use this to uncover any issues with how you've formulated your scenario or how you have documented the path(s). As this is meant as a way to check your scenario/path(s) plan, it's ok to coach your co-worker a little, using this discussion to get to the heart of any problems your scenario or path may have.
Because Category Maturity Scorecards are a standardized process, moderators should follow this testing guide as closely as possible. The moderator will typically be a Product Designer, but this is not strictly required. You are encouraged to have any relevant stakeholders attend the sessions, but it is very important they remain silent.
The goal for analyzing Category Maturity Scorecard data is to establish a baseline measure for the current experience as it relates to the JTBD(s). Over time, our product will change as new features/functions are added/changed. We can review the data collected here to understand how these changes impacted the user experience and use it to make improvements.
It’s important that the moderator and any stakeholders don’t leave the call when the session concludes. Instead, remove the participant and remain on the call. Use this time for the group to debrief on what they just experienced. The notetaker(s) should take notes on this discussion.
By following the Category Maturity Scorecard testing guide, you will have the following measures to report, per feature, not per scenario. However, scenarios may include more than one feature.
The UMUX Lite score is based on the UMUX (Usability Metric for User Experience), created by Finstad, and it is highly correlated with the SUS and the Net Promoter Score. It's intended to be similar to the SUS, but it's shorter and targeted toward the ISO 9241 definition of usability (effectiveness, efficiency, and satisfaction). The UMUX was shortened by Lewis et al., so that the content of the UMUX Lite nicely mimics the Technology Acceptance Model (TAM)
When you apply the UMUX Lite, you'll ask the following questions of each participant using a 5-point scale (1 is lowest, 5 is highest):
Compute the average score for each question separately. For example:
|The issue creation capabilities met my requirements.||4||2||3||5||5|
|Issue creation is easy to use.||5||4||5||5||5|
Use Dovetail to document your findings. Your notes should include the following information (you can create a note taking template for your Dovetail project to make this easier):
|Participant Number||Successful||Failed||Number of Errors Encountered|
|Participant Number||Successful||Failed||Number of Errors Encountered|
(continue with as many scenarios as you used)
(continue with as many participants as you had in the study)
Other notes (Add any additional notes that came from freeform discussion or elsewhere)
Read the UX Research team’s guide for documenting insights in Dovetail.