At GitLab, we use UserTesting.com for certain unmoderated testing for Solution Validation and Category Maturity Scorecard studies. Unmoderated usability testing allows us to obtain evaluative feedback from our users very quickly, thereby allowing us to iterate faster in a more independent manner that requires fewer resources.
Unmoderated usability testing is defined as a way to conduct usability testing without a moderator being present. That means the participant will go through all of the tasks/instructions/questions solo. This is typically accomplished through a service that also has a panel of participants. The output of an unmoderated usability study are videos, which have to be watched and analyzed by the person who kicked-off the research.
For awareness, we don’t have an unlimited amount of studies we can conduct with UserTesting.com, so please consider that when launching studies on UserTesting.com.
It’s important to note that there are pros and cons to using unmoderated testing, and those should be considered before deciding to go with unmoderated vs. moderated. To illustrate:
If using Figma for your prototype, there are a few tips to keep in mind before attaching the link to your test:
Fit to screento fill the participant’s screen.
Show hotspot hints on click. Often participants know to search for these which may bias testing results.
Show Figma UI. This will allow the participants to focus only on your prototype without seeing extra information.
allow the participant to view the prototype, and only the prototype.
How many participants should I request per study? Since we’re using unmoderated for evaluation studies, the sample size should be between 5-8 participants per study.
Can I create a custom panel of existing GitLab users? While it is possible to create a custom panel, we recommend using the UserTesting panel and use the screener questions that are already saved for you to screen for GitLab users in specific roles.
Can I screen for certain criteria? Of course - whatever you want. The stricter the screener criteria, the less likely you are to find the participants you’re looking for. Note that there are already some GitLab screening questions in UserTesting.com for you to utilize. You can find them under the 'Saved Screener Questions' within UserTesting.com.
How do the demographic filters work with all those checkboxes? The way the demographic filters work is as follows: If no filter is selected, the pool of participants would include anyone in the UserTesting.com panel. When you start selecting filters, the pool of participants becomes specific to the filters you are looking for, hence a smaller pool of potential testers. Not selecting any filters does not exclude anyone from the study, on the contrary, you will target more participants in this manner. It's recommended to keep the demographic filters as general as possible and then use the screener questions to find the specific type of participant you are looking for.
Do I have to ask certain screening questions, to meet GitLab's requirements? Yes. At the very least, you must ask the 'Individual contributor licence agreement' question - and possibly the IP Assignment question. Both of which are already saved as a 'Saved Screener Question' within UserTesting.com for you to add to your screening criteria.
I ended up with a not-so-great participant (for example, they didn’t meet the screening criteria, they weren’t very vocal, or they didn’t follow instructions, etc.). What do I do? Within UserTesting.com, you have the ability to rate your participants on a 5-star scale. If you rate them as 1-star, UserTesting.com Support will reach out to you and ask if you would like a replacement participant. You can either say yes or ask for credit instead.
Do I have to worry about how many times I use UserTesting.com? You should be aware that we have a limited number of ‘units’ that we use within the year. It’s not an unlimited number of units. However, you are encouraged to utilize UserTesting.com as much as you need to.
How do I get an account? Through an Access Request. UserTesting.com seats are allocated only to Product Designers and UX Researchers at this time, since those are the disciplines that are closely involved with solution validation and Category Maturity Scorecard testing.
How do recruitment and gratuities work? When recruiting through the UserTesting.com panel, gratuities are taken care of. You don't have to worry about that step at all.
What’s the output of a UserTesting.com study? Video clips and a summary of the data, if survey questions were asked. Video clips can be downloaded to be used in Dovetail.
I’m not finding the kinds of participants I need in the UserTesting.com panel. What do I do? Reach out to @asmolinski2. It is possible to send First Look panelists a link to a UserTesting.com study, if you want to have a segment from that panel complete unmoderated studies. Note that gratuities would have to be handled by the UX Researcher, though.