How to recruit UX Research participants

At GitLab, our UX Research Operations Coordinators work closely with those requesting research to make sure the people they are gathering feedback from are the right ones. If you are filling in for a UX Research Operations Coordinator, use this page as a reference.

How to recruit participants when you aren’t sure where to find them:

1. Identify your target audience (i.e. what characteristics should they have?)
2. Craft your screener
3. Create a Calendly event
4. Open a recruitment request issue
5. Open an incentives request issue

How to recruit participants who are customers you know:

1. Identify your target audience (i.e. what characteristics should they have?)
2. Reach out to your customers and ask for their interest
3. Coordinate directly with your customers to schedule their session. This may include creating a Calendly event
4. Open an incentives request issue

How to recruit participants from issues:

1. Identify your target audience to determine what characteristics they should have
2. Craft your screener
3. Create a Calendly event
4. Solicit participants by posting a link to the screener on the related public issue
5. Open a recruitment request issue
6. Open an incentives request issue

A UX Researcher or UX Research Operations Coordinator may collaborate with you throughout the process, and we’re always available to answer questions.

Identify your target audience

Collecting the data that will help you answer your questions is essential. The only way to do that is to ensure you speak with the right people. Ask yourself the following questions to help define your target audience:

• What research methodology have you chosen?
• How many participants do you need?
• Who do you want to hear from/speak with?
  • What are the tasks your participants should be doing?
  • What are the tools your participants should be using?
  • What is their level of experience?

The answers to these questions should flow directly from the goals and objectives that you’ve previously identified for your study. It’s important that everyone involved in the research study agrees on these and that they’re finalized before we start recruiting. It’s better to start with clear objectives than to try to shoehorn participants into a study for which they simply aren’t a match.

It’s best to think of tasks and responsibilities instead of job titles, because often a person’s job title doesn’t fully correspond to the tasks they perform.

Craft your screener

The screener has a specific function - it’s meant to identify the people who are your target demographic, so that you can ask them the things you really want to know in the study. A best practice is to copy the Screener Template (internal link only) in Google Docs and collaborate with your stakeholders until all questions are finalized. For tips on how to write an effective screener, check out this page. Then, you’ll create a screener in Qualtrics that includes the questions. If you don’t have access to Qualtrics, request it. Recruiting will not begin until the finalized screener is created in Qualtrics and shared with the UX Research Operations Coordinators.

Your questions on the screener must match your participant criteria. This allows the UX Research Operations Coordinator to review your desired criteria and know which answers you want to see on the screener. The more abstract or open-ended you get in the screener, the harder it is for UX Research Operations Coordinators to parse which answer it is you’re looking for. A best practice is to avoid using open-ended questions in screeners.

Participants must agree to each of these questions to take part in moderated studies. You must include these questions in every screener:

• Consent to record the session
• Consent to store the recording in GitLab

Determine if you need the IP Assignment and/or GitLab’s Individual Contributor License Agreement

• Learn more about the IP Assignment and when to show it

Common questions we include are:

• Job title
• Common tasks
• Industry
• Team and company size
• Tools used

There are copies of these questions with standard phrasing available in the UX Research library in Qualtrics. There is also a generic template in Qualtrics based on the screener template noted above. To use this template, create a new project in Qualtrics and select the following options:

• How do you want to start your survey? Use a survey from your library
• Library UX Research & Product
• Survey UXR Generic Screener (from the Templates folder)

After selecting Create project you will be redirected to the survey builder within Qualtrics and will be able to edit the screener however you see fit.

Sometimes, we need to recruit participants from the same organization. In these cases, consider adding a question at the end of the screener to ask potential participants to recruit additional prospects (called snowball sampling). Example:

We are also interested in talking to people from the same organizations/teams, but who have different roles than yourself. Everyone’s interviews will be done separately and everyone will be paid the same amount, once their session is over. Your participation in the session is not contingent on anyone else participating. If you know of any colleagues who would be interested in taking part in this study, please provide their email and their role/job title, and we will send them a link to this screener.

Sometimes, we are able to combine efforts by using a common screener across different studies. This is called a Common Screener.

It’s important to remember that a screener is not the same as a survey. There can be a temptation to pile on extra questions that you want to know the answer to, but aren’t strictly necessary for determining whether that participant should get through the door. You will benefit from including these questions in your discussion guide or script instead.

Create a Calendly event

To schedule participants, you will need to create a Calendly event. The UX Research Operations Coordinator will send individual emails to qualified participants and ask them to book a time with you. Set up Calendly here.

1. Click ‘New Event Type’.
2. Choose ‘Create’ for the One-on-One meeting option.
3. For the Event name, include the topic of the study.
4. For Location, enter a Zoom meeting URL. A best practice is to use a recurring meeting.
5. The Description/Instructions are optional.
6. For Event link, use a short phrase to describe the topic of the sessions.
7. For Event color, choose a color you’re not currently using.
8. For Event Duration, reference your research method. Typically our user interviews and usability sessions run from 30-60 mins.
9. For Date Range, click ‘Edit’ and choose ‘Over a Date Range’ from the drop down menu. Then choose the days you wish to hold your sessions. A best practice is to give yourself 5-7 business days to conduct research sessions.
10. You can click on each day in the calendar to indicate the time(s) you are available for sessions.

Open a recruitment request issue

Prior to opening a recruitment issue, please have all materials ready:

• Screener
• Script
• Recruitment timelines

Without these materials the UX research operations coordinator will not be able to move forward with your request. When prematurely creating recruitment issues it will over-inflate our tracking on turn around time for recruitment issues.

Step 1 - Open a Recruitment Request Issue

This step provides us with all the necessary information we need to complete your request.

• Navigate to the UX Research Project and open a new issue.
• Select the template and fill in the information required.

Step 2 - Wait one-two working days

The UX Research Operations Coordinator will claim the study within one-two working days and work with you to find your participants. They will reach out to you with any further questions and an update on their timeline. When your study is claimed, the coordinator will remove the ReOps::Triage label. It will be replaced to identify they are working on it e.g. ReOps::Cait - for Caitlin Faughnan.

What if I want to get a head start, but I’m not quite ready?

If your study is not yet ready for recruitment and you’re opening the request early, please start your title with WIP or DRAFT. The UX Research Operations Coordinator will claim the study but not work on the request until the issue leaves the WIP/DRAFT phase. When the issue is ready to leave the WIP/DRAFT phase, @mention the UX Research Operations Coordinator assigned to your request.

What will the UX Research Coordinator be responsible for on my request?

The UX Research Operations Coordinator is responsible for launching recruitment campaigns, prioritizing requests, scheduling participants - if required, reimbursing them for their time, and ensuring NDAs, as needed, are signed and stored properly.

What recruitment method will be used?

The UX Research Operations Coordinator assigned to your project will select the best method for recruitment, based on the recruitment criteria. That may include using multiple approaches, depending on the criteria. Options include:

1. Data Warehouse
2. Marketo
3. Respondent.io - a third party recruitment tool
4. Social Outreach
5. Direct Sourcing via LinkedIn
6. Docs Site Banner
7. UserTesting.com - a third party recruitment tool (primarily used by Product Designers to self-serve)

Learn more about these recruitment methods.

Scheduling participants

Once the screener has been sent to potential participants, the UX Research Operations Coordinator will monitor the results and share them with the person who requested the recruiting. This is done via a Google spreadsheet. They will suggest which potential participants best fit your recruiting criteria. The DRI for the research may also provide a list of names to the UX Research Operations Coordinator from the responses that they would like invited.

Once potential participants are agreed upon, the UX Research Operations Coordinator will send the scheduling email to qualified participants via Gmail. The UX Research Operations Coordinator relies on you to update them in a timely manner as to whether any bookings have been made. You should plan to alert them within a couple of business days if no bookings have been made on your calendar, so they can send reminders or fresh invitations. They will repeat this cycle until the request is fulfilled.

Reimbursing participants for their time

The UX Research Operations Coordinator assigned to your request will handle paying the participants. However, it is up to the DRI on the project to let the coordinator know within 48 hours of session completion to ensure we are providing a positive experience for our participants by processing their incentives within a timely manner. When all participants have been paid and there’s no outstanding sessions remaining, the UX Research Operations Coordinator will close out your issue.

When do I submit an Incentives Request with UX Research Operations?

Since you created a Recruitment issue to kick off your process, this is automatically done; as this step is already built in as part of the process. That means you don’t have to do anything and the incentives will automatically be handled by the assigned UX Research Operations Coordinator.

What if I found my own participants and just need to hand out incentives?

If you have a scenario where you didn’t create a Recruitment issue and need to distribute incentives, you’ll need to open an Incentives Request issue.

• Navigate to the UX Research Project and open a new issue.
• Select the Incentives Request template and fill in the information required.
• Paste the link to the spreadsheet in the issue.

One of the UX Research Operations Coordinators will pick up your request within one business day and process the incentives for your participants, provided all the required information is in the spreadsheet. If your study is ongoing, simply comment on the issue or tag the UX Research Operations Coordinator in the spreadsheet when you have more incentives that need to be processed.

Recruitment timelines

Recruitment typically takes 2 weeks for target participants who are plentiful in our research panel and customer database, such as Engineers and DevOps professionals. Recruitment for low-incidence users, like security professionals or users of a newly released feature, may take more than 2 weeks. In rare cases, when your target participants cannot be found, the UX Research Operations Coordinator will suggest other options.

In all cases, the UX Research Operations Coordinator will keep you updated on the progress of the recruitment effort by commenting on the recruiting issue.

Promotional games

If you are planning to recruit users through a promotional game or contest (e.g., Opportunity to win 1 of 3 $30 (or equivalent currency) Amazon Gift cards), please review the following information in the handbook and consult with legal. For information on contacting legal, please refer to how to reach us in the Legal Team handbook page. Engaging legal for approval and creating an incentive request must be completed before conducting research involving promotional game or contests.

Respondent.io process and strategy

For a video overview of how to use Respondent.io, please watch this internal only recording on GitLab Unfiltered. Note: The process outlined in the video is primarily for UX Researchers. If you are interested in using Respondent.io, please contact someone from the UX Research team for assistance.

• Typically, participants begin qualifying immediately.
• Most studies will only need one Respondent campaign.
• CM Scorecards may require multiple campaigns.
• If you are looking to recruit multiple distinct segments, you will likely need one campaign per segment. This is because you set up qualifying criteria for each campaign, so participants are automatically qualified and disqualified according to the logic you set up. For example, if you want to speak with two equal segments of Jenkins users and GitHub Actions users, you should set up two campaigns with an identical screener. For one campaign, set the GitHub actions as a must select item, and set Jenkins as a must select item on the other. You may have Jenkins users be “disqualified” when they visit the first campaign. You can manually qualify them and interview them in that first project, just keep a tally of how many of each user you actually have. This gets messy, but is the quickest way to get the users you want, because Respondent will keep recruiting until they have found double the users that match your qualifying criteria.

Once the coordinator has created the campaign, they will invite the interviewers to join Respondent, advise the interviewer(s) to add their calendars and follow the project, and then move into a supporting role unless otherwise discussed. This means that the interviewer is responsible for monitoring the campaign and inviting their preferred users to schedule. The coordinator should plan to check in every few days, but it is the interviewer’s responsibility to ping the coordinator if they have questions, or if they don’t see any suitable participants after a couple of days.

• Naming your project
  • Create a public name that is not too specific; you don’t want to give away too much about your criteria because people may game your screener. Example: Seeking GitLab users! is fine, while Seeking daily GitLab CI/CD users with 3+ years of experience! is too specific.
  • Give a private name that will allow internal team members to see which project is theirs. For example: Lorie: CI/CD pipeline set up exp.
• Adding your calendar
  • Respondent requires us to schedule, pay, and communicate with participants within the platform. It’s a major taboo to try to lure platform participants elsewhere.
  • Connect your Google calendar in the calendars tab of your project.
  • Navigate to advanced settings and specify the minimum time before new bookings to 8 hours or more. This will prevent people from scheduling surprise sessions for first thing in the morning while you are sleeping.
  • Set the max number of bookings per day to whatever you are comfortable with. A recommendation is 3-4 sessions per day: more than that is quite taxing.
  • Set a buffer time between sessions so you have time to finish your notes and take a quiick break before starting another session. A recommendation is 15 minutes.
• Using Zoom When setting up your Respondent project make sure to use your personal Zoom room link, as you can’t change the link per participant (this means each participant will have the same Zoom room link). Additionally, be sure to turn off the password requirement for these sessions.
• Important platform etiquette:
  • Mark participants as attended promptly after conducting the interview. This creates a notification for the coordinator that there are participants who need to be paid.
  • Only mark someone as no-show if they made no effort to get in touch with you about rescheduling the interview. A no-show is highly punitive for participants’ rating on the platform.
  • Similarly, if you must reschedule a session, enable the participant to reschedule with as much advance notice as possible. If you do not show up for a session, or cancel within 24 hours, we get a negative rating and have to pay the participant anyway.
  • Please keep an eye on your messages within the project. It is recommended that you check them first thing on days that you have interviews scheduled. Participants may reach out to reschedule, or to ask about a technical issue. It’s best to address these up front, rather than waiting in an empty Zoom and getting increasingly frustrated 🙂

Process for providing payment through Respondent.io

Respondent requires us to pay participants directly through the platform.

In order to pay participants in Respondent.io, there are several steps required:

1. Select your project (after launching a study) in Respondent.io
2. Locate people in the Participants tab who have successfully completed the study and click on the Mark As Attended button
3. Fill out a quick survey question to rate the participant (i.e., Positive, Negative, Neutral)
4. Repeat steps 2 and 3 as needed
5. Go to the Payments tab
6. Look at the Payment Method dropdown to check that it says Credit Balance (this is the default option)
7. Select pay button (Note: You can specify a tip above and beyond the set incentive cost if the participant has done additional work, but this is not commonly done)

The payment funds come out of a built in balance. Respondent.io shows this running balance within the Payments Tab and at the top left hand corner of the page.

Recruitment case study

To understand how the above process works in action, the following research study is a great example:

Accessibility solution validation research issue

Identifying target respondents

The stated goals of the study were:

As part of our continued effort to improve the automated testing capabilities of GitLab, we want to ship a new feature for accessibility testing. We want to validate some solutions for this feature. Questions we have are:

• How exactly users want to see Accessibility issues introduced during their development shown to them.
• Which of the proposed designs suffice their use case for Automated Accessibility Testing as part of the CI process.

How many participants did they need?

It was a usability test, so around five users is generally the sweet spot.

What was the timeline?

The Product Designer stated that they were ready to begin any time, and notified the UX Research Operations Coordinator of their vacation time.

Who did they want to speak with?

Frontend engineers was submitted, but that’s too broad. Instead, focusing on their day-to-day activities helped the UX Research Operations Coordinator find accurate matches.

Crafting the screener

In this example, the key questions that recruiting hinged on were really clear: the UX Research Operations Coordinator expected to see a question about job title, common tasks participants performed or job responsibilities they had, and something about how they perceived the importance of accessibility compliance. The UX Research Operations Coordinator also expected to see things that we almost always include, like company size and industry, which we try to get a mixture of when they’re not the object of the study.

This study is unusual in that we’re not asking about what tools people use. Often, we explicitly want to see what someone with or without GitLab experience thinks. It wasn’t relevant for this study, so we’re not collecting that information.

Keeping things short and sweet increases the completion rate of screeners.

Check out the finished screener

Opening a recruitment request

A lot of the conversation above occurred in the recruitment request.

The UX Research Operations Coordinator took great care in creating a segment from our research panel. Because the study was about accessibility, a topic about which many people are passionate, this study was a good one to use custom subject lines and email copy. The UX Research Operations Coordinator sent out an email with the subject line New study - let's talk accessibility. After two emails, we received over 20 responses.

The UX Research Operations Coordinator highlighted in green the people who said they do accessibility testing in their jobs, and strongly agreed that accessibility compliance is important (4 total). They highlighted in orange people who only fit the latter criterion (9). Altogether, this is a surplus of participants, however, we now have some participants that we can reach out to directly when the next accessibility study is available.