At GitLab, our UX Research Operations Coordinators work closely with those requesting research to make sure the people they are gathering feedback from are the right ones. If you are filling in for a UX Research Operations Coordinator, use this page as a reference.
A UX Researcher or UX Research Operations Coordinator may collaborate with you throughout the process, and we’re always available to answer questions.
Collecting the data that will help you answer your questions is essential. The only way to do that is to ensure you speak with the right people. Ask yourself the following questions to help define your target audience:
The answers to these questions should flow directly from the goals and objectives that you’ve previously identified for your study. It’s important that everyone involved in the research study agrees on these and that they’re finalized before we start recruiting. It’s better to start with clear objectives than to try to shoehorn participants into a study for which they simply aren't a match.
It’s best to think of tasks and responsibilities instead of job titles, because often a person’s job title doesn’t fully correspond to the tasks they perform.
The screener has a specific function - it’s meant to identify the people who are your target demographic, so that you can ask them the things you really want to know in the study. A best practice is to copy the Screener Template (internal link only) in Google Docs and collaborate with your stakeholders until all questions are finalized. For tips on how to write an effective screener, check out this page. Then, you’ll create a screener in Qualtrics that includes the questions. If you don’t have access to Qualtrics, request it. Recruiting will not begin until the finalized screener is created in Qualtrics and shared with the UX Research Operations Coordinators.
Your questions on the screener must match your participant criteria. This allows the UX Research Operations Coordinator to review your desired criteria and know which answers you want to see on the screener. The more abstract or open-ended you get in the screener, the harder it is for UX Research Operations Coordinators to parse which answer it is you’re looking for. A best practice is to avoid using open-ended questions in screeners.
Participants must agree to each of these questions to take part in moderated studies. You must include these questions in every screener:
Determine if you need the IP Assignment and/or GitLab's Individual Contributor License Agreement
Common questions we include are:
There are copies of these questions with standard phrasing available in the UX Research library in Qualtrics. There is also a generic template in Qualtrics based on the screener template noted above. To use this template, create a new project in Qualtrics and select the following options:
Use a survey from your libraryUX Research & ProductUXR Generic Screener (from the Templates folder)After selecting Create project you will be redirected to the survey builder within Qualtrics and will be able to edit the screener however you see fit.
Sometimes, we need to recruit participants from the same organization. In these cases, consider adding a question at the end of the screener to ask potential participants to recruit additional prospects (called snowball sampling). Example:
We are also interested in talking to people from the same organizations/teams, but who have different roles than yourself. Everyone's interviews will be done separately and everyone will be paid the same amount, once their session is over. Your participation in the session is not contingent on anyone else participating. If you know of any colleagues who would be interested in taking part in this study, please provide their email and their role/job title, and we will send them a link to this screener.
Sometimes, we are able to combine efforts by using a common screener across different studies. This is called a Common Screener.
It’s important to remember that a screener is not the same as a survey. There can be a temptation to pile on extra questions that you want to know the answer to, but aren’t strictly necessary for determining whether that participant should get through the door. You will benefit from including these questions in your discussion guide or script instead.
To schedule participants, you will need to create a Calendly event. The UX Research Operations Coordinator will send individual emails to qualified participants and ask them to book a time with you. Set up Calendly here.
Prior to opening a recruitment issue, please have all materials ready:
Without these materials the UX research operations coordinator will not be able to move forward with your request. When prematurely creating recruitment issues it will over-inflate our tracking on turn around time for recruitment issues.
Step 1 - Open a Recruitment Request Issue This step provides us with all the necessary information we need to complete your request.
Step 2 - Wait one working day
One of the UX Research Operations Coordinators will claim the study within one working day and work with you to find your participants. They will reach out to you with any further questions and an update. When your study is claimed, the coordinator will add a label to identify they are working on it. ReOps::Cait - for Caitlin Faughnan. ReOps::Jenn - for Jennifer Mundy
If your study is not yet ready for recruitment and you're opening the request early, please start your title with WIP or DRAFT. The UX Research Operations Coordinator will claim the study but not work on the request until the issue leaves the WIP/DRAFT phase. When the issue is ready to leave the WIP/DRAFT phase, @mention the UX Research Operations Coordinator assigned to your request.
The UX Research Operations Coordinator is responsible for launching recruitment campaigns, prioritizing requests, scheduling participants, reimbursing them for their time, and ensuring NDAs, as needed, are signed and stored properly.
The UX Research Operations Coordinator assigned to your project will select the best method for recruitment, based on the recruitment criteria. That may include using multiple approaches, depending on the criteria. Options include:
More detailed information on the above recruitment methods can be found here.
Once the screener has been sent to potential participants, the UX Research Operations Coordinator will monitor the results and share them with the person who requested the recruiting. This is done via a Google spreadsheet. They will suggest which potential participants best fit your recruiting criteria. The DRI for the research may also provide a list of names to the UX Research Operations Coordinator from the responses that they would like invited.
Once potential participants are agreed upon, the UX Research Operations Coordinator will send the scheduling email to qualified participants via Gmail. The UX Research Operations Coordinator relies on you to update them in a timely manner as to whether any bookings have been made. You should plan to alert them within a couple of business days if no bookings have been made on your calendar, so they can send reminders or fresh invitations. They will repeat this cycle until the request is fulfilled.
The UX Research Operations Coordinator assigned to your request will handle paying the participants. However, it is up to the DRI on the project to let the coordinator know within 48 hours of session completion to ensure we are providing a positive experience for our participants by processing their incentives within a timely manner. When all participants have been paid and there's no outstanding sessions remaining, the UX Research Operations Coordinator will close out your issue.
Since you created a Recruitment issue to kick off your process, this is automatically done; as this step is already built in as part of the process. That means you don’t have to do anything and the incentives will automatically be handled by the assigned UX Research Operations Coordinator.
If you have a scenario where you didn’t create a Recruitment issue and need to distribute incentives, you’ll need to open an Incentives Request issue.
Incentives Request template and fill in the information required.One of the UX Research Operations Coordinators will pick up your request within one business day and process the incentives for your participants, provided all the required information is in the spreadsheet. If your study is ongoing, simply comment on the issue or tag the UX Research Operations Coordinator in the spreadsheet when you have more incentives that need to be processed.
Recruitment typically takes 2 weeks for target participants who are plentiful in our research panel and customer database, such as Engineers and DevOps professionals. Recruitment for low-incidence users, like security professionals or users of a newly released feature, may take more than 2 weeks. In rare cases, when your target participants cannot be found, the UX Research Operations Coordinator will suggest other options
In all cases, the UX Research Operations Coordinator will keep you updated on the progress of the recruitment effort by commenting on the recruiting issue.
If you are planning to recruit users through a promotional game or contest (e.g., Opportunity to win 1 of 3 $30 (or equivalent currency) Amazon Gift cards), please review the following information in the handbook and consult with legal. For information on contacting legal, please refer to how to reach us in the Legal Team handbook page. Engaging legal for approval and creating an incentive request must be completed before conducting research involving promotional game or contests.
GitHub actions as a must select item, and set Jenkins as a must select item on the other. You may have Jenkins users be "disqualified" when they visit the first campaign. You can manually qualify them and interview them in that first project, just keep a tally of how many of each user you actually have. This gets messy, but is the quickest way to get the users you want, because Respondent will keep recruiting until they have found double the users that match your qualifying criteria.Once the coordinator has created the campaign, they will invite the interviewers to join Respondent, advise the interviewer(s) to add their calendars and follow the project, and then move into a supporting role unless otherwise discussed. This means that the interviewer is responsible for monitoring the campaign and inviting their preferred users to schedule. The coordinator should plan to check in every few days, but it is the interviewer's responsibility to ping the coordinator if they have questions, or if they don't see any suitable participants after a couple of days.
Seeking GitLab users! is fine, while Seeking daily GitLab CI/CD users with 3+ years of experience! is too specific.Lorie: CI/CD pipeline set up exp.calendars tab of your project.8 hours or more. This will prevent people from scheduling surprise sessions for first thing in the morning while you are sleeping.Using Zoom When setting up your Respondent project make sure to use your personal Zoom room link, as you can't change the link per participant (this means each participant will have the same Zoom room link). Additionally, be sure to turn off the password requirement for these sessions.
attended promptly after conducting the interview. This creates a notification for the coordinator that there are participants who need to be paid.no-show if they made no effort to get in touch with you about rescheduling the interview. A no-show is highly punitive for participants' rating on the platform.Respondent requires us to pay participants directly through the platform.
In order to pay participants in Respondent.io, there are several steps required:
The payment funds come out of a built in balance. Respondent.io shows this running balance within the Payments Tab and at the top left hand corner of the page.
To understand how the above process works in action, the following research study is a great example:
Accessibility solution validation research issue
The stated goals of the study were:
As part of our continued effort to improve the automated testing capabilities of GitLab, we want to ship a new feature for accessibility testing. We want to validate some solutions for this feature. Questions we have are:
- How exactly users want to see Accessibility issues introduced during their development shown to them.
- Which of the proposed designs suffice their use case for Automated Accessibility Testing as part of the CI process.
How many participants did they need?
It was a usability test, so around five users is generally the sweet spot.
What was the timeline?
The Product Designer stated that they were ready to begin any time, and notified the UX Research Operations Coordinator of their vacation time.
Who did they want to speak with?
Frontend engineers was submitted, but that's too broad. Instead, focusing on their day-to-day activities helped the UX Research Operations Coordinator find accurate matches.
In this example, the key questions that recruiting hinged on were really clear: the UX Research Operations Coordinator expected to see a question about job title, common tasks participants performed or job responsibilities they had, and something about how they perceived the importance of accessibility compliance. The UX Research Operations Coordinator also expected to see things that we almost always include, like company size and industry, which we try to get a mixture of when they’re not the object of the study.
This study is unusual in that we're not asking about what tools people use. Often, we explicitly want to see what someone with or without GitLab experience thinks. It wasn't relevant for this study, so we're not collecting that information.
Keeping things short and sweet increases the completion rate of screeners.
Check out the finished screener
A lot of the conversation above occurred in the recruitment request.
The UX Research Operations Coordinator took great care in creating a segment from our research panel (GitLab First Look). Because the study was about accessibility, a topic about which many people are passionate, this study was a good one to use custom subject lines and email copy. The UX Research Operations Coordinator sent out an email with the subject line New study - let's talk accessibility. After two emails, we received over 20 responses.
The UX Research Operations Coordinator highlighted in green the people who said they do accessibility testing in their jobs, and strongly agreed that accessibility compliance is important (4 total). They highlighted in orange people who only fit the latter criterion (9). Altogether, this is a surplus of participants, however, we now have some participants that we can reach out to directly when the next accessibility study is available.