The short answer is approximately 5-10 at any given time. The long answer is that it depends on the type of study and the target population. When each study only requires 5-8 participants who are relatively abundant in our database and community, the number of studies recruited per milestone could be ~10. This is also contingent upon the Product Designer and Product Manager submitting their complete requests early enough to allow coordinators to juggle multiple recruiting efforts at once. When there are surveys that require 50+ respondents, or studies addressing new user groups or features, the number of studies recruited per milestone may be closer to 5. Those studies require sustained effort and creativity.
Never hesitate to bring up a recruitment request - the UX Research Operations Coordinator will work with you to schedule the recruiting effort. Feel free to reach out to @Caitlin on Slack.
When the number of requests exceeds 10, the coordinator is at over capacity. When this occurs, there are too many projects to successfully manage at once, and the following side effects are experienced internally and externally to GitLab:
To deal with situations of over capacity, we have two plans of action:
Given the nature of how we conduct research at GitLab, it’s difficult to have an accurate view into the pipeline at any given time to understand what research projects will be coming up when. However, we can rely on research project volume data from the past and predict the cadence of some projects quarterly to plan accordingly. An example of identified projects that require extra coordination are:
A quarterly planning session is conducted at the start of each quarter. The planning exercise maps out the upcoming research requests, by month, with known research projects, predicted research projects, upcoming events requiring research participants, anticipated changes to our processes (ex: new hires, unmoderated testing, etc), and coordinator unavailability. In addition, check-ins at the beginning of each month take place between UX Research Operations Coordination and the UX Research Director.
The goal with the quarterly planning exercise is to have a fairly accurate view into the quarter at any given time, be able to identify any potential delivery risks, and to start early on addressing those risks with mitigation plans.
The coordinator is responsible for driving and maintaining the research coordination quarterly planning exercise, the monthly check-ins, and keeping the Google Sheet up to date and accurate.
To keep research projects on track, at least 2 other UX Researchers are trained to work as UX Research Operations Coordinators when UX Research Operations Coordinator are unavailable or if they are at over capacity and need additional assistance.
The UX Research Operations Coordinator is responsible for:
Example coverage areas for training:
Ideally, a back-up plan is rarely used if proper planning has been done.
Even with accurate planning, over-capacity can still occur. When this happens, coordinators have several actions they can take:
Inform the team - Be transparent. Let the #ux_research channel know, along with managers, that UX Research Operations Coordination is over capacity at the moment along with an estimated timeframe on when they expect things to get back to normal. This automatically sets expectations for existing and new requests. Lastly, follow-up with a post when capacity returns to normal to let the channel know.
Reset expectations on any deliverable dates - Communicate to the key people on the projects impacted that their deliverable dates may change.
Inform participants - Be transparent. If participants are negatively impacted, let them know that we’re busy, express a sincere apology, and reset timing expectations with them.
Ask for help - UX Research Operations Coordinators need to ask their trained UX Research Team back-ups for help. UX Research Operations Coordinators will need to delegate the right duties to the UX Researchers to result in the biggest impact.
New incoming requests will take longer - We don’t stop taking on new research when we’re at over capacity. Instead, we set expectations early with new requests. New requests will be buffered accordingly to accommodate being at over capacity and project owners are educated as to why.
Encourage self-service - UX Research Operations Coordinators can point people to self-serve resources with a link to instructions in the handbook (ex: using Respondent and User Interviews to find participants, etc).
Since UX Research Operations Coordinators have unique duties that are critical to the success of research at GitLab, when they are unavailable, recruitment efforts slow down. This results in a series of problems for all parties.
To address this, two UX Researchers are available to act as back-up when coordinators are unavailable. Back-ups complete any in-progress tasks, accept new requests, and keep existing requests on track.
Prior to the coordinator becoming unavailable, they create an issue to provide their assigned back-up(s) with a clearly detailed list of:
It’s the responsibility of the coordinator to make sure their back-up understands their duties and responsibilities.
In some cases, absolutely. This works best when two or more studies are quite similar (e.g. both require interviews on a similar timeline and have an identical screener).
It is not usually advisable to combine recruitment for studies that are quite different from each other, for example a large survey and a set of user interviews. This is due to the different timeframes for these different studies.
In order to get people scheduled for interviews, we generally want to keep the feedback loop as tight as possible - by inviting qualified participants to schedule for an interview asap after they respond to the screener. This means that the whole recruitment process might take only a matter of days. Conversely, surveys requiring a large number of responses may require multiple rounds of emails through Qualtrics over a period of weeks. By the time you remember to reach out to some users who responded early on, so much time has passed that you're likely to see a very low response rate. This results in us needing to recruit twice.
GitLab First Look is a group of people who have opted in to receive research studies from GitLab. This panel is a good fit for studies that are aimed at GitLab users who are software developers and related roles.
Respondent.io is a recruitment service that is a good choice for studies aimed at software professionals who are not necessarily GitLab users. This has been a decent source of security professionals and some other harder-to-reach users. Respondent is also a great choice for when you need users quickly, or are juggling many studies.
UserTesting.com is a unmoderated usability testing platform. We can use their panel to recruit users for usability tests using their platform.
Social outreach. Social posts may go out on GitLab's brand channels. The UX Research Operations Coordinator uses the Social Request template in the corporate marketing project to request this type of post. This is a good choice for studies primarily aimed at GitLab users. Product Managers, Product Designers, and other teammates are highly encouraged to help promote their studies to their networks.
Direct Sourcing. We can utilize LinkedIn to approach and source potential participants. Anyone at GitLab is able to request a LinkedIn Recruiter license. This Unfiltered video and slide deck provide an overview on how to use Linkedin Recruiter to source participants for your study. Please note that your LinkedIn Recruiter seat is likely limited to sending only 200 InMails your first week. If you go over this amount, your account will be disabled for a week. After the first week, you can send up to 1000 InMails a day. If you go over this limit, your account will be disabled for a day.
Docs site banner. We can occasionally use a banner on the docs site to promote larger surveys. It is a lightweight way to quickly fulfill surveys that otherwise would be restricted to First Look. This method is typically reserved for high priority research team initiatives. The docs team may decline any banner request.
GitHub actions as a must select item, and set Jenkins as a must select item on the other. You may have Jenkins users be "disqualified" when they visit the first campaign. You can manually qualify them and interview them in that first project, just keep a tally of how many of each user you actually have. This gets messy, but is the quickest way to get the users you want, because Respondent will keep recruiting until they have found double the users that match your qualifying criteria.Once the coordinator has created the campaign, they will invite the interviewers to join Respondent, advise the interviewer(s) to add their calendars and follow the project, and then move into a supporting role unless otherwise discussed. This means that the interviewer is responsible for monitoring the campaign and inviting their preferred users to schedule. The coordinator should plan to check in every few days, but it is the interviewer's responsibility to ping the coordinator if they have questions, or if they don't see any suitable participants after a couple of days.
Seeking GitLab users! is fine, while Seeking daily GitLab CI/CD users with 3+ years of experience! is too specific.Lorie: CI/CD pipeline set up exp.calendars tab of your project.8 hours or more. This will prevent people from scheduling surprise sessions for first thing in the morning while you are sleeping.Using Zoom When setting up your Respondent project make sure to use your personal Zoom room link, as you can't change the link per participant (this means each participant will have the same Zoom room link). Additionally, be sure to turn off the password requirement for these sessions.
attended promptly after conducting the interview. This creates a notification for the coordinator that there are participants who need to be paid.no-show if they made no effort to get in touch with you about rescheduling the interview. A no-show is highly punitive for participants' rating on the platform.Ask any questions in #ux_research.
Respondent requires us to pay participants directly through the platform.
In order to pay participants in Respondent.io, there are several steps required:
The payment funds come out of a built in balance. Respondent.io shows this running balance within the Payments Tab and at the top left hand corner of the page.
As a small token of thanks (and to ensure a higher participation/completion rate) we send gifts to research participants. These can be requested by opening an issue in the UX research project.
| Interview Length | Incentive |
|---|---|
| 30 Minutes | $60 |
| 45 Minutes | $90 |
| 60 Minutes | $120 |
| 75 Minutes | $150 |
| 90 Minutes | $180 |
| 2 Hours | $240 |
Surveys, card sorts, and tree tests: Opportunity to win 1 of 3 $30 (or equivalent currency) Amazon Gift cards, or a GitLab swag item.
Design evaluations: Unpaid.
Swag can also be sent on an ad hoc basis, as requested by researchers, PMs, and others. This is a nice touch and an opportunity for personalization after a particularly good conversation. UX Research Operations Coordinators can reach out to the corporate events team (Emily Kyle) for swag codes.
If you are planning to recruit users through a promotional game or contest (e.g., Opportunity to win 1 of 3 $30 (or equivalent currency) Amazon Gift cards), please review the following information in the handbook and consult with legal. For information on contacting legal, please refer to how to reach us in the Legal Team handbook page. Engaging legal for approval and creating an incentive request must be completed before conducting research involving promotional games or contests.
Requests for thank you gifts should be fulfilled at least twice per week so that users receive their gift promptly after participating in research. It's important to maintain a customer service mindset when interacting with participants - receiving their gift is the end of the research cycle and another touchpoint in their relationship with GitLab. Participants who take part through Respondent will have their incentives processed the next business day by the UX Research Operations Coordinator - only if the UX Researcher on the project has marked the participants as attended.
Tango Card/Rewards Genius can be used to send gift cards to users around the world. View a recording (GitLab internal) going through the process of sending a Tango Card payment to a research participant.
Here are suggestions for the most frequently used rewards:
| Country | Vendor |
|---|---|
| US | Reward link preferred* |
| India | Amazon.in |
| Netherlands | Amazon.nl |
| Canada | Amazon.ca |
| UK | Amazon.co.uk or Amazon.co.uk Ireland |
| Australia | Amazon.com.au |
| Germany | Amazon.de |
| Austria | Amazon.de Austria |
| Poland | Amazon.de Poland |
| Luxembourg | Amazon.de Luxembourg |
| Spain | Amazon.es |
| France | Amazon.fr |
| Italy | Amazon.it |
| Argentina | Reward link Argentina |
| Belgium | Reward link Belgium |
| Brazil | Reward link Brazil |
| Portugal | Reward link Portugal |
| Singapore | Reward Link Singapore |
| Saudi Arabia | Reward link Saudi Arabia |
| UAE | Reward Link UAE |
| Other | Pre-paid mastercard** |
*Reward Link gifts contain multiple local vendors for recipients to choose from.
**We send a pre-paid Mastercard to all participants in countries that are unsupported by TangoCard. These cards were designed for international travelers and are accepted everywhere, even if the card's currency does not match the participant's local currency.
Contact success@tangocard.com with questions or to expand the options available in the catalog.
Rybbon is another incentive distributor that we use, particularly when the country a participant is based in is unavailable on Tango Card/Rewards Genius.
Rybbon uses an API feature to present rewards that are relevant for each participant based on their location. As such, we only provide the USD reward value on the platofrm and Rybbon handles the conversion automatically. Each researcher has their own folder with a campaign set up to send incentives. This is so that the email communication that our participants recieve is more personable. There's also a general folder for the UX Research Operations Coordinator for incentive requests from the wider team. Specific campaigns can also be set up for longer projects, as well as an integration with Qualtrics if required.
Unsupported Countries in Rybbon: No rewards are available in the following countries. Belarus, Congo, Cuba, Iran, Iraq, Korea North, Lebanon, Libya, Somalia, South Sudan, Sudan, Syria, Ukraine, Yemen, Zimbabwe.
Check out Rybbon's FAQ page or reach out to Caitlin on Slack with any questions.
The accounting team funds Tango Card/Rewards Genius, Rybbon, and Respondent accounts with a lump sum amount from the pre-approved research incentives budget. We issue thank you gifts from that prepaid amount.
Reporting on spending:
The UX Research Operations Coordinator sends a report by the 1st of every month to ap@gitlab.com with the following information:
The report should be for the previous month (e.g., the report should be sent by 11/1 for cards send between 10/1 and 10/31).
Requesting new funds to the account:
Funding in the left side navigationllamb@gitlab.com with any questions about GitLab's accounting process.funding@tangocard.com with any questions about Tango Card's funding process.Reporting on spending:
The UX Research Operations Coordinator sends a report by the 1st of every month to jmiller@gitlab.com with the following information:
Requesting new funds to the account:
When sending emails to GitLab First Look, coordinators may customize emails in a few ways.
I hope you had a great weekend! We have a new study available and would love to include your perspective! Here are all the details:We have a new study for you!; See if you're a match with our new study; etc.)sent from display name to Name from GitLabWe still want to hear from you!; Reminder: Take our quick survey)Below are email/message templates that you can use for communicating with research participants. Whenever possible, we strongly encourage you to customize each message for the individual and circumstance.
SURVEY: How would you score GitLab's usability? New! Help us measure GitLab's usability over time Survey for you! How would you rate GitLab's usability?It's time for our regular System Usability Scale survey! This is a critical research project that helps us measure GitLab's usability over time, and we can't do it without you!
Your feedback is really valuable and it will only take a few minutes - will you help us out?
As a thank you for your time, you'll have the opportunity to win 1 of 3 $30 (or equivalent currency Amazon gift cards).
Hi! We have a study coming up, and I was wondering if you might be interested in participating with us. Based on your response to our survey, you look like a great fit! Sessions are taking place from `[XX-XX]`, and they last about `[XX]` minutes over Zoom (videoconference).
For this round of testing, we’ll be chatting about `[Replace with research subject. Example: What tools you use, what your process is like, etc.]`.
Participants who complete a session will be compensated with a `[Replace with compensation. Example: $60 Amazon gift card, or approximate value in your home currency]`.
If you are interested, go here `[Link to your Calendly]` and choose one time and day that works for you as soon as possible. There are limited spots available.
Please let me know if you have any questions.
Hi all! :wave: We are in the process of `[Replace with research subject]` to `[Replace with research goals for context]`.
We need internal customers to answer a few questions. If you would like to help us out, please reply to this survey `[Link to research survey]`. Thank you!
The GitLab R&D department conducts a substantial amount of research, and it is challenging to drive awareness of all research initiatives for which we need participation from internal users and members of the wider GitLab community. To drive awareness, we need to seize opportunities to promote GitLab First Look wherever possible. For example:
UX Research Operations Coordinators should set aside time to interact with users on social. It's recommended to post any new UX blog posts and search gitlab ux at least once per week.
When appropriate, link to GitLab First Look (and grab an image to upload) in responses to comments about GitLab's UX. This is important for raising awareness of the research program, growing the panel, and performing research with users who may have had negative experiences with GitLab.
Some sample responses:
Thank you so much! Would you consider joining our research program if you haven't already? You can specify which areas of the product you're interested in: https://about.gitlab.com/community/gitlab-first-look/I'm so sorry to hear you've had a negative experience! Would you consider giving feedback through our research program? You can specify exactly which parts of the product you're interested in: https://about.gitlab.com/community/gitlab-first-look/At the conclusion of each UX Research Group Conversation, the UX Research Operations Coordinator will aim to choose one of the presented research projects to showcase, provided a project meets the criteria below. The goal of showcasing our work is to not only let the community know about the great work the team is doing, but to also let our participants and users know how their valuable feedback has impacted the user experience within the GitLab product.
Projects that we want to showcase are chosen on the following criteria:
The UX Research Operations Coordinator will take the following steps to showcase a research project:
In addition to highlighting a single research project, we want to broadcast out all of the research insights that teams have found from their various research projects. Since most studies require UX Research Operations Coordinator involvement, they're able to track the details of specific studies. The UX Research Operations Coordinator will work with the DRIs of research projects to obtain the following information:
That information will be compiled and shared out through Slack (channels: #ux-research and cross-post to: #ux, #product) in a 2-3 week cadence (or whenever there are insights to share out).
At GitLab, when we communicate with our research study participants, we take GDPR and CAN-SPAM requirements seriously.