Gitlab hero border pattern left svg Gitlab hero border pattern right svg

UX Research

  1. You are here:
  2. Handbook
  3. Engineering
  4. UX Department
  5. UX Research

On this page

UX Research

The goal of UX Research at GitLab is to connect with GitLab users all around the world and gather insight into their behaviors, motivations and goals when using GitLab. These insights are then used to inform and strengthen product and design decisions.

How to request research

  1. For usability testing, user interviews, card sorts, surveys or if you are not sure what form of research needs to take place, create a new issue using the Research proposal template in the UX research project.

    For beta testing, create a new issue using the Beta testing proposal template in the UX research project.

  2. @ mention the relevant UX Researcher, UX Designer, and Product Manager for the product stage. Ensure you answer all questions outlined in the template.

    • You can find out who the relevant UX Researcher and/or UX Designer is by looking at the team page and filtering by the UX department.

    • Anybody across GitLab can raise a proposal, this includes UX Researchers.

  3. The UX Researcher will review the issue and may respond with some follow-up questions.

    • We want to lessen the time that researchers spend within issues and/or Slack soliciting research requirements. As a rule of thumb, if we have gone back and forth more than 3 times, it's time for a video call.

    • Similarly, some features are more complex than others. A UX Researcher needs to fully understand the feature they are testing and why they are testing it. Sometimes, it's much easier to get a grasp on the feature's history, your existing plans and plans for the future when you talk with us. In cases such as these, the UX Researcher will schedule a kick-off call with all relevant stakeholders (including the UX Designer and Product Manager) to run through the research proposal.

  4. In collaboration with the Product Manager, the UX Researcher will determine the priority of the study and schedule it accordingly.

  5. The UX Researcher will book a wash-up meeting with all relevant stakeholders when the results of the study are available.

Wash-up meetings

  1. During a wash-up meeting, all stakeholders collaboratively agree upon a list of improvements/actions to take forward. UX Researchers are responsible for scheduling wash-up meetings. At least 24 hours prior to the wash-up meeting, a Google document containing the research study’s key findings will be added to the meeting’s calendar invitation.

  2. In order to ensure that the meeting is an effective use of everybody’s time, you should familiarize yourself with the study’s key findings prior to the meeting and come prepared with suggestions.

How to create a design evaluation (guidance for UX Designers)

  1. Create a new issue using the Design evaluation template in the UX research project and @ mention the relevant UX Researcher and Product Manager for the product stage.

  2. Update the Design evaluation with the following:
    • Add as much information as you can to the issue.
    • Label the issue with the area of GitLab you’re testing (for example, navigation), the status of the issue (in progress) and the Product stage (for example, manage).
    • Mark the issue as confidential until the research is completed so it doesn’t influence user behavior.
    • Assign the issue to yourself.

  3. The UX Researcher will quickly review the issue and advise whether it is feasible to send the study during the time frame you have requested (Occasionally, we may need to stagger when the study is sent since we do not want to bombard users with research studies). The UX Researcher will add a milestone to the issue.

  4. Create your test in UsabilityHub. UX Designers share a login for UsabilityHub. The credentials are stored in 1Password. Some tips for creating a test:
    • Use a descriptive test name. Make sure it includes the issue number of the research proposal.
    • We don't incentivize users to participate in design evaluations. In order to encourage participation, keep the test length under 10 minutes (UsabilityHub will provide you with an estimated completion time as you build your test).
    • Try to avoid priming users. For example, when writing tasks for a first click or navigation test, where possible, avoid using terms that appear in the interface or which give the answer away.
    • Always question whether you are using the right research methodology. For example, if you are asking lots of straight-up questions, would it be best to ditch the design evaluation and write a survey instead? Check out the UsabilityHub guides for common use cases for each methodology.
    • Remember what users say and what they do can be very different. Just because users prefer a design, doesn't mean they can use it. Find out more about when you should use preference tests.
    • Read more about how to test visual design.
    • If you're ever in doubt when creating your test, reach out to a UX Researcher!

  5. When your test is ready, update the Design evaluation issue with any outstanding information and ping the relevant UX Researcher for a final review.

  6. Once you and the UX Researcher are happy with the test. The UX Researcher will distribute the test to subscribers of GitLab First Look.

  7. When you and the UX Researcher feel that sufficient data has been collected. The test can be closed in UsabilityHub.

  8. You are responsible for analyzing any tests you have created. You are always welcome to reach out to a UX Researcher for assistance.

  9. Document the study within a Google document. The report should consist of:
    • Research hypotheses
    • Research methodology
    • Findings
    • Next steps/recommendations.
  10. Update the Design evaluation issue with the following:
    • Link to the report.
    • Unmark the issue as confidential.
    • Update the status of the issue to done.
    • Close the issue. You should stay assigned to closed issues so it's obvious who completed the research.

How to review a design evaluation (guidance for UX Researchers)

  1. UX Researchers should familiarize themselves with the process outlined in How to create a design evaluation (guidance for UX Designers)

  2. When sending a design evaluation to GitLab First Look users, ensure standard processes are followed. For example, distributing the study to a test sample of users, sending the study to GitLab First Look subscribers who have opted in to design evaluations from the relevant product stage, etc.

Beta Testing at GitLab

Aims

FAQs about Beta Testing

  1. How can I request beta testing?

    You can request beta testing by following the instructions outlined in 'How to request research'.

  2. How do users see a feature in beta?

    First iteration: Test feature(s) as they are added to a release candidate.

    Future iterations: Route beta users to a Canary version of GitLab that runs pre-release code. Use feature flags.

  3. How do users provide feedback about a feature?

    By completing a survey.

  4. What if I don't know what release candidate the feature will be added in?

    That's okay. All survey questions should be finalized in advance of the first release candidate, so that the UX Researcher can build the survey and the accompanying mail campaign ahead of time.

    When the feature is added to a release candidate, ping the relevant UX Researcher in the issue you have created. The UX Researcher will then distribute the survey to users.

  5. What if the feature is added to a late release candidate? Is there any point in still beta testing the feature?

    Yes, you can still beta test the feature.

    The overall aims of your study will be to:

    • Generate feature awareness.
    • Identify bugs and/or improvements for a feature.
    • Gather general user feedback.

    It may be beneficial to delay sending the survey until the official release on the 22nd in order to save users the effort of downloading a release candidate.

  6. Can I request beta testing for a feature that isn't labeled as a beta release?

    Yes, you can test any feature in an upcoming release candidate/milestone.

  7. How long will a feature be tested for?

    This depends on which release candidate the feature is added to and what the ultimate aims of the study are.

    If the feature is added to an early release candidate. Your main aim might be to:

    • Collect quotes from users that we can use in release posts.

    In which case, there's not much point running the survey past the 22nd of the month.

    If the feature is added to a later release candidate. Your main aims might be to:

    • Generate feature awareness.
    • Identify bugs and/or improvements for a feature.
    • Gather general user feedback.

    Therefore, we recommend running the survey until it stops receiving responses.

  8. What are some example questions that I could ask users?

    • How could the feature be improved? (Open text)
    • What, if anything, didn’t work as you expected it to? (Open text)
    • Does this feature help you accomplish X? (Rating scale)
      • Please explain your answer (Open text)
    • What triggers would prompt you to use this feature? (Open text)
    • How likely is it that you would use this feature? (Rating scale)
      • Why is it unlikely that would use this feature? (Open text)
    • What do you most like about the feature? (Open text)
    • Overall, how easy was it to use the feature? (Rating scale)

GitLab First Look

GitLab First Look (formerly the UX Research Panel) is a group of users that have opted in to receive research studies from GitLab. GitLab First Look is managed and maintained by the UX Research team. To find out more or to join, please visit GitLab First Look.

UX Researchers

UX Researcher onboarding

If you are just starting out here at GitLab, welcome! Make sure to review all the pages here in the UX Research section of the handbook, they will help you get oriented. There is also a specific page dedicated to UX Researcher onboarding.

Research methods

Research methods include, but are not limited to:

UX Researcher tools

All UX Researchers have individual accounts for the following tools:

SurveyMonkey - All UX Researchers should utilize SurveyMonkey for surveys. Please use the GitLab Theme to style your survey. The GitLab logo should be positioned in the top left hand corner of every page (applied automatically, resize to small).

UsabilityHub - Used to build design evaluations, such as first click tests, preference tests and five second tests. UsabilityHub should not be used for surveys.

MailChimp - Used to send campaigns to subscribers of GitLab First Look.

OptimalWorkshop - Used for card sorts and tree testing. We do not have an ongoing subscription to OptimalWorkshop. We purchase a monthly license as and when required.

Zoom Pro Account - We use Zoom to run usability testing sessions and user interviews.

How we work

Like other departments at GitLab, UX Researchers follow the GitLab Workflow and the Product Development Timeline.

We follow a monthly research cycle and use milestones to schedule our work. A typical monthly research cycle is as follows:

8th of each month (or next business day)

8th to 21st of each month.

22nd of each month.

22nd to the last day of the month.

1st to 7th of each month.

Depending on the scope of each research study, a UX Researcher typically works on 1-3 studies per research cycle. Sometimes, larger studies may need to span more than 1 research cycle.

Scheduling of issues

How we decide what to research

Working on a research study

  1. Update the Research proposal with the following:
    • Label the issue with the area of GitLab you’re testing (for example, navigation), the status of the issue (in progress) and the Product stage (for example, manage).
    • Add a milestone to the issue.
    • Mark the issue as confidential until the research is completed so it doesn’t influence user behavior.
    • Assign the issue to yourself.
    • Add a checklist of actions that you plan to undertake. Next to each item, add an estimated deadline of when you plan to complete each item. This makes it easier for people to understand where the research is up to.
    • Add related issue numbers.
  2. Conduct the research. Ensure you keep the checklist up-to-date.
  3. Schedule a wash-up meeting.
    • Ensure you attach the study's key findings to the calendar invitation at least 24 hours prior to the meeting.
    • Add any supporting evidence (such as user videos) to the Research proposal issue.
  4. Record and facilitate the wash-up meeting.
  5. After the wash-up meeting, undertake any actions you have agreed to take forward, such as creating new issues.
  6. Document the study within a Google document. The report should consist of:
    • Research hypotheses
    • Research methodology
    • Findings
    • Next steps/recommendations as agreed upon in the wash-up meeting.
  7. Update the Research proposal with the following:
    • Link to the report.
    • Unmark the issue within the UX research project as confidential. (In some cases the issue may need to remain confidential if sensitive information is shared. If you’re unsure of whether an issue should remain confidential, please check with Sarah O’Donnell @sarahod).
    • Update the status of the issue to done.
    • Close the issue. You should stay assigned to closed issues so it's obvious who completed the research.

Checklist templates

The following are examples of checklists that you may want to add to a Research proposal.

Usability Testing

User Interviews

Surveys

Design Evaluations

Card Sorts

UX Research label

Both the GitLab CE project and GitLab EE project contain a UX Research label. The purpose of this label is to help UX Designers and Product Managers keep track of issues which they feel may need UX Research support in the future or which are currently undergoing UX Research.

UX Researchers are not responsible for maintaining the UX Research label.

The UX Research label should not be used to request research from UX Researchers. Instead, please follow the process outlined in How to request research.

How to send a study to users of GitLab First Look.

Note: These instructions are for UX Researchers only. If you are not a UX Researcher but would like to send a study to users of GitLab First Look, please raise a research proposal.

  1. In MailChimp, create a new email campaign.
  2. Click Add recipients under the To heading.
  3. Choose GitLab First Look as your list.
  4. Under Segment or Tag use the drop-down menu to select the segment of users you want to contact.
  5. Leave the From field as is (emails are sent from firstlook@gitlab.com. Any emails sent to this address will forward to all UX Researchers).
  6. Click Add Subject. Update the Subject field to: Quick, new research study available! and click Save.
  7. Click Design Email. You are now going to design the content for your email campaign.
    • Click the Saved templates tab.
    • UX Research's templates are: First Look - Survey, First Look - Usability Testing, First Look - User Interviews, First Look - Design Evaluations, First Look - Card Sort and First Look - Beta Testing. Select the template you wish to use and click Next.
    • Review the content on the template, you will need to make some small alterations (such as researcher email address, dates of the study, etc). You can do this by clicking on the block of text you wish to change. This will bring up a WYSIWYG editor on the right-hand side of your screen.
    • You will also need to update the CTA URL to your study URL. You can do this by clicking on the CTA. This will bring up a WYSIWYG editor on the right-hand side of your screen where you can add a URL.
    • You should not make any styling changes to the templates. It's important that the design of emails from GitLab First Look stay consistent and are distinguishable from other email campaigns sent from GitLab.
  8. Once you are ready to test your email campaign. Click Preview and Test in the top right corner of the screen.
    • Enter preview mode to see how your email will be displayed on Desktop and Mobile.
    • Send a test email to yourself. When the email arrives, double check the copy and any URLs.
  9. Once you are happy with the design of your email campaign, click Save & Close.
  10. You are now ready to send your email campaign! Click Send in the top right-hand corner of the screen.
  11. Move the campaign to the UX Research folder.

Building a new segment in MailChimp

Segments are useful for setting up test pools of users, restricting the number of users who receive a study and for matching users to studies which are aligned with their interests.

  1. In MailChimp, navigate to Lists and click GitLab First Look.
  2. Click Manage Contacts and select Segments.
  3. Click Create Segment.
  4. Update any to all so the statement reads: Contacts match all of the following conditions:
  5. In the drop-down menu select any of the questions which appear under Groups. Once you have selected a question, MailChimp will automatically display a list of possible options. For example, if you select the question of: What stages of the DevOps life-cycle interest you?, you will be given a list of options which relate to product stages (Plan, Create, Verify, etc).
  6. Click Add
  7. Repeat as necessary. You can add up to 5 conditions per segment. At the very minimum, you should always add conditions for:

    • User group

      In order to avoid user fatigue, the GitLab First Look mailing list is split into user groups. Each user group consists of 100 users. We cycle through the user groups consecutively. For example: If the latest research study was sent to User Groups: 5, 6 and 7, then the proceeding research study should go to User Group: 8 and onwards.

      To find out which user groups you should send your study to, you need to first check which user groups received the research study prior to your own. You can do this by:

      1. Click into the UX Research folder.
      2. Ensure the list is sorted by: Last Updated.
      3. Click into the campaign at the top of the list.
      4. Click the Details tab of the campaign.
      5. User groups will be listed beside Recipients.

      Currently, there are no hard rules on how many user groups you can send a campaign to. However, you should exercise common sense. For example, there is little sense in a test segment exceeding 100 users (1 user group). Similarly, if you only require 5 users for usability testing, depending on your screening requirements, you may wish to contact 1-2 user groups in order to avoid disappointing users who you are unable to schedule with.

    • Research methodology (What type of studies would you like to take part in?)

    • Product stage (What type of studies would you like to take part in?) or Meltano interest (Would you like to receive studies about Meltano?)

    • We want to personalize the studies we send to users. This means users only receive studies which are relevant to their interests. In turn, this helps GitLab First Look retain users and achieve a greater response rate to studies. Always consider what other conditions might be useful to your campaign. For example, it isn't useful for Core users to receive a study which asks for their feedback on an Ultimate feature.

  8. Click Preview Segment. This will show you how many recipients match your conditions.
  9. Click Save Segment.
  10. Give the newly created segment a descriptive name.

Incentives

Amazon gift cards are country specific. When purchasing a gift card, ensure you use the appropriate Amazon store for a user's preferred country.

Personas

Existing personas are documented within the handbook.

New personas or updates to existing personas can be added at any time.

Personas should be:

Edit this page — open Web IDE — please contribute. Creative Commons License