Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Training Module - SaaS Basics

  1. You are here:
  2. Handbook
  3. Support Team Handbook
  4. Support Learning & Training
  5. Training Module - SaaS Basics

GitLab SaaS Basic - Training Module

Introduction

This training module provides Support Engineers with the basics of answering GitLab.com (SaaS) product related tickets, for both new and existing Support team members who are looking to start working on SaaS tickets.

Goals

At the end of this module, you should be able to:

General Timeline and Expectations

Stage 0: Create Your Module

  1. Create an issue using the GitLab-com SaaS Basic template by making the Issue Title: -
  2. Add yourself and your trainer as the assignees.
  3. Set milestones, if applicable, and a due date to help motivate yourself!

Consider using the Time Tracking functionality so that the estimated length for the module can be refined.

Stage 1: Overview and Chatops

-Done with Stage 1

  1. Read the GitLab.com Overview page.
  2. If you have not already received an Auditor account for GitLab.com, create a new GitLab.com account that will serve as your admin account using the username and email address format of username-auditor and username+auditor@gitlab.com, confirm it via email, and then enable 2FA on it. Example access request.
    • If your regular GitLab account has the following details:
      • username:awesome_user7
      • email: awesomeuser@gitlab.com
    • Your auditor account should be:
      • username: awesome_user7-auditor
      • email: awesomeuser+auditor@gitlab.com

Regardless of whether you receive admin access later, you should set up chatops access.

  1. Read the first two sections: the Chatops intro and how it works.
  2. If you don't already have it, get Chatops access.
    • Note: Use the "Google" sign in on the ops instance to create an account if needed.
  3. Read about commonly used chatops commands.
  4. Practice using chatops by finding your own user and one of the support groups. You can direct message chatops by finding 'GitLab Chatops' under Apps in Slack.
    • What's your user ID? ID:
    • What's the group ID for gitlab-com/support? ID:
    • Find a feature flag to search for and get its current status. Take a screenshot and post it in a comment to the issue. Please blur out any user/group-identifiable information or don't include one with scoping.
  5. Take a look at our improving chatops for Support epic. Consider contributing to one of the issues in the future. As more of these functions become available, the more we can move cases out of the admin-only section.

Stage 2: GitLab.com Accounts

Note: Stage 2 and 3 are reversible. For those who want a more "structured" introduction to procedural tickets, start here. If you're already familiar with GitLab and are used to troubleshooting Self-managed, consider doing Stage 3 first.

See also Stage 4 for a place to record pairing sessions.

Keep in mind that only GitLab team members are admins, and that accounts are bound by our Terms of Use as covered in the overview.

Bottom Line: Never share information that a user would not have access to even if it's about another user within their organization. You can still suggest possible solutions based on the description of the problem (and even what you find out about the account), but definitive information cannot be shared.

Triaging

Many common tickets are triaged to other places.

  1. Account Deletion and Data requests: Read only the Overview piece for each workflow so you can see where these should be routed to. There are also macros in ZenDesk for these cases.
    1. Data Access Requests (typically GDPR Article 15)
    2. Account Deletion (typically GDPR Driven)
  2. Subpoenas and other legal (court) requests. See Subpoenas and other requests for information
    • Note: If the customer request is not tied to an active court case or other legal matter, it may fall under one of the other cases below.
  3. Read over the example cases to move to Security.
    1. Copyright (DMCA) Removal Requests has its own email. Instead of redirecting the user to email the dmca account, the ticket can be moved to the Security queue.

Email Receiving

One of the main differences of working with GitLab.com is that we receive many requests from free users because we are the administrators of GitLab.com. One of the most common cases is not receiving confirmation, password, or other system notifications from GitLab.

  1. Read the Confirmation Emails workflow.
  2. Answer at least 1 ticket with an email receiving workflow: (insert ticket link here)
  3. Briefly look over Real Time Blocklist Delisting for when GitLab emails are being marked as spam.

Other Account Issues

  1. Take a look at the list of GitLab.com workflows or read the admin only section (Stage 4) and make note what other common cases we take care of, but require admin access.

Remember: When triaging tickets and you're unsure if a user is asking about a Self-managed instance or GitLab.com, use chatops to check if the user exists. The API searches for primary email and secondary emails starting 13.7 as described in this issue.

Stage 3: GitLab.com Architecture and Troubleshooting

Keep in mind:

  1. As previously noted, GitLab.com users do not have admin accounts. Please do not send any doc links that are self-managed only (check badging at the top of the page or section for CORE/STARTER/PREMIUM/ULTIMATE ONLY) or anything from the administration section.
  2. Troubleshooting problems on SaaS and Self-managed are the same or similar. The main difference is where and how we gather the necessary information.

Architecture

  1. Review materials relevant to GitLab.com Architecture
    1. Read about the GitLab architecture up to and including the Simplified component overview section.
      • Note: You're not expected to remember everything, but to get a general sense of GitLab's architecture. Feel free to read the details on any component based on your interest.
    2. Give the Production architecture page a brief lookover.
      • Note: Mostly, you'll want a broad understanding of the "Current", "Service", and "Network Architecture" sections.
    3. Give the GitLab.com Setting docs page a brief review to know what settings we publicize.
      • Note: Instance level settings are split between admin dashboard, gprd-base settings (equivalent of gitlab.rb), and secrets (which only infra has access to).

Searching logs and filing issues

  1. Learn about finding and filing errors. Note: Use your dev account to sign into Kibana and Sentry.
    1. Using Kibana.
      • Note: Typical usage is rails and sidekiq logs. HAproxy and CloudFlare logs are only available to infra team. Ask in the #production Slack channel if necessary.
    2. Searching Kibana, Sentry, and filing GitLab issues from Sentry.
    3. As an exercise, visit a non-existing page on GitLab.com, such as gitlab.com/gitlab-com/doesntexist. Search Kibana for the 404 error using your username. Add a screenshot of the relevant portion of the error you find in Kibana as a comment to this issue.
    4. Search Sentry using the correlation_id. You may not find anything and the search is not reliable, but take a screenshot of your search and results and add it as a comment to this issue anyway.
    5. Exercise: use the import workflow to search for an import error.
    6. Importing customer projects is the only current case of "getting around" a bug or lack of feature. Read the rest of the import criteria to learn when we do these as a courtesy.
  2. If an issue can only be resolved through rails console access, you can file a "GitLab.com Console Escalation" issue.

Places to reproduce

The best way to troubleshoot as always is to try to reproduce it.

  1. Remember that you have access to 3 groups for each plan tier gitlab-gold, gitlab-silver, and gitlab-bronze.
    • Since these are public, you can also link them in tickets as long as you make the project public as well.
  2. You can check the Auto DevOps project to see what a typical job for various parts of the pipeline may be. It's on a semi-monthly schedule to catch what's changed pre and post release.

Specific cases

Because GitLab is the steward of GitLab.com, we sometimes need to troubleshoot some of our integrations. We have workflows for some cases. You don't need to read these in detail now, but keep in mind that these are available:

  1. Custom domain verification on GitLab.com
  2. Service Desk Troubleshooting

IP blocks and log requests

  1. At times, the infra team blocks users due to alerts of possible abuse. Learn about sending notices.
    • Note: If needed, ping the CMOC to send these out.
  2. Read about identifying the cause of IP blocks. You won't find a block, but try doing a search by IP of your own IP to see what gets logged in the rails log.
  3. When responding to users about possible blocks, you'll need to keep in mind what information you're allowed to provide. See the log and audit requests workflow for more information.

Security issues

Security's focus is instance wide security breaches and vulnerabilities. They do however take care of other account related cases (covered in previous stage).

  1. Commonly, users often file commit associations as a "security" issue. Follow the investigate commits on determining whether it's a security incident.

Stage 4: Keeping up to date and Pairings

Keeping up to date and asking questions:

  1. Check the "Dotcom Support Sync" meeting doc for recent updates/issues (linked from calendar entry and SWIR), and consider attending when in your timezone (3 week rotation). Otherwise, check and add to the doc weekly.
    1. If you are 50%+ focused on .com, please consider this a must and also add yourself to one of the chair/notetaker rotations.
  2. Consider adding yourself to the every 6 weeks cross-region GitLab.com crush sessions (check Support calendar).
  3. Join the #support_gitlab-com Slack channel where you can ask questions about tickets, process, etc.

If you are doing this module as part of onboarding, feel free to remove the pairing section.

  1. Have 5 pairing sessions specifically geared towards answering GitLab.com tickets or issues.

  2. If you are doing Stage 5 (admin access), do 2 more focusing on tickets or issues that require admin access.

Stage 5: GitLab.com admin access (with manager approval)

  1. Typically at week 3-4 at GitLab, you and your manager should check in that you're ready for admin access. If you are, complete GitLab.com Admin access Training Module

Internal Requests

  1. Update notification settings to Watch which will send you notifications for all activity, or Custom for new issues in the following project:
    1. internal requests. Alternatively, given there are a lot of different requests, only subscribe to the following labels:
  2. You will also see Personal Account and Deletions requests, which are auto-assigned to team members with 50%+ SaaS focus. For training purposes, you can ask a teammate to give one to you. Follow the appropriate workflow below to fulfill the request and link the issue that you've done.
    1. Data (GDPR Article 15) Requests
      1. (Optional as these are very rare) Link to issue:
    2. Account Deletion Requests
      1. Link to issue:
    3. Discuss with your manager if you should to be added to the auto-assigned rotation. To be added into the rotation, make sure your SaaS focus is 50%+.
  3. For internal requests, you only need to work on the requests that relate specifically to GitLab.com. Many of the other issues as labelled are specific to other areas (such as Console or Licensing).
    1. Check out the list of templates.
    2. Read the Servicing Internal Requests workflow which covers most internal issues.
      1. (Optional as these are rare) Link to issue:

User Accounts

Aside from the ones covered in Stage 2, the most common user requests have to do with disabling 2FA and name squatting. However, most of these workflows apply in other cases as well.

  1. When making any changes to a user account, make sure to use an admin note.
  2. For all 2FA disabling requests, use the Account Ownership Verification. This workflow may be used for some other cases as well where we want to verify the ownership.
    • Link to an account ticket requiring verification:
  3. Name Squatting Requests apply to groups and users. Included here since it is often for users.
  4. Except in the cases above, or wherever else there is a clear ask of the action (including impersonating the user), always ask for permission to take action.

Other Common Requests

  1. reCaptcha complaints are often due to actions on a specific issue by a specific user being marked as spam. Learn how to mark as ham.
    • Note: The spam logs are not searchable, so requests have be made almost immediately after receiving the recaptcha prompt.
  2. Reinstating blocked accounts should also be confirmed with the abuse team first. In cases of conduct violations, you can ask in the issue to move the ticket to the Security queue for them to respond directly.
  3. Personal Identifiable Information Removal Requests are uncommon, but read the workflow to understand Support's responsibility.

Stage 6: GitLab.com console (with manager approval)

Depending on your prior experience and readiness, open the module listed below. Your manager may decide to have you pick this up later once you have more familiarity with GitLab.

  1. Open a module on GitLab.com Console.

Please also submit MRs for any improvements that you can think of! The file is located in an issue template in the 'support-training` repository.

Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license
View page source — Edit in Web IDE — please contribute. Creative Commons License