GitLab Support On-Call Guide

Maintained by:

On-call in GitLab Support

There are four on-call rotations in Support:

CMOC - in which Support Engineers drive public communication in the incident management process during GitLab.com operational emergencies.
Customer Emergencies- in which Support Engineers respond to and coordinate the resolution of emergency reports from Self-managed and GitLab.com customers
US Federal Emergencies - which is similar to Customer Emergencies, but has different hours and a US Citizen requirement to participate in
Support Manager On-call - in which Support Manager are an escalation point for issues raised by Support Engineers and GitLab team memebers and missed PagerDuty notifications

For customers that have Priority Support, the Support Engineering Team is on-call in these capacities and available to assist with emergencies. What constitutes an emergency is defined in our definitions of support impact.

We take on-call seriously. There are escalation policies in place so that if a first responder does not respond fast enough another team member or members is/are alerted. Such policies aren't expected to ever be triggered, but they cover extreme and unforeseeable circumstances.

Expectations for Support on-call

Be alert and available

When you are on call you are expected to be available and ready to respond to PagerDuty pings as soon as possible, and certainly within the emergency response time set by our Service Level Agreements.

If you have plans outside of your work space while being on call, then being available may require bringing a laptop and reliable internet connection with you.

You should not be chained to your desk, but you should be equipped to acknowledge and act on PD alerts in a timely manner.

Be proactive in communicating your availability. Sometimes you can't be immediately available for every minute of your on-call shift. If you expect to be unavailable for a short period of time, send an FYI in Slack.

Communicate

Involve relevant stakeholders: whether it's the e-group, a TAM, subject matter experts or Support leadership, customer and operational emergencies should be known. See your rotation specific workflow for more detailed notes.

Ask for help when needed

Rest assured: escalation is okay – other GitLab team members are happy to help. Caring for our customers is a shared responsibility. Tag a Slack support-team Group if you haven't gotten help in your Slack thread. Tag the support managers if you need to escalate further.

If another support engineer joins your emergency call, feel free to assign them a role to divide up the labor.

So and so would you please (take notes, reach out to this product team and ask for help, look up the code for this and see what you can find)?

Take care of yourself

Make a real effort to de-stress during your on-call shift. After being on-call, consider taking time off, as noted in the main GitLab Handbook. Just being available for emergencies and outages causes stress, even if there are no pages. Resting is critical for proper functioning. Just let your team know.

When you're in a call, you do not need to provide immediate answers. You're allowed to pause for a few minutes for researching, asking for help, etc. Make sure to communicate – let the the other folks on the call know what you're doing. Example: "I need a few minutes to work through the code here and make sense of it".

How Support On-call works

Schedule and escalation policy

By using a regional, follow-the-sun style of on-call Support we try to keep rotations within waking hours. As a result, each region may organize their on-call rotations in a slightly different way, or have different requirements for hours of coverage.

PagerDuty is the single source of truth for on-call hours, rotation order and escalation policies.

NOTE: If a new alert has not been acknowledged after 10 minutes, the Support Manager on-call is alerted. After a further 5 minutes, if there is no acknowledgement, Senior Support Managers are alerted.

There are several ways to view current and future schedules:

On gitlab.io: View the current and next week's assignments
In PagerDuty: View the Customer Emergencies schedule and the escalation policy
In Google Calendar, subscribe to your on-call schedule

Your first on-call shift

For new team members approaching their first on-call shift, your Support onboarding issue includes a section suggesting that you shadow a current on-call to gain familiarity with the process. After completing your shadow shift, work with your manager to get yourself added to the on-call rotation. For your first on-call week we recommend asking your Onboarding Buddy to be available as a secondary to help if an emergency comes in.

Make the schedule work for you

Your role is to make sure someone is available to respond to emergencies during the week you are scheduled. Flexibility is possible – you can split work with others, or schedule overrides for a few hours or days. You don't have to change vacation plans, or be at your desk all week! It's OK to take a walk outside, if you have your phone and reception. This way you can acknowledge the page, and locate someone to help (using Slack).

If you prefer to work with a colleague as a secondary, discuss with team members or your manager and find partners who like sharing the role. You can work together during the week, and update PagerDuty as you wish (options include: split days into mornings and evenings, take alternate days, work as a primary and secondary). Your manager can play an active role in helping pair people who want to work like this.

TIP: In Google Calendar, add Busy entries for the days/items you are on-call. Because your Google Calendar is linked to Calendly for Customer Calls, Busy entries ensure you will not receive round-robin calls during your on-call shift.

Swapping on-call duty

To swap on-call duty with a colleague:

Check with your team, see who has availability, and get confirmation from the person willing to cover you. Generally, it's best to ask for help as far in advance as possible.
If you're unable to find coverage, alert your manager.
To swap an on-call shift, or temporarily replace someone, create an override on the main schedule in PagerDuty.
1. Open the correct schedule in PagerDuty.
2. Click on the relevant block of time in PagerDuty.
3. Select "override" and fill in the name of the person you are swapping with.

See the PagerDuty documentation for complete steps.

Considerations in APAC

Shifts start on Tuesday and end on Monday

Due to the global on-call week starting on Monday, 09:00 Amsterdam time, on-call shifts for APAC begin on Tuesday and end on Monday. The APAC team has decided to retain this shift configuration as it allows for easier planning to take a long weekend with Monday off and as a buffer to ease into the next on-call. See the latest discussion in this Support Team Meta issue.

Support Engineer shifts are 4-hour long shifts

APAC spans many timezones. Using an 8-hour shift to cover APAC business hours results in an on-call shift that can finish as late 9pm local time for APAC east based team members, and that starts as early as 04:30 local time for APAC west team members. Consequently the support engineer shifts are split into 2 groups, APAC 1 and APAC 2, allowing each team member to cover hours within their normal working hours.

Note that support manager on call shifts remain the full 8 hours.

Australia (Employees of GitLab PTY Ltd)

Team members employed by GitLab PTY Ltd must take time off in lieu within two weeks of completing their on-call shift. Time in lieu should be requested via PTO by Roots, selecting the On-Call Time in Lieu option.

For more details, see the GitLab PTY Australia Specific Benefits page.

New Zealand (Employees of GitLab PTY Ltd NZ)

Team members employed by GitLab PTY Ltd NZ must avoid taking an on-call shift which falls on a New Zealand public holiday. If this cannot be avoided, your manager must be informed.

For more details, see the GitLab PTY Ltd NZ Specific Benefits page.

Starting on-call

Before your shift starts, always double-check that your alerts are working. Send a test page to make sure that you are receiving alerts correctly.

When your on-call shift starts, you should get notification(s) that your shift is starting (email or text, depending on your PagerDuty preferences).

Seeing which manager is on-call

To see who the current manager on-call is you can:

use ChatOps to query the on-call schedule with /chatops run oncall manager
log in to PagerDuty view the Support Managers Service
View the individual on-call schedules for
Ask in #spt_managers (where you may or may not be referred to the above steps!)

Paging the on-call manager

First, try pinging the on-call manager by name in Slack.
If it's important and urgent, you can:
- Use the /pd-support-manager command to page the on-call manager
- Manually create a new incident from the Support Managers Service

Mobile phone service and data reimbursement

We want to minimize the affect of on-call duty on your life. One way we do that is by offsetting any impact on your personal expenses.

You may expense the cost of your mobile phone service for the month when you begin your on-call rotation. This is limited to your service cost itself, not any costs relating to the phone device, to a personal hotspot device or to services for other people on your phone plan.

We understand you may have plans outside of your normal workspace while you're on-call. If as a result you need to use your phone to provide internet service to your computer, then you may include additional data charges in your expense report.