There are four on-call rotations in Support:
For customers that have Priority Support, the Support Engineering Team is on-call in these capacities and available to assist with emergencies. What constitutes an emergency is defined in our definitions of support impact.
We take on-call seriously. There are escalation policies in place so that if a first responder does not respond fast enough another team member or members is/are alerted. Such policies aren't expected to ever be triggered, but they cover extreme and unforeseeable circumstances.
When you are on call you are expected to be available and ready to respond to PagerDuty pings as soon as possible, and certainly within the emergency response time set by our Service Level Agreements.
If you have plans outside of your work space while being on call, then being available may require bringing a laptop and reliable internet connection with you.
You should not be chained to your desk, but you should be equipped to acknowledge and act on PD alerts in a timely manner.
Be proactive in communicating your availability. Sometimes you can't be immediately available for every minute of your on-call shift. If you expect to be unavailable for a short period of time, send an FYI in Slack.
Involve relevant stakeholders: whether it's the e-group, a CSM, subject matter experts or Support leadership, customer and operational emergencies should be known. See your rotation specific workflow for more detailed notes.
Note: You may sometimes be required to contact GitLab users on behalf of another GitLab team (such as the SIRT team). Please follow the Sending Notices workflow to action these requests.
Rest assured: escalation is okay – other GitLab team members are happy to help. Caring for our customers is a shared responsibility. Tag a Slack support-team Group if you haven't gotten help in your Slack thread. Tag the support managers if you need to escalate further.
If another support engineer joins your emergency call, feel free to assign them a role to divide up the labor.
So and so would you please (take notes, reach out to this product team and ask for help, look up the code for this and see what you can find)?
Make a real effort to de-stress during your on-call shift. After being on-call, consider taking time off, as noted in the main GitLab Handbook. Just being available for emergencies and outages causes stress, even if there are no pages. Resting is critical for proper functioning. Just let your team know.
When you get a notification from PagerDuty give yourself a few minutes to prepare.
When you're in a call, you do not need to provide immediate answers. You're allowed to pause for a few minutes for researching, asking for help, etc. Make sure to communicate – let the other folks on the call know what you're doing. Example: "I need a few minutes to work through the code here and make sense of it".
If you've had a long week with multiple pages from PagerDuty or particularly long calls, consider asking someone to cover a day or some portion of a day so you can get some rest.
By using a regional, follow-the-sun style of on-call Support we try to keep rotations within waking hours. As a result, each region may organize their on-call rotations in a slightly different way, or have different requirements for hours of coverage.
PagerDuty is the single source of truth for on-call hours, rotation order and escalation policies.
NOTE: If a new alert has not been acknowledged after 10 minutes, the Support Manager on-call is alerted. After a further 5 minutes, if there is no acknowledgement, Senior Support Managers are alerted.
There are several ways to view current and future schedules:
#spt-gg-forest slack channel, you can see who is on-call for the different schedules todayChanges to PagerDuty schedules will happen once a quarter with additions and removals handled via the Support PagerDuty Worksheet. Permissions on the Worksheet will be limited to Support Management and the Support Operations team.
The Support Operations team will unlock the Support PagerDuty Worksheet at the start of the second month of the quarter. At this time, they will notify the Support Managers and ask them to review and submit changes as necessary. During the final month of the quarter, the Support Operations team will once again lock the Worksheet and implement the changes to go live at the beginning of the next quarter.
When requesting changes to the Shadow schedules, please create a new Issue using the appropriate Issue Template.
When adding someone to the schedule the manager will need to manually add the email address of the user they want added to the rotation.
When removing a user from a schedule the manager will need to delete the email address of the user they are removing, leaving the cell blank.
For new team members approaching their first on-call shift, your Support onboarding issue includes a section suggesting that you shadow a current on-call to gain familiarity with the process. After completing your shadow shift, work with your manager to get yourself added to the on-call rotation. For your first on-call week we recommend asking your Onboarding Buddy to be available as a secondary to help if an emergency comes in.
Your role is to make sure someone is available to respond to emergencies during the week you are scheduled. Flexibility is possible – you can split work with others, or schedule overrides for a few hours or days. You don't have to change vacation plans, or be at your desk all week! It's OK to take a walk outside, if you have your phone and reception. This way you can acknowledge the page, and locate someone to help (using Slack).
If you prefer to work with a colleague as a secondary, discuss with team members or your manager and find partners who like sharing the role. You can work together during the week, and update PagerDuty as you wish (options include: split days into mornings and evenings, take alternate days, work as a primary and secondary). Your manager can play an active role in helping pair people who want to work like this.
To swap on-call duty with a colleague:
See the PagerDuty documentation for complete steps.
Due to the global on-call week starting on Monday, 09:00 Amsterdam time, on-call shifts for APAC begin on Tuesday and end on Monday. The APAC team has decided to retain this shift configuration as it allows for easier planning to take a long weekend with Monday off and as a buffer to ease into the next on-call. See the latest discussion in this Support Team Meta issue.
APAC spans many timezones. Using an 8-hour shift to cover APAC business hours results in an on-call shift that can finish as late 9pm local time for APAC east based team members, and that starts as early as 04:30 local time for APAC west team members. Consequently the support engineer shifts are split into 2 groups, APAC 1 and APAC 2, allowing each team member to cover hours within their normal working hours.
Note that support manager on call shifts remain the full 8 hours.
Team members employed by GitLab PTY Ltd must take time off in lieu within two
weeks of completing their on-call shift. Time in lieu should be requested via
Time Off by Deel, selecting the On-Call Time in Lieu option.
For more details, see the GitLab PTY Australia Specific Benefits page.
Team members employed by GitLab PTY Ltd NZ must avoid taking an on-call shift which falls on a New Zealand public holiday. If this cannot be avoided, your manager must be informed.
For more details, see the GitLab PTY Ltd NZ Specific Benefits page.
Before your shift starts, always double-check that your alerts are working and your PagerDuty contact is up to date. Send a test page to make sure that you are receiving alerts correctly.
When your on-call shift starts, you should get notification(s) that your shift is starting (email or text, depending on your PagerDuty preferences).
You may want to monitor all on-call pages received by the Support team. You can
do this by joining the #spt_on-call
Slack channel.
To see who the current manager on-call is you can:
/chatops run oncall manager#spt-gg-forest slack channel, the regional bot message lists who is on-call for the different schedules today#spt_managers (where you may or may not be referred to the above steps!)/pd-support-manager command to page the on-call managerWe want to minimize the affect of on-call duty on your life. One way we do that is by offsetting any impact on your personal expenses.
You may expense the cost of your mobile phone service for any month that you are performing on-call duties. This is limited to your service cost itself, not any costs relating to the phone device, to a personal hotspot device or to services for other people on your phone plan.
We understand you may have plans outside of your normal workspace while you're on-call. If as a result you need to use your phone to provide internet service to your computer, then you may include additional data charges in your expense report.
PagerDuty phone and SMS notifications can come from a variety of different phone numbers and as such, it is important to stay up to date with this to avoid missed pages. You should use this PagerDuty documentation page to download the most recent vCard or setup an automatically updated PagerDuty contact on your device.
If you use a "do not disturb" mode on your device, you should also allow the PagerDuty contact to bypass this.