On-Call

Maintained by:

Expectations for On-Call

If you are on call, then you are expected to be available and ready to respond to PagerDuty pings as soon as possible, but certainly within any response times set by our Service Level Agreements in the case of Customer Emergencies. This may require bringing a laptop and reliable internet connection with you if you have plans outside of your work space while being on call, as an example.
We take on-call seriously. There are escalation policies in place so that if a first responder does not respond fast enough another team member or members is/are alerted. Such policies are essentially expected to never be triggered, but they cover extreme and unforeseeable circumstances.
Because GitLab is an asynchronous workflow company, @mentions of On-Call individuals in slack will be treated like normal messages and no SLA for response will be attached or associated with them.
Provide support to the release managers in the release process.
As noted in the main handbook, after being on-call take time off. Being available for issues and outages will wear you off even if you had no pages, and resting is critical for proper functioning. Just let your team know.
- Specific to team members in Australia review the Time in lieu policy here
For on-call it's the responsibility of the team members to act in compliance with local rules and regulations. If ever in doubt, please reach out to your manager and/or aligned People Business Partner.

Customer Emergency On-Call Rotation

We do 7 days of 8-hour shifts in a follow-the-sun style, based on your location.
After 10 minutes, if the alert has not been acknowledged, support management is alerted. After a further 5 minutes, everyone on the customer on-call rotation is alerted.
Unless advised otherwise by the technical account manager or your manager, assume good intent to give every emergency ticket from customers with priority support the emergency SLA.
After 30 minutes, if the customer has not responded to our initial contact with them, let them know that the emergency ticket will be closed and that you are opening a normal priority ticket on their behalf. Also let them know that they are welcome to open a new emergency ticket if necessary.
You can view the schedule and the escalation policy on PagerDuty. You can also opt to subscribe to your on-call schedule, which is updated daily.
After each shift, if there was an alert / incident, the on call person will send a hand off email to the next on call explaining what happened and what's ongoing, pointing at the right issues with the progress.
If you need to reach the current on-call engineer and they're not accessible on Slack (e.g. it's a weekend, or the end of a shift), you can manually trigger a PagerDuty incident to get their attention, selecting Customer Support as the Impacted Service and assigning it to the relevant Support Engineer.
See the GitLab Support On-Call Guide for a more comprehensive guide to handling customer emergencies.

GitLab.com Reliability On-Call Rotation

Infrastructure Engineer On-Call

The Infrastructure department's Reliability Engineering teams provide 24x7 on-call coverage for the production environment. For details, please see incident-management.

Engineering Incident Manager

Incident manager rotation is staffed by certain team members in the Development and Infrastructure departments.
More information regarding the Incident Manager role, including shift schedules, responsibilities can be found in the Incident Manager on-boarding page.

Development Team On-Call Rotation

This on-call process is designed for GitLab.com operational issues that are escalated by the Infrastructure team.
Development team currently does NOT use PagerDuty for scheduling and paging. On-call schedule is maintained in this schedule sheet.
Issues are escalated in the Slack channel #dev-escalation by the Infrastructure team.
First response SLO is 15 minutes. If no response within the first 5 minutes, the infrastructure team will call the engineer's phone number on the schedule sheet.
Development engineers do 4-hour shifts.
Engineering managers do monthly shifts as scheduling coordinators.
Check out process description and on-call workflow when escalating GitLab.com operational issue(s).
Check out more detail for general information of the escalation process.

Security Team On-Call Rotation

Security Operations (SecOps)

SecOps on-call rotation is 7 days of 24-hour shifts.
After 15 minutes, if the alert has not been acknowledged, the Security Manager on-call is alerted.
You can view the Security Operations schedule on PagerDuty.
When on-call, prioritize work that will make the on-call better (that includes building projects, systems, adding metrics, removing noisy alerts). Much like the Production team, we strive to have nothing to do when being on-call, and to have meaningful alerts and pages. The only way of achieving this is by investing time in trying to automate ourselves out of a job.
The main expectation when on-call is triaging the urgency of a page - if the security of GitLab is at risk, do your best to understand the issue and coordinate an adequate response. If you don't know what to do, engage the Security manager on-call to help you out.
More information is available in the Security Operations On-Call Guide and the Security Incident Response Guide.

Security Managers

Security Manager on-call rotation is 7 days of 12-hour shifts.
Alerts are sent to the Security Manager on-call if the SecOps on-call page isn't answered within 15 minutes.
You can view the Security Manager schedule on PagerDuty.
The Security Manager on-call is responsible to engage alternative/backup SecOps Engineers in the event the primary is unavailable.
In the event of a high-impact security incident to GitLab, the Security Manager on-call will be engaged to assist with cross-team/department coordination.

Quality Team On-Call Rotation

Quality department's on-call do not include work outside GitLab's normal business hours. The process is defined on our pipeline on-call rotation page.
The rotation is on a weekly basis separated out into 2 timezones APAC/EMEA and AMER and each timezone has a primary and secondary on-call DRI.
The primary DRI is responsible for reporting and resolving triage activities while the secondary DRI is involved to prepare for taking over as primary in the following week.
This on-call rotation is to ensure accurate and stable test pipeline results that directly affects our continuous release process.
The list of pipelines which are monitored are defined on our pipeline page.
The schedule and roster is defined on our schedule page.

PagerDuty

We use PagerDuty to set the on-call schedules, and to route notifications to the appropriate individual(s).

Swapping On-Call Duty

Team members covering a shift for someone else are responsible for adding the override in PagerDuty. They can delegate this task back to the requestor, but only after explicitly confirming they will cover the requested shift(s). To set an override, click the "Schedule an Override" button from the side navigation on the Schedule page or after selecting the relevant block of time on the calendar or timeline view. This action defaults the person in the override to you — PagerDuty assumes that you're the person volunteering an override. If you're processing this for another team member, you'll need to select their name from the drop-down list. Also see this article for reference.

Adding and removing people from the roster

When adding a new team member to the on-call roster, it's inevitable that the rotation schedule will shift. The manager adding a new team member will add the individual towards the end of the current rotation to avoid changing the current schedule, if possible. When adding a new team member to the rotation, the manager will raise the topic to their team(s) to make sure everyone has ample time to review the changes.