If you're observing issues on GitLab.com—or on a team that works with
customers or users who are observing issues—a member of the chatops project
can use the command /chatops run oncall prod in the #production Slack channel or in a direct message (DM) with the ChatOps app.
If you're not a member of the chatops project, you can ask request access.
The GitLab ChatOps bot will return the names of the
Engineer On Call (EOC) and the Incident Manager On Call (IMOC). Please @ mention
the engineer in Slack and reference the GitLab issue that contains details of
the issue, if one exists.
Please keep in mind that communication through Slack is asynchronous, so you aren't guaranteed an immediate response.
PagerDuty
If you need an immediate response from the engineer on-call (EOC) type /pd
trigger in the #production Slack channel, and choose the "gitlab-production"
service. Include a brief title. This should only be used for production emergencies.
We use PagerDuty to set the on-call
schedules, and to route notifications to the appropriate individual(s). There
are escalation policies in place for Production issues (i.e. GitLab.com
downtime), Security concerns, and Customer emergencies.
Expectations for On-Call
If you are on call, then you are expected to be available and ready to respond to PagerDuty pings as soon as possible, but certainly within any response times set by our Service Level Agreements in the case of Customer Emergencies. This may require bringing a laptop and reliable internet connection with you if you have plans outside of your work space while being on call, as an example.
We take on-call seriously. There are escalation policies in place so that if a first responder does not respond fast enough another team member or members is/are alerted. Such policies are essentially expected to never be triggered, but they cover extreme and unforeseeable circumstances.
Triage the infrastructure issue tracker, applying appropriate labels and reaching out others when needed, so the rest of the team can focus on the WoW.
Act as an umbrella to avoid team randomization.
Provide support to the release managers in the release process.
Automate as much as possible to get rid of toil, create new alerts and tools to enhance the on-call experience.
As noted in the main handbook, after being on-call take time off. Being available for issues and outages will wear you off even if you had no pages, and resting is critical for proper functioning. Just let your team know.
To swap an on-call shift or temporarily replace someone input this as an override on the main schedule in PagerDuty.
This is done by clicking on the relevant block of time in PagerDuty, selecting "override" and
filling in the name of the person you swapped with. Also see this article for reference.
Changing the rotation of the current schedule
When adding new team member it's inevitable that the rotation schedule will shift. The manager adding a new team member will add the individual towards the end of the current rotation to avoid changing the current schedule, however as there might be a need to add an individual to the top of the rotation a change in the current rotation might be unavoidable. When adding a new team member to the rotation, the manager will raise the topic on the weekly call to make sure everyone has ample time to make changes.
Customer Emergency On-Call Rotation
We do 7 days of 8-hour shifts in a follow-the-sun style, based on your location.
After 10 minutes, if the alert has not been acknowledged, everyone on the customer on-call rotation is alerted. After a further 5 minutes, management is alerted.
Unless advised otherwise by the technical account manager or your manager, assume good intent
to give every emergency ticket from customers with priority support the emergency SLA.
After 45 minutes, if the customer has not responded to our initial contact with them, let them know that the emergency ticket will be closed and that you are opening a normal priority ticket on their behalf. Also let them know that they are welcome to open a new emergency ticket if necessary.
After each shift, if there was an alert / incident, the on call person will send a hand off email to the next on call explaining what happened and what's ongoing, pointing at the right issues with the progress.
If you need to reach the current on-call engineer and they're not accessible on Slack (e.g. it's a weekend, or the end of a shift), you can manually trigger a PagerDuty incident to get their attention, selecting Customer Support as the Impacted Service and assigning it to the relevant Support Engineer.
The Infrastructure department's Reliability Engineering teams provide 24x7 on-call coverage for the production environment. For details, please see incident-management.
Site Reliability Engineers (SREs)
We do 7 days of 8-hour shifts in a follow-the-sun style, based on your location.
After 15 minutes, if the alert has not been acknowledged, the Incident Manager On-call (IMOC) will be notified.
For each shift we need at least an extra SRE available, to support and collaborate in case of complex incidents, mainly for severities 1 and 2.
After each shift, the on-call person will drive the oncall handover.
The main expectation when being on-call is to triage the page, and to determine the urgency and severity of the incident. We aim to resolve incidents using well documented runbooks.
If you need the IMOC or there's any doubt about their availability then you should immediately add them as responders on the PagerDuty incident. You can do so by running a dedicated Response Play. As shown above, response Plays can be run directly from the PagerDuty web UI. For added convenience, we've upgraded to the Slack V2 integration which allows running the Response Plays in Slack.
See this YouTube video for a recorded walkthrough.
When on-call, a secondary focus is managing the SRE On-call board. The board is prioritized to track on-going incidents first, and then to include issues focused on improving on-call rotations.
Database Support
For database-related issues we have support from OnGres, a consultancy that specializes in Postgresql databases. Only a responding EOC or IMOC should ever page OnGres.
For production issues, the SLA of OnGres first response, is categorized as follows:
Severity 1 and 2 - Critical issues. The system is down or not responding. SLA: 15 minutes.
Severity 3 - Non-critical issues. Significant system degradation or limited capabilities that impact on the production system but it still works on a degraded mode. SLA: 1 hour.
Severity 4 - Low impact issues. Any system malfunctioning that does not impede normal production operation but may limit non-significantly the performance or capabilities of the system. SLA: 4 hours.
Alerts are not triggered via automation. All escalations to OnGres are initiated by an incident responder using the "Request OnGres Support" PagerDuty Response Play (see "Response Plays for IMOC or OnGres" in the image above).
Our Staff DBRE is alerted whenever OnGres support is requested. They are available to provide expert-level database knowledge to the SRE who escalated the issue, if OnGres is unable to repair the database to normal operations.
Over time, most scenarios that required consulting OnGres should be addressed in a runbook that the SRE on-call can execute confidently, alone. Or, preferably, by evolving our tools to handle issues with automation.
Managers
We do 4 days of 12-hour shifts during weekdays, and a ~72-hour shift over the weekend.
Managers are responsible for coordinating communications across all parties actively working on the resolution of an ongoing incident, including both infrastructure and support engineers.
Status updates for ongoing incidents–both internally to the company and external to customers–are the responsibility of the manager on-call.
How to page current production on-call
From Slack you can page by using the /pd trigger slash command. This will trigger high urgency notification rules and it will escalate as needed.
Security Team On-Call Rotation
Security Operations (SecOps)
SecOps on-call rotation is 7 days of 24-hour shifts.
After 15 minutes, if the alert has not been acknowledged, the Security Manager on-call is alerted.
When on-call, prioritize work that will make the on-call better (that includes building projects, systems, adding metrics, removing noisy alerts). Much like the Production team, we strive to have nothing to do when being on-call, and to have meaningful alerts and pages. The only way of achieving this is by investing time in trying to automate ourselves out of a job.
The main expectation when on-call is triaging the urgency of a page - if the security of GitLab is at risk, do your best to understand the issue and coordinate an adequate response. If you don't know what to do, engage the Security manager on-call to help you out.
The Security Manager on-call is responsible to engage alternative/backup SecOps Engineers in the event the primary is unavailable.
In the event of a high-impact security incident to GitLab, the Security Manager on-call will be engaged to assist with cross-team/department coordination.
Development Team On-Call Rotation
This on-call process is designed for GitLab.com operational issues that are escalated by the Infrastructure team.
Development team currently does NOT use PagerDuty for scheduling and paging. On-call schedule is maintained in this schedule sheet.
Issues are escalated in the Slack channel #dev-escalation by the Infrastructure team.
First response SLO is 15 minutes. If no response within the first 5 minutes, the infrastructure team will call the engineer's phone number on the schedule sheet.
Development engineers do 4-hour shifts.
Engineering managers do monthly shifts as scheduling coordinators.
In principle, it is straightforward to add or remove people from the on-call schedules, through the same "schedule editing" links provided above for setting overrides. However, do not change the timezone setting (located in the upper left corner of the image below) unless you absolutely most certainly intend to. As indicated in the image below, when editing a schedule (adding, removing, changing time blocks, etc.), make sure you keep the timezone setting in the upper left corner constant. If you change the timezone setting, PagerDuty will not move the time 'blocks' for on-call duty, but instead it will assume that you meant to keep the selected time blocks (e.g. "11am to 7pm") in the new timezone. As a result, your new schedule may become disjointed from the old ones (old = the schedule as set before the "change on this date" selection), and gaps may appear in the schedule.
As shown above, response Plays can be run directly from the PagerDuty web UI. For added convenience, we've upgraded to the Slack V2 integration which allows running the Response Plays in Slack. 
See this YouTube video for a recorded walkthrough.
