Incident Management

1. You are here:
2. Engineering
3. Infrastructure
4. Incident Management

Maintained by:

On this page

• Incident Management
  • Ownership
  • Roles and Responsibilities
    • Engineer on Call (EOC) Responsibilities
    • Guidelines on Security Incidents
    • Incident Manager on Call (IMOC) Responsibilities
    • Communications Manager on Call (CMOC) Responsibilities
      • How to engage the CMOC? (only during weekdays)
  • Corrective Actions
    • Best practices and examples, when creating a Corrective Action issue:
  • Runbooks
  • Who is the Current EOC?
  • When to Contact the Current EOC
  • Reporting an Incident
    • Report an Incident via Slack
    • Report an Incident via Email
  • Definition of Outage vs Degraded vs Disruption
    • High Severity Bugs
  • Security Incidents
• Communication
  • Status
• Severities
  • Incident Severity
  • Alert Severities
• Incident Workflow
  • Purpose
  • Assignees
  • Labeling
    • Workflow Labeling
    • Root Cause Labeling
    • Required Labeling
    • Optional Labeling
  • Workflow Diagram
  • Incident Board
• Near Misses
  • Background
  • Handling Near Misses

If you're a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you're a GitLab team member looking for who is currently the Engineer On Call (EOC), please see the Who is the Current EOC? section.

If you're a GitLab team member looking for the status of a recent incident, please see the incident board. For detailed information about incident status changes, please see the Incident Workflow section.

Incident Management

Incidents are anomalous conditions that result in—or may lead to—service degradation or outages. These events require human intervention to avert disruptions or restore service to operational status. Incidents are always given immediate attention.

The goal of incident management is to organize chaos into swift incident resolution. To that end, incident management provides:

1. well-defined roles and responsibilities and workflow for members of the incident team,
2. control points to manage the flow information and the resolution path,
3. a root-cause analysis (RCA),
4. and a post-incident review that assigns a severity classification after assessing the impact and scope of the incident.

When an incident starts: we use the #incident-management slack channel for chat based communication. There is a Situation Room Zoom link in the channel description for incident team members to join for synchronous communication. There will be a link to an incident issue in the #incident-management channel. We prefer to keep collaborative work towards incident mitigation in a thread based off of the original incident issue announcement. This makes it easier for incoming oncall for EOC and CMOC to look for status on handoffs.

Ownership

There is only ever one owner of an incident—and only the owner of the incident can declare an incident resolved. At anytime the incident owner can engage the next role in the hierarchy for support. With the exception of when GitLab.com is not functioning correctly, the incident issue should be assigned to the current owner.

Roles and Responsibilities

It's important to clearly delineate responsibilities during an incident. Quick resolution requires focus and a clear hierarchy for delegation of tasks. Preventing overlaps and ensuring a proper order of operations is vital to mitigation. The responsibilities outlined in the roles below are cascading–and ownership of the incident passes from one role to the next as those roles are engaged. Until the next role in the hierarchy engages, the previous role assumes all of the subsequent roles' responsibilities and retains ownership of the incident.

These definitions imply several on-call rotations for the different roles.

Engineer on Call (EOC) Responsibilities

1. As EOC, your highest priority for the duration of your shift is the stability of GitLab.com.
2. The Single Source of Truth (SSOT) for who is EOC is the SRE Schedule in Pagerduty.
3. Alerts that are routed to Pagerduty need to acknowledged within 15 minutes, otherwise they will be escalated to the oncall IMOC.
   1. Alert-manager alerts in #alerts and #alerts-general are an important source of information about the health of the environment and should be monitored during working hours.
   2. If the Pagerduty alert noise is too high, your task as an EOC is clearing out that noise by either fixing the system or changing the alert.
   3. If you are changing the alert, it is your responsibility to explain the reasons behind it and inform the next EOC that the change occurred.
   4. Each event (may be multiple related pages) should result in an issue in the production tracker. See production queue usage for more details.
4. If sources outside of our alerting are reporting a problem, and you have not received any alerts, it is still your responsibility to investigate. Declare a low severity incident and investigate from there.
   1. Low severity (S3/S4) incidents (and issues) are cheap, and will allow others a means to communicate their experience if they are also experiencing the issue.
   2. "No alerts" is not the same as "no problem"
5. GitLab.com is a complex system. It is ok to not fully understand the underlying issue or its causes. However, if this is the case, as EOC you should engage with the IMOC to find a team member with the appropriate expertise.
   1. Requesting assistance does not mean relinquishing EOC responsibility. The EOC is still responsible for the incident.
   2. The GitLab Organizational Chart and the GitLab Team Page, which lists areas of expertise for team members, are important tools for finding the right people.
6. As soon as an S1/S2 incident is declared, join the The Situation Room Permanent Zoom. The Zoom link is in the #incident-management topic.
   1. GitLab works in an asynchronous manner, but incidents require a synchronous response. Our collective goal is high availability of 99.95% and beyond, which means that the timescales over which communication needs to occur during an incident is measured in seconds and minutes, not hours.
7. Keep in mind that a GitLab.com incident is not an "infrastructure problem". It is a company-wide issue, and as EOC, you are leading the response on behalf of the company.
   1. If you need information or assistance, engage with Engineering teams. If you do not get the response you require within a reasonable period, escalate through the IMOC.
   2. As EOC, require that those who may be able to assist to join the Zoom call and ask them to post their findings in the incident issue or active incident Google doc. Debugging information in Slack will be lost and this should be strongly discouraged.
8. By acknowledging an incident in Pagerduty, the EOC is implying that they are working on it. To further reinforce this acknowledgement, post a note in Slack that you are joining the The Situation Room Permanent Zoom as soon as possible.
   1. If the EOC believes the alert is incorrect, comment on the thread in #production. If the alert is flappy, create an issue and post a link in the thread. This issue might end up being a part of RCA or end up requiring a change in the alert rule.
9. Be inquisitive. Be vigilant. If you notice that something doesn't seem right, investigate further.
10. After the incident is resolved, the EOC should review the comments and ensure that the corrective actions are added to the issue description, regardless of the incident severity. If it is an S1/S2 incident, the EOC should start on performing an incident review, presenting the incident review in the synchronous review meeting if the incident is an S1 or a review has been requested via the ~review-requested label. It is expected that the incident review is completed within 14 days of the incident.

Guidelines on Security Incidents

At times, we have a security incident where we may need to take actions to block a certain URL path or part of the application. This list is meant to help the Security Engineer On-Call and EOC decide when to engage help and post to status.io.

If any of the following are true, it would be best to engage an Incident Manager:

1. There is a S1/P1 report or security incident.
2. An entire path or part of functionality of the GitLab.com application must be blocked.
3. Any unauthorized access to a GitLab.com production system

In some cases, we may choose not to post to status.io, the following are examples where we may skip a post/tweet. In some cases, this helps protect the security of self managed instances until we have released the security update.

1. If a partial block of a URL is possible, for example to exclude problematic strings in a path.
2. If there is no usage of the URL in the last week based on searches in our logs for GitLab.com.

Incident Manager on Call (IMOC) Responsibilities

1. The SSOT for who is IMOC is the SRE Managers Schedule in Pagerduty
2. The IMOC should monitor ongoing incidents and engage with the incident if it escalates to a user-impacting (S1 or S2) incident.
3. The IMOC should engage if requested by the EOC.
4. The time that the IMOC engages should be recorded in the incident issue by the IMOC co-assigning themselves to the issue. If GitLab.com is down, record the time in the Google working doc.
5. For non-critical issues, or critical (S1, S2) issues with a short duration, the IMOC may also take on the role of CMOC.
   • Due to limited people on the IMOC rotation, there may be times of the day when the CMOC (if available; see How to engage the CMOC) is a more friendly choice.
6. The IMOC should ensure that the appropriate team members from other teams engage within an appropriate amount of time.
   1. During a user-impacting incident, the appropriate time response time is minutes.
   2. If the IMOC is unable to obtain a response through Slack channels, they should escalate to a manager or director to obtain assistance.
7. They evaluate information provided by team members, lend technical direction, and coordinate troubleshooting efforts.
8. If applicable, coordinate the incident response with business contingency activities.
9. After the incident is resolved, the IMOC is responsible for conducting the post-incident review.
10. In the case of high severity bugs that affect a customer, the IMOC will also be responsible for making sure Incident Reviews are coordinated with Engineering and go through the proper Incident Review process.
11. In the event of a Severity 1 incident which has been running for an hour or more or appears that it will be a long-running Severity 1 incident, page Infrastructure leadership via email at severity-1@gitlab.pagerduty.com or via the GitLab Production - Severity 1 Escalation service in PagerDuty (app or website) with a link to the incident.

Communications Manager on Call (CMOC) Responsibilities

For serious incidents that require coordinated communications across multiple channels, the IMOC will select a CMOC for the duration of the incident during the incident declaration process.

The GitLab support team staffs an oncall rotation and via the Incident Management - CMOC service in PagerDuty. They have a section in the support handbook for getting new CMOC people up to speed.

During an incident, the CMOC will:

1. Be the voice of GitLab during an incident by updating our end-users, internal parties, and executive-level managers through updates to our status page hosted by Status.io.
2. Update the status page at regular intervals in accordance with the severity of the incident.
3. Notify GitLab Community Team via Slack using the @community-team handle at the start of an incident.

How to engage the CMOC? (only during weekdays)

If, during an incident, EOC or IMOC decide to engage CMOC, they should do that by paging the on-call person:

• Using the /pd-cmoc command in Slack or
• Directly from PagerDuty in the Incident Management - CMOC Rotation schedule in PagerDuty. That can be done by navigating to Incidents page in PagerDuty, and then creating the new incident while picking Incident Management - CMOC as Impacted Service.

Important to clarify that the CMOC covering hours does not include the weekends. 24x7 coverage for CMOC is being worked in support-team-meta#2822.

Corrective Actions

Corrective Actions (CAs) are work items that we create as a result of an incident. They are designed to prevent or reduce the likelihood and/or impact of an incident recurrence.

Best practices and examples, when creating a Corrective Action issue:

• They should be SMART: Specific, Measurable, Achievable, Relevant and Time-bounded.
• Avoid creating CAs that:
  • Are too generic (most typical mistake, as oposed to Specific)
  • Only fix incident symptoms.
  • Introduce more human error.
  • will not help to keep the incident from happening again.
• Examples: (taken from several best-practices Postmortem pages)

Runbooks

Runbooks are available for engineers on call. The project README contains links to checklists for each of the above roles.

In the event of a GitLab.com outage, a mirror of the runbooks repository is available on at https://ops.gitlab.net/gitlab-com/runbooks.

Who is the Current EOC?

The chatops bot will give you this information if you DM it with /chatops run oncall prod.

When to Contact the Current EOC

The current EOC can be contacted via the @sre-oncall handle in Slack, but please only use this handle in the following scenarios.

1. You need assistance in halting the deployment pipeline. note: this can also be accomplished by Reporting an Incident and labeling it as ~severity::2.
2. You are conducting a production change via our Change Management process and as a required step need to seek the approval of the EOC.
3. For all other concerns please see the Getting Assistance section.

The EOC will respond as soon as they can to the usage of the @sre-oncall handle in Slack, but depending on circumstances, may not be immediately available. If it is an emergency and you need an immediate response, please see the Reporting an Incident section.

Reporting an Incident

If you are a GitLab team member and would like to report a possible incident related to GitLab.com and have the EOC paged in to respond, choose one of the reporting methods below. Regardless of the method chose, please stay online until the EOC has had a chance to come online and engage with you regarding the incident. Thanks for your help!

Report an Incident via Slack

Type /incident declare in the #production channel in GitLab's Slack and follow the prompts. This will open an incident issue. If you suspect the issue is an emergency, tick the "Page the engineer on-call" box - not the incident manager or communications manager boxes. You do not need to decide if the problem is an incident, and should err on the side of paging the engineer on-call if you are not sure. We have triage steps below to make sure we respond appropriately. Reporting high severity bugs via this process is the preferred path so that we can make sure we engage the appropriate engineering teams as needed.

Incident Declaration Slack window

Incident Declaration Results

As well as opening a GitLab incident issue, a dedicated incident Slack channel will be opened. The "woodhouse" bot will post links to all of these resources in the main #incident-management channel. Please note that unless you're an SRE, you won't be able to post in #incident-management directly. Please join the dedicated Slack channel, created and linked as a result of the incident declaration, to discuss the incident with the on-call engineer.

Report an Incident via Email

Email gitlab-production-eoc@gitlab.pagerduty.com. This will immediately page the Engineer On Call.

Definition of Outage vs Degraded vs Disruption

This is a first revision of the definition of Service Disruption (Outage), Partial Service Disruption, and Degraded Performance per the terms on Status.io. Data is based on the graphs from the Key Service Metrics Dashboard

Outage and Degraded Performance incidents occur when:

1. Degraded as any sustained 5 minute time period where a service is below its documented Apdex SLO or above its documented error ratio SLO.
2. Outage (Status = Disruption) as a 5 minute sustained error rate above the Outage line on the error ratio graph

SLOs are documented in the runbooks/rules

To check if we are Degraded or Disrupted for GitLab.com, we look at these graphs:

1. Web Service
   • Error Ratio
   • Apdex
2. API Service
   • Error Ratio
   • Apdex
3. Git service(public facing git interactions)
   • Error Ratio
   • Apdex
4. GitLab Pages service
   • Error Ratio
   • Apdex
5. Registry service
   • Error Ratio
   • Apdex
6. Sidekiq
   • Error Ratio
   • Apdex

A Partial Service Disruption is when only part of the GitLab.com services or infrastructure is experiencing an incident. Examples of partial service disruptions are instances where GitLab.com is operating normally except there are:

1. delayed CI/CD pending jobs
2. delayed repository mirroring
3. high severity bugs affecting a particular feature like Merge Requests
4. Abuse or degradation on 1 gitaly node affecting a subset of git repos. This would be visible on the Gitaly service metrics

High Severity Bugs

In the case of high severity bugs, we prefer that an incident issue is still created via Reporting an Incident. This will give us an incident issue on which to track the events and response.

In the case of a high severity bug that is in an ongoing, or upcoming deployment please follow the steps to Block a Deployment.

Security Incidents

If an incident may be security related, engage the Security Operations on-call by using /security in Slack. More detail can be found in Engaging the Security On-Call.

Communication

Information is an asset to everyone impacted by an incident. Properly managing the flow of information is critical to minimizing surprise and setting expectations. We aim to keep interested stakeholders apprised of developments in a timely fashion so they can plan appropriately.

This flow is determined by:

1. the type of information,
2. its intended audience,
3. and timing sensitivity.

Furthermore, avoiding information overload is necessary to keep every stakeholder’s focus.

To that end, we will have: 

1. a dedicated Zoom call for all incidents. A link to the Zoom call can be found in the topic for the #incident-management room in Slack.
2. a Google Doc as needed for multiple user input based on the shared template
3. a dedicated #incident-management channel for internal updates
4. regular updates to status.gitlab.com via status.io that disseminates to various media (e.g. Twitter)
5. a dedicated repo for issues related to Production separate from the queue that holds Infrastructure's workload: namely, issues for incidents and changes.

Status

We manage incident communication using status.io, which updates status.gitlab.com. Incidents in status.io have state and status and are updated by the incident owner.

Definitions and rules for transitioning state and status are as follows.

Status can be set independent of state. The only time these must align is when an issues is

Severities

Incident Severity

Incident severities encapsulate the impact of an incident and scope the resources allocated to handle it. Detailed definitions are provided for each severity, and these definitions are reevaluated as new circumstances become known. Incident management uses our standardized severity definitions, which can be found under availability severities.

Alert Severities

1. Alerts severities do not necessarily determine incident severities. A single incident can trigger a number of alerts at various severities, but the determination of the incident's severity is driven by the above definitions.
2. Over time, we aim to automate the determination of an incident's severity through service-level monitoring that can aggregate individual alerts against specific SLOs.

Incident Workflow

Purpose

In order to effectively track specific metrics and have a single pane of glass for incidents and their reviews, specific labels are used. The below workflow diagram describes the two paths an incident can take from open to closed. The two paths are dictated by the severity of the issue, S1 and S2 incidents require a review. In certain cases an S3 or S4 incident can take the review workflow path. Details here

Assignees

The EOC and the IMOC, at the time of the incident, are the default assignees for an incident issue. They are the assignees for the entire workflow of the incident issue.

Labeling

The following labels are used to track the incident lifecycle from active incident to completed incident review. Label Source

Workflow Labeling

In order to help with attribution, we also label each incident with a scoped label for the Infrastructure Service (Service::) and Group (group::) scoped labels.

Root Cause Labeling

Labeling incidents with similar causes helps develop insight into overall trends and when combined with Service attribution, improved understanding of Service behavior. Indicating a single root cause is desirable and in cases where there appear to be multiple root causes, indicate the root cause which precipitated the incident.

The EOC, as DRI of the incident, is responsible for determining root cause.

The current Root Cause labels are listed below. In order to support trend awareness these labels are meant to be high-level, not too numerous, and as consistent as possible over time.

Required Labeling

These labels are always required on incident issues.

Optional Labeling

In certain cases, additional labels will be added as a mechanism to add metadata to an incident issue for the purposes of metrics and tracking.

Workflow Diagram

Incident Board

The board which tracks all GitLab.com incidents from active to reviewed is located here.

Near Misses

A near miss, "near hit", or "close call" is an unplanned event that has the potential to cause, but does not actually result in an incident.

Background

In the United States, the Aviation Safety Reporting System has been collecting reports of close calls since 1976. Due to near miss observations and other technological improvements, the rate of fatal accidents has dropped about 65 percent. source

As John Allspaw states:

Near misses are like a vaccine. They help the company better defend against more serious errors in the future, without harming anyone or anything in the process.

Handling Near Misses

When a near miss occurs, we should treat it in a similar manner to a normal incident.

1. Open an incident issue, if one is not already opened. Label it with the severity label appropriate to the incident it would have caused, had the incident actually occurred. Label the incident issue with the ~Near Miss label.
2. corrective actions should be treated in the same way as those for an actual incident.
3. Ownership of the incident review should be assigned to the team-member who noticed the near-miss, or, when appropriate, the team-member with the most knowledge of how the near-miss came about.