Reliability Engineering

1. You are here:
2. Engineering
3. Infrastructure
4. Team
5. Reliability Engineering

Maintained by:

On this page

• Who We Are
• Vision
• Tenets
• Team
  • Team members
• How We Work
  • Project Squads - Squad List
    • Roles
    • Project Epic Template
    • Milestone
  • How We Work – EOC Queue
  • How We Work – Customer Request Queue
  • How We Work – Issue Triage
  • How We Work – Corrective Actions workflow
    • Refinement of open CAs
    • Working on CAs
• Getting Assistance

If you are a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you are a GitLab team member looking for assistance from Reliability Engineering, please see the Getting Assistance section.

Who We Are

Reliability Engineering is responsible for all of GitLab's user-facing services, with their primary responsibility being GitLab.com. Site Reliability Engineers (SREs) ensure the availability of these services, building the tools and automation to monitor and enable this availability. These user-facing services include a multitude of environments, including staging, GitLab.com, and dev.GitLab.org, among others (see the list of environments).

Vision

Reliability Engineering ensures that GitLab's customers can rely on GitLab.com for their mission-critical workloads. We approach availability as an engineering challenge and empower our counterparts in Development to make the best possible infrastructure decisions. We own and iterate often on how we manage incidents and continually derive and share our learnings by conducting thorough reviews of those incidents.

Tenets

1. Change Management, Incident Management, Incident Review and Delta Management are owned by Reliability Engineering.
2. Each team member is able to work on all team projects.
3. The team is able to reach conclusions independently all the time, consensus most of the time.
4. Career development paths are clear.
5. The team maintains a database of SRE knowledge through documentation, training sessions, and outreach.
6. We leverage the GitLab product where we can in our toolchain.

Team

The Reliability Engineering team is composed of DBREs and SREs. As the role titles indicate, they have different areas of specialty but shared ownership of GitLab.com's availability.

Team members

How We Work

We maintain a single source of truth epic for all work underway for the Reliability team. That epic can be found at GitLab SaaS Reliability - work queue and represents the current state of project work assigned within squads. That epic references projects detailed in the form of sub-epics.

As Reliability, we have three main work streams and each varies in the type of work. Those three work streams are:

1. Projects - These are pre-planned units of work, managed via an epic and a single DRI who collaborates with other members of Reliability.
2. EOC Queue - This is a quarterly milestone consisting of corrective actions and EOC tooling maintenance.
3. Customer Request Queue - This is a quarterly milestone of all small-scale asks which originate from teams outside of Reliability and which are not tied a Project of corrective action. Examples here include adding DNS entries, provisioning/de-provisioning system access, and assisting Support.

Project Squads - Squad List

Projects are assigned and handled within squads. A squad is composed of a minimum of 2 SREs who are responsible for all the sub-epics and issues within a project. Reliability engineering managers are responsible for the formation of squads and the assignment of projects to the respective squads.

The squad works in a collaborative, transparent approach and uses the strengths of respective SRE to deliver the project in the highest quality within the least amount of time. Responsibilities of squads:

1. Scope with timelines and deliver on the issues within the assigned epic
2. Assign individual issues within an epic to individual squad members
3. Hold weekly recorded demos, where possible as per the project needs
4. Have a balance of experience and subject matter skills
5. Participate as Infrastructure stable counterparts to other Engineering stage groups
6. Each squad has a lead who keeps the status of the project updated as frequently as necessary, minimum weekly

Roles

Squad lead - Leads the squad to make hard project decisions and act on them immediately. Engineering Manager - Facilitate communication(upfront context) and strong alignment between squads

Project epics use the following template in order to capture, current status, DRI, an overview, and any references that help a reader to build context.

Project epics also are labeled with the team::Reliability label and either workflow-infra::Proposal for projects proposed and not yet begun or workflow-infra::In Progress for projects which are underway.

Project epic descriptions have this structure:

1. Status - Current status of the project preceded by the ISO formatted date (YYYY-MM-DD) of the latest status update. Example: "2021-07-20 - DRI selected and initial project sync meeting held."
2. DRI - Directly responsible individual working on the specific issue. This is the GitLab handle of the DRI, @GitLabHandle
3. Overview - Narrative description of the project intended to provide additional context. This is a few paragraphs setting context for someone new to the project and wanting to learn more and contribute.
4. Reference - Links out to existing artifacts which allow team members to build context. These are links to documents and any dependent projects or prior art.

Project Epic Template

#### Status
2021-YY-XX - {+ Current status +}

***
#### DRI
{+ GitLab username of the DRI +}

***
#### Overview
{+ Overview of the project and its goals +}

***
#### Reference
1. {+ enumerated list of any reference links for a reader building context +}

example project epics:

1. https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/471

Milestone

The squad lead can use milestone under the gitlab-com/gl-infra to keep track of what is going to be achieved in the next block of time.

• Duration: It's suggested that the milestone is only for 2 weeks since that gives individual contributors enough time to dive deep into an issue and fix it, but it's not long enough that if we have to pivot because of a change of priorities it won't disrupt the planning, we don't want to end up in a situation where we spend a lot of time planning 1 month and then have it change completely a week in because of a large priority change.
• Interruption work: If individual contributors have to work on an issue outside of the planned milestone they should add it to the milestone that they are working on. This gives a better overview of what kind of interruptions we had during that milestone.
• Issue boards: If your squad is using issue boards you can edit the board to be scoped to a specific milestone. Like this, you can keep track of the progress on the milestone.
• Number of issues doesn't matter: The higher the number of issues inside of the milestone doesn't mean the more productive the squad is. This is not to be used as a metric.

What are the benefits?

• Gives the squad a common goal that everyone is clear on what's being worked on in the coming days.
• Reduce scope, if an issue can't fit in 2 weeks or the milestone duration it's clear that it's a big issue and needs to be broken down.
• Tracking interruptions.
• Gives a high overview to the engineering manager of what is being worked on.
• Gives a sense of urgency that something needs to be finished in the allocated time, hopefully, this won't lead to burnout but a discussion about our workload and a sense that we need to focus on 1 issue at a time.
• A sense of accomplishment, when the milestone is close and the squad looks back at what achieved in that milestone is can give them a great sense of accomplishment!

Below are some examples of past milestones that piloted this process:

• https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/153
• https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/152

How We Work – EOC Queue

We maintain a single queue of corrective actions and work which the EOC on duty should fall back to when not actively engaged in incident management. We apply a quarterly milestone to issues in this queue and use a board based on the milestone to indicate priority and where work is at within the workflow.

• EOC Queue Milestones
• EOC Queue Board

How We Work – Customer Request Queue

We maintain a single queue of active customer requests (asks from other departments are considered customer requests). Work here will be added via a recurring triage process of issues generated from outside the team.

• Customer Request Queue Milestones
• Customer Request Queue Board

How We Work – Issue Triage

There is also a primary backlog for Reliability board which serves as a singular point of triage for work which has been generated externally to a specific Reliability team. That board is located at Reliability Team - Backlog.

Issues in the Reliability team work streams come from several sources:

1. Issues generated by customers (teams outside Infrastructure) via one of the paths documented in the Getting Assistance section. These issues are automatically added to the triage backlog and are processed by Reliability managers.
2. Issues generated as Corrective Actions for incidents.
3. Issues generated as miscellaneous small tasks found in the day-to-day of an SRE/DBRE/EM

What the above all have in common is that they have two labels associated with them, workflow-infra::Triage and team::Reliability.

Issues in the Reliability Team - Backlog are moved from workflow-infra::Triage to the respective milestone or gathered into a project epic and placed in the workflow-infra::Ready state.

How We Work – Corrective Actions workflow

Corrective Actions are issues arising from incidents. See the link for the suggested way to create them.

We use this board to track corrective actions work. Corrective Actions are also an important performance indicator for the Infrastructure Department.

Currently there is a squad assigned to this work with the focus of 1) refining all open CAs and 2) Burning down the backlog of open CAs. The slack channel for that squad is #infra-corrective-actions. Anyone is welcome and encouraged to help with the work of CAs.

Refinement of open CAs

The process is as follows:

1. Choose an open, unrefined "corrective action" issue (ie it does not have a "ca::refined label") in the GitLab Infrastructure Team project. Add the "workflow-infra::Under Review" label, and assign it to yourself. This ensures that other engineers don't pick up the same issue for refinement asynchronously. Issues with the label "ca::triage" or any of "severity::1", "severity::2", "priority::1", "priority::2" should be prioritized for refinement. This list can be a useful place to start.
2. Read the issue carefully, including any linked incidents, to get context.
3. If there is no linked incident, this issue may be mislabeled. Confirm with the issue creator and if there is no incident related to this issue, remove the "corrective action" label. In this case you are done refining this issue.
4. Ensure it has a severity label on it - this is the highest severity of linked incident(s).
5. Ensure it has a priority label on it, indicating the urgency of the issue. For guidance on priority level, see the engineering triage handbook page. When in doubt, feel free to ask the issue creator or other relevant engineers for input.
6. Based on the above, possible actions are:
   1. Cancel the issue by adding the label "workflow-infra::Cancelled" and then closing the issue
      1. Because it is no longer relevant due to other changes in the infrastructure, whether already executed or planned.
      2. You consider it an infrequent enough or low priority problem that you don't think it is currently worth prioritizing the work given the team's current resources.
      3. In either case, add a note to the issue explaining the reason for cancelling.
      4. Don't hesitate to cancel old, low priority CAs. An issue can always be reopened if it is deemed to be an ongoing problem, in which case we can increase the severity/priority and reassess.
   2. Indicate that this CA is ready for work by adding the label "workflow-infra::Ready"
      1. This indicates that the issue is well scoped (to be able to be completed with at most a few days' engineering time), and has the necessary information and context for an engineer to pick up for work.
   3. Promote to an Epic
      1. Sometimes CAs are high priority but very large in scope (more than a few days' worth of work). In this case the CA can be promoted to an Epic. Update the description of the new epic to show that it was created from a CA and make sure that the incident is referenced and linked in that paragraph.
      2. If it falls within the CA squad's scope, leave the Corrective Action label in place. Otherwise remove it with a note.
      3. New issues that are created related to an Epic with a "corrective action" label should not get that label, so as not to artificially inflate the numbers for tracking purposes.
   4. Move to another team/project
      1. Sometimes it becomes clear that a particular corrective action is not an Infrastructure issue. Add a note and move the issue to the appropriate project. If the issue is the responsibility of a stage group, consider adding the ~infradev label.
   5. Needs more information
      1. If context is missing or scope is unclear, ping the relevant engineers or managers in the issue to gain clarity so that a determination can be made for one of the above four cases. Sometimes we bring issues we need more input on up for discussion at one of our weekly Reliability Discussion meetings for this purpose.
   6. Mark the issue as "workflow-infra::Ready" and add the label "ca::refined" to indicate the CA squad has looked at it. This indicates the issue is A. still relevant B. well scoped C. has enough context for the engineer picking up the issue and D. is ready for work.
7. When refinement is complete and the issue has one of above outcomes, label the issue with "ca::refined" to indicate refinement is completed and remove the "workflow-infra::Under Review" label. Unassign yourself from the issue unless you know you will be the one taking on the work.
8. Sometimes when refining a series of Corrective Action issues, a pattern may emerge that indicates that perhaps project level work is called for. Feel free to create an issue explaining your findings and bring this to the attention of the current DRI for corrective actions and/or an engineering manager.

Working on CAs

1. Choose a CA from the Ready column from the above linked board that has no engineer assigned to it. Start with the highest severity/priority level issue that you feel able to take on.
2. Assign yourself to the issue and move it to the "in progress" column or add the label "workflow-infra::In Progress".
3. Reach out to other engineers as needed for more context or pairing. CAs cover a very wide range of infrastructure areas, and you are not expected to know everything. Often it is much more efficient to pair with someone more knowledgable in that domain.
4. When the work is complete, remember to close the issue or move it to the "Done" column.

Getting Assistance

If you're a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you'd like our assistance, please use one of the issue generation templates below and the work will be routed appropriately:

• Open a General Request Issue - follow this link to create a general issue for Reliability Engineering.
• Open a Customer Questions and Sales Enablement Issue - follow this link to seek assistance in answering questions for prospects or current customers.
• Open a request for Infrastructure to join customer call.

We can also be reached in Slack in the #production channel for questions related to GitLab.com and in the #infrastructure-lounge channel for all other questions.