If you are a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.
If you are a GitLab team member looking for assistance from Reliability Engineering, please see the Getting Assistance section.
Reliability Engineering is responsible for all of GitLab's user-facing services, with their primary responsibility being GitLab.com. Site Reliability Engineers (SREs) ensure the availability of these services, building the tools and automation to monitor and enable this availability. These user-facing services include a multitude of environments, including staging, GitLab.com, and dev.GitLab.org, among others (see the list of environments).
Reliability Engineering ensures that GitLab's customers can rely on GitLab.com for their mission-critical workloads. We approach availability as an engineering challenge and empower our counterparts in Development to make the best possible infrastructure decisions. We own and iterate often on how we manage incidents and continually derive and share our learnings by conducting thorough reviews of those incidents.
The Reliability Engineering team is composed of DBREs and SREs. As the role titles indicate, they have different areas of specialty but shared ownership of GitLab.com's availability. The team is broken down into three sub-teams, each with their own area of ownership.
The Core Infra teams owns core infrastructure tooling, network ingress/egress, CDNs, DNS, and secrets management.
The people of Core Infra are:
The Datastores team owns our persistent storage platforms, with PostgreSQL on gitlab.com being the top priority.
Datastores is:
The Observability team owns the monitoring and alerting infrastructure for GitLab.com, as well as our caching/queuing infrastructure.
Observability is:
Though we are organized around teams focused on specific domains, the workflow for Reliability remains the same across Reliability teams. The reasoning behind this is to allow for team members to readily engage in projects where they have expertise, but which are related to domains maintained by another Reliability team. An example here would be an SRE from the Datastores team adding their Ansible expertise to a project with a DRI in Core Infra.
As Reliability, we have three main work streams and each varies in the type of work. Those three work streams are:
We maintain a single source of truth epic for all work underway for the Reliability team. That epic can be found at GitLab SaaS Reliability - work queue and represents the current state of project work for the team. That epic references projects detailed in the form of sub-epics.
Projects are assigned and handled within squads. A squad is composed of a minimum of 2 SREs who are responsible for all the sub-epics and issues within a project. Reliability engineering managers are responsible for the formation of squads and the assignment of projects to the respective squads.
The squad works in a collaborative, transparent approach and uses the strengths of respective SRE to deliver the project in the highest quality within the least amount of time. Responsibilities of squads
Project DRI - Leads the squad to make hard project decisions and act on them immediately. Engineering Manager - Facilitate communication(upfront context) and strong alignment between squads
Project epics use the following template in order to capture, current status, DRI, an overview, and any references that help a reader to build context.
Project epics also are labeled with the team::Reliability label and either workflow-infra::Proposal for projects proposed and not yet begun or workflow-infra::In Progress for projects which are underway.
Project epic descriptions have this structure:
@GitLabHandle#### Status
2021-YY-XX - {+ Current status +}
***
#### DRI
{+ GitLab username of the DRI +}
***
#### Overview
{+ Overview of the project and its goals +}
***
#### Reference
1. {+ enumerated list of any reference links for a reader building context +}
example project epics:
We maintain a single queue of corrective actions and work which the EOC on duty should fall back to when not actively engaged in incident management. We apply a quarterly milestone to issues in this queue and use a board based on the milestone to indicate priority and where work is at within the workflow.
We maintain a single queue of active customer requests (asks from other departments are considered customer requests). Work here will be added via a recurring triage process of issues generated from outside the team.
There is also a primary backlog for Reliability board which serves as a singular point of triage for work which has been generated externally to a specific Reliability team. That board is located at Reliability Team - Backlog.
Issues in the Reliability team work streams come from several sources:
Corrective Actions for incidents.What the above all have in common is that they have two labels associated with them, workflow-infra::Triage and team::Reliability.
Issues in the Reliability Team - Backlog are moved from workflow-infra::Triage to the respective milestone or gathered into a project epic and placed in the workflow-infra::Ready state.
If you're a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.
If you'd like our assistance, please use one of the issue generation templates below and the work will be routed appropriately:
We can also be reached in Slack in the #production channel for questions related to GitLab.com and in the #infrastructure-lounge channel for all other questions.
