Reliability Engineering

1. You are here:
2. Engineering
3. Infrastructure
4. Team
5. Reliability Engineering

Maintained by:

On this page

• Who We Are
• Vision
• Tenets
• Team
  • Core Infra
  • Datastores
  • Observability
• How We Work
  • Work Generated Internally by a Reliability Team
  • Work Generated Externally to the Reliability Team
• Getting Assistance

If you are a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you are a GitLab team member looking for assistance from Reliability Engineering, please see the Getting Assistance section.

Who We Are

Reliability Engineering is responsible for all of GitLab's user-facing services, with their primary responsibility being GitLab.com. Site Reliability Engineers (SREs) ensure the availability of these services, building the tools and automation to monitor and enable this availability. These user-facing services include a multitude of environments, including staging, GitLab.com, and dev.GitLab.org, among others (see the list of environments).

Vision

Reliability Engineering ensures that GitLab's customers can rely on GitLab.com for their mission-critical workloads. We approach availabilty as an engineering challenge and empower our counterparts in Development to make the best possible infrastructure decisions. We own and iterate often on how we manage incidents and continually derive and share our learnings by conducting thorough reviews of those incidents.

Tenets

1. Change Management, Incident Management, Incident Review and Delta Management are owned by Reliability Engineering.
2. Each team member is able to work on all team projects.
3. The team is able to reach conclusions independently all the time, consensus most of the time.
4. Career development paths are clear.
5. The team maintains a database of SRE knowledge through documentation, training sessions, and outreach.
6. We leverage the GitLab product where we can in our toolchain.

Team

The Reliability Engineering team is composed of DBREs and SREs. As the role titles indicate, they have different areas of specialty but shared ownership of GitLab.com's availability. The team is broken down into three sub-teams, each with their own area of ownership.

Core Infra

Team Page

The Core Infra teams owns core infrastructure tooling, network ingress/egress, CDNs, DNS, and secrets management.

The people of Core Infra are:

Datastores

Team Page

The Datastores team owns our persistent storage platforms, with PostgreSQL on gitlab.com being the top priority.

Datastores is:

Observability

Team Page

The Observability team owns the monitoring and alerting infrastructure for GitLab.com, as well as our caching/queuing infrastructure.

Observability is:

How We Work

Each team withing Relaibility manages its own backlog and has its own sprints.

There is also a primary backlog for Reliability board which serves as a singular point of triage for work which has been generated externally to a specific Reliability team. That board is located at Reliability Team - Backlog.

Issues in team backlogs come from two sources:

Work Generated Internally by a Reliability Team

1. The teams themselves derive ideas for work that align with their areas of ownership and their roadmap. An issue is created with the team's label and the ~"workflow-infra::Triage" label, this automatically places it on the team's backlog board.
2. The issue then takes one of several paths on the team's backlog either:
   1. prioritized and kept on the board for a future sprint and the ~"workflow-infra::Triage" label is removed and the ~"workflow-infra::Ready" label is added.
   2. pulled into a sprint milestone during the team's sprint planning session and the ~"workflow-infra::Triage" label is removed and the ~"workflow-infra::Ready" label is added.
   3. closed if it is determined it won't be pulled into a sprint in the near future and the ~"workflow-infra::Triage" label is removed.

Work Generated Externally to the Reliability Team

1. Work is generated from a group outside of a specific Reliability team and an issue is created with the labels ~"team::Reliability" and ~"workflow-infra::Triage". This places it on the Reliability - Backlog board and in a triage process for assignment to the correct subteam.
2. Using the Reliability - Backlog board, the daytime IMOC reviews the board for issues in the ~"workflow-infra::Triage" column and the issue is either:
   1. moved to the appropriate team board by removing the ~"team::Reliability" label and adding the appropriate team label.
   2. closed if it is determined it won't be pulled into a sprint in the near future and the ~"workflow-infra::Triage" label is removed.
   3. brought to the Reliability managers sync meeting to be triaged by the group.
3. If the issue made it to a team backlog board, it will be processed by the team and either:
   1. prioritized and kept on the board for a future sprint and the ~"workflow-infra::Triage" label is removed and the ~"workflow-infra::Ready" label is added.
   2. pulled into a sprint milestone during the team's sprint planning session and the ~"workflow-infra::Triage" label is removed and the ~"workflow-infra::Ready" label is added.
   3. closed if it is determined it won't be pulled into a sprint in the near future and the ~"workflow-infra::Triage" label is removed.

Getting Assistance

If you're a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you'd like our assistance, please use one of the issue generation templates below and the work will be routed appropriately:

• Open a General Request Issue - follow this link to create a general issue for Reliability Engineering.
• Open a Customer Questions and Sales Enablement Issue - follow this link to seek assistance in answering questions for prospects or current customers.
• Open a request for Infrastructure to join customer call.

We can also be reached in Slack in the #production channel for questions related to GitLab.com and in the #infrastructure-lounge channel for all other questions.