Reliability Engineering

1. You are here:
2. Engineering
3. Infrastructure
4. Team
5. Reliability Engineering

On this page

• Who We Are
• Vision
• Getting Assistance
  • External Customer Escalations
• Deploying changes to GitLab.com
• Tenets
• On-Call
• Reliability Teams
  • General
  • Observability
  • Foundations
  • Database Reliability
  • Practices
• How We Work
  • General Workflow
  • Sources of work
  • OKRs

If you are a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you are a GitLab team member looking for assistance from Reliability Engineering, please see the Getting Assistance section.

Who We Are

Reliability Engineering is responsible for all of GitLab's user-facing services, with their primary responsibility being GitLab.com. Site Reliability Engineers (SREs) ensure the availability of these services, building the tools and automation to monitor and enable this availability. These user-facing services include a multitude of environments, including staging, GitLab.com, and dev.GitLab.org, among others (see the list of environments).

Vision

Reliability Engineering ensures that GitLab's customers can rely on GitLab.com for their mission-critical workloads. We approach availability as an engineering challenge and empower our counterparts in Development to make the best possible infrastructure decisions. We own and iterate often on how we manage incidents and continually derive and share our learnings by conducting thorough reviews of those incidents.

Getting Assistance

If you're a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you'd like our assistance, please use one of the issue generation templates below and the work will be routed appropriately:

• Open a General Request Issue - follow this link to create a general issue for the Reliability Team.

We can also be reached in Slack in the #production channel for questions related to GitLab.com and in the #infrastructure-lounge channel for all other questions.

External Customer Escalations

Assistance from the Infrastructure Team is occasionally required to help solve or troubleshoot external customer issues.

1. Reach out to the Support Team to ensure that they are aware of the external customer issue.
2. File an issue in the Reliability Issue Tracker
3. Ping @reliability-ems on Slack in the #reliability-lounge channel and include a link to the newly created issue.

Deploying changes to GitLab.com

All larger features, configuration changes, and new services must go through a Production Readiness review.

Tenets

1. Change Management, Incident Management, and Incident Review are owned by Reliability Engineering.
2. Each team member is able to perform oncall for all services.
3. The team is able to reach conclusions independently all the time, consensus most of the time.
4. Career development paths are clear.
5. The team maintains a database of SRE knowledge through documentation, training sessions, and outreach.
6. We leverage the GitLab product where we can in our toolchain.

On-Call

• Manage alerts and incidents as described on the Incident Management Page

Reliability Teams

General

The General Team supports the Reliability Team's overall vision by supporting services for GitLab.com that do not fit the mission of the other Reliability Teams.

Observability

The Observability team maintains metrics and logs platforms for GitLab SaaS and is responsible for Prometheus, Thanos, Grafana, and Logging.

Foundations

The Foundations team builds, runs and owns the core infrastructure for GitLab.com.

Database Reliability

The Database Reliability Team is responsible for the PostgreSQL engine for GitLab.com services, as well as a range of related systems which help to ensure the availability and reliability of the database engine.

Practices

The Practices Team has SRE's that work full time directly with Stage Groups Teams to focus on their Reliability and Infrastructure concerns for GitLab.com when their need is greater than the General Team can support.

How We Work

We maintain a single source of truth epic for all work underway for the Reliability team. That epic can be found at GitLab SaaS Reliability - work queue and represents the current state of project work assigned within teams. That epic references projects detailed in the form of sub-epics.

General Workflow

1. GitLab team members open new issues in Reliability Issue Tracker
2. New issues go through a management and prioritization process

Sources of work

1. Issues generated, mostly by Reliability team members that are necessary to meet our OKRs.
2. Issues generated by GitLab team members outside of Reliability via one of the paths documented in the Getting Assistance section.
3. Issues generated as Corrective Actions for incidents
4. Issues generated as miscellaneous small tasks found in the day-to-day of an SRE/DBRE/EM

OKRs

OKRs for the Reliability team that require status tracking should be updated each Wednesday. When updating the progress percentage of any given KR, it is not necessary to provide extensive notes. One sentence with a link to a larger update is sufficient in most cases. It is also acceptable to do a check-in without providing a note on a single occasion, but not over several check-ins.

In addition to status updates every Wednesday, all objectives and key results assigned to the current quarter must have the following set:

1. The ~Sub-Department::Reliability label applied.
2. The corresponding Reliability team label using the convention ~Reliability::<name>. (e.g. ~Reliability::Practices).
3. For work that is scheduled, assigned to one of the quarterly milestone.
4. Health status set to something other than "None", all OKRs should be set to "On-track" until we believe there will be an issue meeting the objective or the key result.
5. A DRI assigned.