Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Reliability Engineering

  1. You are here:
  2. Engineering
  3. Infrastructure
  4. Team
  5. Reliability Engineering

If you are a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you are a GitLab team member looking for assistance from Reliability Engineering, please see the Getting Assistance section.

Who We Are

Reliability Engineering is responsible for all of GitLab's user-facing services, with their primary responsibility being GitLab.com. Site Reliability Engineers (SREs) ensure the availability of these services, building the tools and automation to monitor and enable this availability. These user-facing services include a multitude of environments, including staging, GitLab.com, and dev.GitLab.org, among others (see the list of environments).

Vision

Reliability Engineering ensures that GitLab's customers can rely on GitLab.com for their mission-critical workloads. We approach availability as an engineering challenge and empower our counterparts in Development to make the best possible infrastructure decisions. We own and iterate often on how we manage incidents and continually derive and share our learnings by conducting thorough reviews of those incidents.

Ownership

The Reliability Team maintains service ownership as defined in the GitLab Service Ownership Policy

Getting Assistance

If you're a GitLab team member and are looking to alert Reliability Engineering about an availability issue with GitLab.com, please find quick instructions to report an incident here: Reporting an Incident.

If you'd like our assistance, please use one of the issue generation templates below and the work will be routed appropriately:

We can also be reached in Slack in the #production channel for questions related to GitLab.com and in the #infrastructure-lounge channel for all other questions.

External Customer Escalations

Assistance from the Infrastructure Team is occasionally required to help solve or troubleshoot external customer issues.

  1. Reach out to the Support Team to ensure that they are aware of the external customer issue.
  2. Support Team to file an issue in the Reliability Issue Tracker
  3. Ping @reliability-ems on Slack in the #reliability-lounge channel and include a link to the newly created issue.

Deploying changes to GitLab.com

All larger features, configuration changes, and new services must go through a Production Readiness review.

Tenets

  1. Change Management, Incident Management, and Incident Review are owned by Reliability Engineering.
  2. Each team member is able to perform oncall for all services.
  3. The team is able to reach conclusions independently all the time, consensus most of the time.
  4. Career development paths are clear.
  5. The team maintains a database of SRE knowledge through documentation, training sessions, and outreach.
  6. We leverage the GitLab product where we can in our toolchain.

On-Call

Reliability Teams

General

The General Team supports the Reliability Team's overall vision by supporting services for GitLab.com that do not fit the mission of the other Reliability Teams.

Observability

The Observability team maintains metrics and logs platforms for GitLab SaaS and is responsible for Prometheus, Thanos, Grafana, and Logging.

Foundations

The Foundations team builds, runs and owns the core infrastructure for GitLab.com.

Database Reliability

The Database Reliability Team is responsible for the PostgreSQL engine for GitLab.com services, as well as a range of related systems which help to ensure the availability and reliability of the database engine.

Practices

The Practices Team has SRE's that work full time directly with Stage Groups Teams to focus on their Reliability and Infrastructure concerns for GitLab.com when their need is greater than the General Team can support.

How We Work

We are mainly working from OKRs with the ~sub-department::Reliability label. Our remaining work is adhoc issues on the Reliability issue backlog.

General Workflow

  1. GitLab team members open new issues in Reliability Issue Tracker
  2. New issues go through a management and prioritization process

Sources of work

  1. Issues generated, mostly by Reliability team members that are necessary to meet our OKRs.
  2. Issues generated by GitLab team members outside of Reliability via one of the paths documented in the Getting Assistance section.
  3. Issues generated as Corrective Actions for incidents
  4. Issues generated as miscellaneous small tasks found in the day-to-day of an SRE/DBRE/EM

OKRs

OKRs for the Reliability team that require status tracking should be updated each Wednesday.

Labels

All Objectives and Key Results should have the following labels:

  1. ~OKR
  2. ~division::Engineering
  3. ~"Department::Infrastructure & Quality"
  4. ~Sub-Department::Reliability
  5. The current quarter label. (e.g. ~FY24-Q3`)
  6. The corresponding Reliability team label using the convention ~Reliability::<name>. (e.g. ~Reliability::Practices)
  7. For work that is scheduled, assigned to one of the quarterly milestone

Description

The description of an Objective or Key Result should include the "why" for making this a focus area for the quarter and how the key result will be scored.

Example

Status updates/check ins

OKRs should be updated every Wednesday with an update of the % completed and a dated update in the comments for current status.

If progress is on track and the % completed is getting updated as expected, it is not necessary to provide extensive status updates.

One sentence with a link to a larger update in a corresponding epic is sufficient in most cases.

It is also acceptable to do a check-in (ie update the % complete) without providing a note on a single occasion, but not over several check-ins.

When status changes to "needs attention" or "at risk", we need to provide context as to why the status has changed, what action is being taken to address the issues and what assistance is required to enable team members not close to the work to understand the situation without requesting additional status updates.

The OKR Handbook has a starting point for when to add Needs Attention and At Risk to Health Status for OKRs

Mid quarter check ins

6 weeks from the start of the quarter is the midpoint check in which requires a more detailed check in. In addition to weekly status update information all objectives and key results should have:

  1. The update should have the title "Mid quarter check-in"
  2. Any known or emerging risks to achieving the result should be noted
  3. Any health status that is not "On track" should contain an update on how to get back on track or the impact will be if not achieved

Retrospective

At the end of the quarter the last check in requires a restrospective. The DRI for the objective or key result should add the following:

  1. The update should have the title "Restrospective"
  2. The score should be finalised
  3. The health should be accurate
  4. Document what went well
  5. Document what could have gone better
  6. Document what could be tried next time
  7. Tag the owner of the parent objective to enable them to complete the same proccess for the objective
  8. Add any thank you or congratulations messages that are appropriate for the effort
  9. Close the objective or key result

Other requirements

In addition to status updates every Wednesday, all Objectives and Key Results assigned to the current quarter must have the following set:

  1. Health status set to something other than "None"; all OKRs should be set to "On-track" until we believe there will be an issue meeting the objective or the key result.
  2. A DRI assigned. The DRI is responsible for providing the weekly updates. This can be a team member who is responsible for the work, or the Engineering Manager for the team. There should only be one DRI
Edit this page View source