Reliability:Foundations Team

Maintained by:

Mission

The mission of the Reliability:Foundations team at GitLab is to Build, Run and Own the entire lifecycle of the core infrastructure for GitLab.com.

The team is focused on owning the reliability, scalability, and security of the existing core infrastructure. We seek to reduce the effort required to provide our core infrastructure services, and to enable other teams to self-serve core infrastructure that allows them to more efficiently/effectively run their services for GitLab.com.

In order to enable Infrastructure, Development & Product teams to build their services for GitLab.com and fulfill their respective missions, we work to make the consumption of our services as simple as possible.

We seek to build our services on top of GitLab features, and use cloud vendor managed products to reduce complexity, improve efficiency and deliver new capabilities more quickly, where they are the right choice.

While the team does not explicitly have any product responsibilities, we endeavor to contribute the lessons we learn from running at-scale production systems back to the product teams, and advocate for GitLab to contain features that would allow us to DogFood.

Vision

The Foundations Team supports the rest of Infrastructure and Development by providing the resources that other teams build upon. We do so by working collaboratively, iteratively, and striking the right balance between delivering results quickly yet safely.

Ownership

Services

The Services that the Foundations team is responsible for fall into two general categories: Core and Edge.

Core

Service	Description	Co-Ownership?
K8s	K8S workloads deployments, Cluster addons	Autodeploy remains with Delivery, and anything Delivery related is co-owned with Delivery
Config	Terraform, Chef, Image Builds	The core TF repos are owned by Foundations, while specific modules may be maintained by the teams that use them
Service discovery	Consul
Secrets Management	Vault	Vault is offered as a service to enable teams to manage their own secrets
Ops	Ops.gitlab.net, Ops Runners

Edge

Service	Description	Co-Ownership?
CDN	Cloudflare
DNS	AWS Route 53, Cloudflare
Load Balancing	HAProxy, Ingress
Networking	Cloud VPCs, Cloudflare
Rate Limiting	Rate limiting	shared ownership with development teams for specific endpoints and with abuse
RBAC/IAM	Teleport, GCP IAM permissions and project creation

Overlap

Given the nature of this team's scope, several services the Foundation team works with are either co-owned by other teams or directly impact their work, as noted above.

Who are our Stakeholders?

Our primary customers are other teams in the Infrastructure department. Our services have particular overlap and impact on the Delivery teams and Reliability::General. Other teams outside of Infrastructure that we collaborate with regularly are Support and various teams in the Security organization.

Performance Indicators

We've adopted a version of the SPACE framework for Performance Indicators.

For more context, see the related discussion issue.

Satisfaction ([DIB(https://handbook.gitlab.com/handbook/values/#diversity-inclusion)])
- Foundations Engineering Manager will send out a monthly team satisfaction survey with a range of questions that seek to capture a sense of trust, belonging, inclusion and feeling empowered.
- Success Criteria: no lower than 4 out of 5 average for questions around psychological safety, satisfaction with the team and feeling supported.
- Current status: green
  - To encourage people to be as honest as possible, we are not sharing specific results beyond the team.
Performance (Results)
- OKRs - OKRs are generated each quarter based on current commitments while also including spare capacity for unplanned work.
  - Success Criteria: > 80% completion of OKRs
  - Current Status: tbd pending final review of our cost savings efforts scores.
  - OKR completion rates by quarter:
    - (FY24 Q2): current 77.3%
    - (FY24 Q1): 88%
- Service SLOs
  - Success Criteria: Meets or Exceeds availability SLOs for services we own.
  - TODO: Create and add link to overview dashboard
Activity (Results)
- Corrective Actions Over Time (specific to the Foundations Team)
  - Success Criteria: Meets or exceeds the Reliability SLO
  - Current Status: Green for Sev 1 and Sev 2 CAs.
- Track the rate of closed MRs for projects for services we own.
  - Success Criteria: TBD. Gather data to see if there is any correlation between MR activity and reliability numbers for services we own.
- TODO: Add Sisense or Tableau dashboards to visualize these issue metrics
Communication and Collaboration
- Customer Satisfaction
  - Foundations Engineering Manager will send out quarterly surveys to the rest of Infrastructure with questions regarding ease of collaboration, ease of getting needs met and pain points.
  - Success Criteria: average rating of 4 out of 5 for questions about ease of collaboration and getting needs met.
  - Current Status: Green
  - Most recent Survey Results: June 2023:
    - How Easy is it to collaborate with Foundations? 4.25 out of 5
    - How Easy is it to get help from Foundations? 4.25 out of 5
Efficiency and Flow
- Issue/MR Metrics
  - Issue Lead Time
    - Success Criteria: Meets or exceeds current Reliability SLO
  - Throughput times for MRs by team members
    - Success Criteria: TBD
  - TODO: Add Sisence or Tableau dashboards to visualize these metrics

Team Members

Person	Role
Anna Liisa Moter	Engineering Manager, Reliability Engineering
Filipe Santos	Senior Site Reliability Engineer
Marcel Chacon	Senior Site Reliability Engineer
Milad Irannejad	Senior Site Reliability Engineer
Pierre Guinoiseau	Senior Site Reliability Engineer
Sarah Walker	Site Reliability Engineer

Key Technical Skills

The Foundations Team must maintain a broad and diverse set of technical skills while also maintaining the ability to switch contexts frequently. Some of these technical skills include:

Cloudnative Engineering - Proficiency in Kubernetes and the associated ecosystem of running cloudnative services.
Infrastructure as Code - Proficiency in Chef and Terraform
Network Systems - Understanding of network concepts and experience with our Edge stack (see Edge services above)

Common Links

How We Work

### Values

In addition to striving to embrace GitLab's values, the Foundations team seeks to embody the following:

Excellence - We do work at a high technical standard, and build maintainable systems with a long term perspective.
Connection - We work in a way that is collaborative and feels connected to each other.
Fun! - While the work we do is important, we can laugh, make jokes and enjoy our work day to day.
Trust and Safety - We build trust by doing what we say we’re going to do when we say we’ll do it, or communicating clearly when and why that didn’t happen. We feel safe to show up and say what is on our minds. We feel safe asking for help.
Ownership - We are responsible for the services we own, and we can make considered decisions about the things we are responsible for, while caring about the stakeholders who consume our work.
Courage - We are willing to try new things in service of improving our infrastructure. We accept the associated risks and take responsibility when things go wrong.

Processes

Team Sync meetings

We have weekly synchronous meetings in two different time zones to encourage discussion, give status updates, and connect as a team.
- Agenda
- The meetings are recorded.
We have monthly retrospective meetings in two different time zones.
- We celebrate our wins and look for ways we can improve team processes.
- These meetings are not recorded to create a safe space for sharing transparently.
- We use an online tool, teleretro.com, which allows for running retros across multiple time zones, enabling brainstorming without being able to see others' responses, merging of related topics, and voting for what to discuss in more depth.
- Action items for iterative improvements on processes are brought back to the team syncs following the monthly retro meetings.
We have twice weekly Triage meetings available to us to ensure that incoming external requests are seen and responded to quickly. This allows us to meet our SLO for P2 issues being responded to within 3 days of creation.
- Incoming issues are scoped and prioritized in the Triage meetings
  - We use T-shirt labels to estimate the size of incoming issues, per the definitions agreed upon in our agenda. This helps not only in our planning efforts, it also helps to track our historical velocity.
  - When assessing incoming issues, we review the assigned priority label, and assign one if a priority label is missing.
  - If the issue is deemed not to fall within Foundations' ownership, it is reassigned to the appropriate team with a note to that team's EM.
  - If the issue is deemed to be not actionable with the provided information, we will ask the issue creator for the required information. If we do not get any response within a 2 week timeframe, the issue will be closed with a note to the creator.
  - Some issues are assessed to be no longer relevant, or the benefit is so outweighed by the cost of implementation that it is not worth doing. In those cases, we apply the label ~"workflow-infra::Cancelled" and close the issue with a note to the creator explaining our reasoning.
- Any triaged issues that are ready for work are then brought to team syncs for discussion and assignment, unless they are marked urgent (P1 or P2), in which case we will bring them to the team via Slack.

Standup

We have Geekbot automated checkins on Mondays and Fridays in the #g_infra_foundations_social channel

Prioritization of work

The reality is that Foundations, like many other teams, has more work assigned and planned than the team currently has capacity for. This makes prioritization an important, ongoing process.
Our team members are assigned work related to our OKRs, which is by default to be prioritized over other tasks.
In the process of that work, we often encounter unexpected tech debt, especially as we are still in the early days of the team and we are onboarding services into our ownership.
When encountering such tech debt, we must weigh and prioritize the additional work.
- Tech debt that is an absolute blocker in order to move forward with project work must be addressed.
- At other times, we may encounter a big chunk of tech debt that, if we take the time to address it now, will pay dividends by making future work easier, but also means that other work that was committed to this quarter may not get completed. In such cases, we encourage team discussion to align on the decision, and documenting said decision making in issues and epics, so that we can clearly point to this when giving context for the re- or de- prioritization of work we committed to.
- Yet other times, we must make the decision to document the non-blocking tech debt we encountered in an issue and place it in the backlog to be addressed at a later time, in order to be able to deliver on the work we said we would do by the given timeframe.
We must remain vigilant of scope creep
- When planning out work related to larger projects, it is important to distinguish between "nice to haves" and "essential" pieces of work in order to stay focused on our outcomes and avoid scope creep.
- Similarly, external requests that at first glance appear small in scope can quickly snowball into much larger scope. Team members are encouraged to bring these to the team's attention, and the EM can step in and reset expectations and clarify commitments as needed.
Interrupt driven work is another reality of the nature of this team.
- Engineers participate in SRE oncall rotations, which can take them completely away from project work for the week, and sometimes more time afterwards for follow up work.
- Other times there are incoming external requests that are unplanned for but unblock another team or customer, and that must be prioritized over ongoing work. When necessary, we discuss this as a team.
  - We make an effort to distribute this type of work in an equitable manner across the team.
  - External requests should come through issues created in the Reliability tracker with the ~"team::Foundations" label. If other teams reach out directly to team members for help with work, we should redirect them to request work through the proper channels so that the team can properly prioritize and scope the work, and individuals are not pulled off prioritized tasks through back channels.

Communication with stakeholders

GitLab OKRs capture our commitments for each quarter and are generally updated every Tuesday
- Updates should give sufficient context for leadership to understand current status and explain any changes in health status, completion dates or deliverables.
GitLab Epics capture large pieces of work and those labeled "In Progress" are also generally updated on Tuesdays
GitLab Issues capture smaller, concrete pieces of work, and those labeled "In Progress" should be updated twice weekly or whenever a portion of work has been completed.
Slack is the default method of reaching out between team members
- #g_infra_foundations is for work related discussions, external requests, etc
- #g_infra_foundations_social is for socializing and standups
- #g_infra_foundations_notifications is for automated MR notifications
- We also have team slack handles that enable us to get the whole team's attention at once
  - @infra-dwarves is our internal handle, meant for intra-team communication
  - @infra-foundations can be used by external stakeholders when they need to get our attention outside of #g_infra_foundations
We currently have monthly sync meetings with Delivery::Systems, as they are the biggest "customer" of Foundations and it is important to stay closely aligned with expectations and ongoing work.