Scalability Team

1. You are here:
2. Handbook
3. Engineering
4. Infrastructure
5. Team
6. Scalability Team

On this page

• Common Links
• Mission
• Vision
• Team Members
• Team counterparts
• How do I engage with the Scalability Team?
• How does the Scalability Team engage with Stage Groups?
• How we work
  • Meetings and Scheduled Calls
  • Project Management
  • Prioritization Process
  • Triage rotation
  • Issues
  • Issue boards
  • Labels
    • Workflow labels
    • Priority labels
• Choosing something to work on

Common Links

Mission

The Scalability team is responsible for GitLab and GitLab.com at scale, working on the highest priority scalability items in the application in close coordination with Reliability Engineering teams and providing feedback to other Engineering teams so they can become better at scalability as well.

Vision

As its name implies, the Scalability team enhances the availability, reliability and, performance of GitLab by observing the application's capabilities to operate at GitLab.com scale. The Scalability team analyzes application performance on GitLab.com, recognizes bottlenecks in service availability, proposes short term improvements and develops long term plans that help drive the decisions of other Engineering teams.

Short term goals include:

• Refine existing, define new, and document Service Level Objectives for each of GitLab's services.
• Continuously expose the top 3 critical bottlenecks that threaten the stability of GitLab.com.
• Work on scoping, planning and defining the implementation steps of the top critical bottleneck.
• Define and track team KPI's to track impact on GitLab.com and GitLab as an application.

Team Members

The following people are members of the Scalability Team:

Team counterparts

The following members of other functional teams are our stable counterparts:

We work with all engineering teams across all departments as a representative of GitLab.com as one of the largest GitLab installations, to ensure that GitLab continues to scale in a safe and sustainable way.

The Memory team is a natural counterpart to the Scalability team, but their missions are complementing each other rather than overlap:

Simply put:

• The Scalability team is focused on all work that affects GitLab.com SLOs.
• The Memory team is focused on general GitLab resource consumption and performance.

How do I engage with the Scalability Team?

1. Start with an issue in the Scalability team tracker: Create an issue.
2. You are welcome to follow this up with a Slack message in #g_scalability.
3. Please don't add any workflow labels to the issue. The team will triage the issue and apply these.
4. We use our Workflow board to track the workflow of issues.

How does the Scalability Team engage with Stage Groups?

When we observe a situation on GitLab.com that needs to be addressed alongside a stage group, we first raise an issue in the Scalability issue tracker that describes what we are seeing. We try to determine if the problem lies with the action the code is performing, or the way in which it is running on GitLab.com. For example, with queues and workers, we will see if the problem is in what the queue does, or how the worker should run.

If we find that the problem is in what the code is doing, then we engage with the EM/PM of that group to find the right path forward. If work is required from that group, we will create a new issue in the gitlab-org project and use the Availability and Performance Refinement process to highlight this issue.

How we work

Meetings and Scheduled Calls

We prefer to work asynchronously as far as possible but still use synchronous communication where it makes sense to do so.

To that end, we have very few scheduled calls.

• Weekly on Fridays we have a demo call. The purpose of this call is to showcase something that you have been working on during that week. It does not have to be perfect or polished. Items should be added to the agenda 24 hours ahead of the meeting. If there are no agenda items at that time, the call is cancelled for that week. A playlist of the recordings is available on GitLab Unfiltered.
• Monthly (in the second week of a month) we have a social call. The purpose of this call is to catch up with each other and calls may be themed to start the conversation. These are set up as double-calls, where there are two calls booked to suit people in all time zones. The intention is for everyone to attend at least one call, but to please join both if possible. Since this is only done monthly, this should hopefully not be too much of a burden to take a call at an unusual hour.

Lastly, in order to keep people connected, team members schedule at least one coffee-chat with another team member each week. These are at times that will best suit them as it may be an unusual hour given the various timezones and working hours for each person.

Project Management

We use Epics, Issues, and Issue Boards to organize our work, as they complement each other.

The single source of truth for all work is Scaling GitLab.com epic. This is considered as the top-level epic from which all other epics are derived.

Epics that are added as children to the top-level epic are used to describe projects that the team undertakes. Projects must have a problem statement and a goal, and issues are added to these Epics if they can be shown to further or support the goals of that project.

Having all projects at this level allows us to use a single list for prioritization and enables us to prioritize work for different services alongside each other. Projects are prioritized in line with the OKRs for the current quarter.

For each project we use the start date and due date fields on the Epic.

• Start Date: set to the expected start date and updated to be the actual start date when the project begins.
• Due Date: set to be the expected end date. This should be seen as a target, and this target is re-evaluated every few weeks while the project is in progress. The date that a project actually ended is taken from the date that the epic was closed.

Project status is maintained in the description of the top-level epic so that it is visible at a glance. This is auto-generated. You can watch a short demo of this process to see how to use status labels on the epics to make use of this automation.

Example organization is shown on the diagram below:

Note If you are not seeing the diagram, make sure that you accepted all cookies.

Prioritization Process

The diagram below describes how the work gets created in the Scalability team, and added to the top-level epic Scaling GitLab.com epic:

Note If you are not seeing the diagram, make sure that you accepted all cookies.

The process contains 6 cyclical stages:

1. Observe - What is causing SLA and SLO degradations on GitLab.com? Monitor the 4 golden signals provided by general metrics (latency, traffic, errors, saturation) for each service, looking for SLA breaches (for latency, errors, saturation) and prioritising for the worst breaches
2. Analyse - Why is availability being reduced, do we have all information, and are our metrics sufficient? Investigate the major causes leading to reduction in availability on GitLab.com. What are the reasons for these degradations and outages? Investigate to understand the cause.
3. Proposed Improvements - Issue with a (partial, temporary or full, permanent) proposal is created on the Scalability tracker, with one or more additional issues in other trackers as required, including estimated SLA improvements for services affected. Improvements can include changes to the infrastructure, changes to the application, and changes to our observability. Issues involving substantial effort will be promoted to Epics so that issues encapsulating this effort can be tracked as a project.
4. Triage - Prioritise changes based on pre-defined set of rules and according to expected availability improvements. Tickets can be either delegated to engineering teams via the infra/dev process, delegated to infrastructure (via TBD process), or implemented by the scalability team
5. Development & Deployment - The work on developing and ensuring that the change has no unexpected effects is executed by the owner defined in the previous stage.
6. Assessment - Assessment of the implemented change is done through retrospecting on the expected and observed state. The retrospective process is documented in an issue that is marked related to the original issue driving the change. Can we see the changes we expected following the deployment of this change? If not, why is this?

Triage rotation

We have automated triage policies defined in the triage-ops project. These perform tasks such as automatically labelling issues, asking the author to add labels, and creating weekly triage issues.

We currently have two weekly triage issues:

1. Board refinement - walk through the current project board and move issues forward towards workflow-infra::Ready where possible.
2. Service::Unknown refinement - lists issues with Service::Unknown with the goal of adding a defined service, where possible.

We rotate the triage ownership each month, with the current triage owner responsible for picking the next one (a reminder is added to their last triage issue).

Issues

Issue is being implemented if:

1. Issue has a team member assigned to it.
2. Assigned issue has a priority label set.
3. Issue has "~workflow-infra::In Progress" set.

Issue is resolved when:

1. The problem defined in the issue has been addressed.
2. Issue description is updated with a graph comparing before/after state (if applicable).
3. Issue has "~workflow-infra::Done" set.

Issue boards

The Scalability team issue boards track the progress of ongoing work. Purpose of some of the more important issue boards are described below:

1. Workflow board
   • Tracks the whole team ongoing workload.
2. Abandoned work board
   • Tracks the work that is not progressing.
3. Individual services board, for example Sidekiq board
   • Tracks the workload for the individual service.
4. Priority board
   • Tracks the workload based on issue priorities.

Labels

The Scalability team routinely uses the following set of labels:

1. The team label, team::Scalability.
2. Priority labels Scalability::.
3. Scoped workflow-infra labels.
4. Scoped Service labels.

The team::Scalability label is used in order to allow for easier filtering of issues applicable to the team that have group level labels applied.

The priority labels allow us to track the issues correctly and raise/lower priority of work based on both external and internal factors.

This means that the highest priority is given to working on issues that improve Gitlab.com SLO's either immediately and directly, or by unblocking other issues to achieve the same.

Workflow labels

The Scalability team leverages scoped workflow labels to track different stages of work. They show the progression of work for each issue and allow us to remove blockers or change focus more easily.

The standard progression of workflow is from top to bottom in the table below:

State Label	Description
	Problem is identified and effort is needed to determine the correct action or work required.
	Proposal is created and put forward for review. SRE looks for clarification and writes up a rough high-level execution plan if required. SRE highlights what they will check and along with soak/review time and developers can confirm. If there are no further questions or blockers, the issue can be moved into "Ready".
	Proposal is complete and the issue is waiting to be picked up for work.
	Issue is assigned and work has started. While in progress, the issue should be updated to include steps for verification that will be followed at a later stage.
	Issue has an MR in review.
	MR was merged and we are waiting to see the impact of the change to confirm that the initial problem is resolved.
	Issue is updated with the latest graphs and measurements, this label is applied and issue can be closed.

There are three other workflow labels of importance:

State Label	Description
	Work in the issue is being abandoned due to external factors or decision to not resolve the issue. After applying this label, issue will be closed.
	Work is not abandoned but other work has higher priority. After applying this label, team Engineering Manager is mentioned in the issue to either change the priority or find more help.
	Work is blocked due external dependencies or other external factors. Where possible, a blocking issue should also be set. After applying this label, issue will be regularly triaged by the team until the label can be removed.

Priority labels

The Scalability team uses priority labels as a means to indicate order under which work is next to be picked up. Priorities are roughly defined as:

Choosing something to work on

We work from our main epic: Scaling GitLab on GitLab.com.

Most of our work happens on the current in-progress sub epic. This is always prominently visible from the main epic's description. From there, work takes place on the board associated to the current in-progress epic.

Priority labels take precedence; we don't use issue ordering in boards or epics for priorities. Workflow labels to the right are higher priority than those to the left.