| Workflow | Team workflow | |
| GitLab.com | @gitlab-org/scalability |
|
| Issue Trackers | Scalability | |
| Slack Channels | #g_scalability / @scalability-team |
#infrastructure-lounge (Infrastructure Group Channel), #incident-management (Incident Management), #alerts-general (SLO alerting), #mech_symp_alerts (Mechanical Sympathy Alerts) |
The Scalability team is responsible for GitLab and GitLab.com at scale, working on the highest priority scaling items related to the application and GitLab.com. The team works in close coordination with Reliability Engineering teams and provides feedback to other Engineering teams so they can become better at scalability as well.
As its name implies, the Scalability team enhances the availability, reliability and, performance of GitLab by observing the application's capabilities to operate at GitLab.com scale. The Scalability team analyzes application performance on GitLab.com, recognizes bottlenecks in service availability, proposes (and develops) short term improvements and develops long term plans that help drive the decisions of other Engineering teams.
Short term goals include:
The Infrastructure Department is concerned with the availability and performance of GitLab.com.
GitLab.com's service level availability is visible on the SLA Dashboard, and we use the General GitLab Dashboard in Grafana to observe the service level indicators (SLIs) of apdex, error ratios, requests per second, and saturation of the services.
These dashboards show lagging indicators for how the services have responded to the demand generated by the application.
We are working on a maturity model for our services. The current iteration of this model, along with values for the services we've evaluated so far, is in a Google sheet (GitLab-internal).
We have a leading indicator of application demand. This is shown on the Application Demand Dashboard.
The dashboard is reviewed monthly as part of the triage rotationand our observations are noted on the indicator review issue.
Application demand is used as a way to identify that there may have been an application level change that alters how the application makes requests of the underlying infrastructure. If we observe a change in rate, this tells us that we need to investigate and understand the cause of the change.
The following people are members of the Scalability Team:
| Person | Role |
|---|---|
| Rachel Nienaber | Engineering Manager, Scalability |
| Jacob Vosmaer | Staff Backend Engineer, Scalability |
| Sean McGivern | Staff Backend Engineer, Scalability |
| Bob Van Landuyt | Senior Backend Engineer, Scalability |
| Craig Miskell | Site Reliability Engineer, Scalability |
| Quang-Minh Nguyen | Senior Backend Engineer, Scalability |
The following members of other functional teams are our stable counterparts:
| Person | Role |
|---|---|
| Marin Jankovski | Senior Engineering Manager, Infrastructure, Delivery & Scalability |
| André Luís | Frontend Engineering Manager, Create:Source Code, Create:Code Review, Delivery & Scalability |
We work with all engineering teams across all departments as a representative of GitLab.com as one of the largest GitLab installations, to ensure that GitLab continues to scale in a safe and sustainable way.
The Memory team is a natural counterpart to the Scalability team, but their missions are complementing each other rather than overlap:
| Scalability Team | Memory Team |
|---|---|
| Focused on GitLab.com first, self-managed only when necessary. | Focused on resolving application bottlenecks for all types of GitLab installations. |
| Driven by set SLO objectives, regardless of the nature of the issue. | Focused on application performance and resource consumption, in all environments. |
| Primary concern is preventing disruptions of GitLab.com SLO objectives through changes in the application architecture. | Primary concern is managing the application performance for all types of GitLab installations. |
Simply put:
workflow labels to the issue. The team will triage the issue and apply these.Alternatively, mention us in the issue where you'd like our input.
When issues are sent out way, we will do our best to help or find a suitable owner to move the issue forward. We may be a development team's first contact into the Infrastructure department and we endeavour to treat these requests with care so that we can help to find an effective resolution for the issue.
If you're working on a feature that has specific scaling requirements, you can create an issue with the review request template. Some examples are:
This template gives the Scalability team the information we need to help you, and the issue will be shown on our build board with a high priority.
A few weeks after a review has been closed, a follow-up comment is added to ask for feedback in a survey. This helps us understand if the process has been helpful and if there are improvements that can be made. We go through the feedback provided in the surveys each quarter.
This process is an example of doing something that doesn't scale; as we do more of these, we'll learn what topics can be covered more efficiently by training, documentation, and tooling.
When we observe a situation on GitLab.com that needs to be addressed alongside a stage group, we first raise an issue in the Scalability issue tracker that describes what we are seeing. We try to determine if the problem lies with the action the code is performing, or the way in which it is running on GitLab.com. For example, with queues and workers, we will see if the problem is in what the queue does, or how the worker should run.
If we find that the problem is in what the code is doing, then we engage with the EM/PM of that group to find the right path forward. If work is required from that group, we will create a new issue in the gitlab-org project and use the Availability and Performance Refinement process to highlight this issue.
We prefer to work asynchronously as far as possible but still use synchronous communication where it makes sense to do so.
To that end, we have very few scheduled calls.
Lastly, in order to keep people connected, team members schedule at least one coffee-chat with another team member each week. These are at times that will best suit them as it may be an unusual hour given the various timezones and working hours for each person.
As a small team covering a wide domain, we need to make sure that everything we do has sufficient impact. If we do something that only the rest of the Scalability team knows about, we haven't 'shipped' anything. Our 'users' in this context are the infrastructure itself, SREs, and Development Engineers.
Impact could take the form of changes like:
In order to make others aware of the work we have done, we should advertise changes in the following locations:
#development#eng-managers#dev_tip_of_the_day#development-guidelines#infrastructure-lounge#infra-staffDocumentation or tutorial videos should also be added to the README.md in our team repository.
We use Epics, Issues, and Issue Boards to organize our work, as they complement each other.
The single source of truth for all work is Scaling GitLab.com epic. This is considered as the top-level epic from which all other epics are derived.
Epics that are added as children to the top-level epic are used to describe projects that the team undertakes.
Having all projects at this level allows us to use a single list for prioritization and enables us to prioritize work for different services alongside each other. Projects are prioritized in line with the OKRs for the current quarter.
Project status is maintained in the description of the top-level epic so that it is visible at a glance. This is auto-generated. You can watch a short demo of this process to see how to use status labels on the epics to make use of this automation.
Example organization is shown on the diagram below:
Note If you are not seeing the diagram, make sure that you accepted all cookies.
The epic for the project must have the following items:
exit criterion label in the epic and are linked in the description.The Scalability team issue boards track the progress of ongoing work.
On the planning board, the goal is to get issues into a state where we have enough information to build the issue.
However, not all issues that are workflow-infra::Ready to be built should be scheduled for development right away. Some
issues may be too big, or have too low a priority. This means not all issues that are workflow-infra::Ready on the
planning board will move to the build board immediately.
Please see the triage rotation section for when to move issues between the boards.
| Planning | Building |
|---|---|
| Planning Board | Build Board |
| Issues where we are investigating the work to be done. | Issues that will be built next, or are actively in development. |
   |
    |
The Scalability team routinely uses the following set of labels:
team::Scalability.Scalability::.workflow-infra labels.Service labels.The team::Scalability label is used in order to allow for easier filtering of
issues applicable to the team that have group level labels applied.
The priority labels allow us to track the issues correctly and raise/lower priority of work based on both external and internal factors.
This means that the highest priority is given to working on issues that improve Gitlab.com SLO's either immediately and directly, or by unblocking other issues to achieve the same.
The Scalability team leverages scoped workflow labels to track different stages of work. They show the progression of work for each issue and allow us to remove blockers or change focus more easily.
The standard progression of workflow is from top to bottom in the table below:
| State Label | Description |
|---|---|
|
Problem is identified and effort is needed to determine the correct action or work required. |
|
Proposal is created and put forward for review. SRE looks for clarification and writes up a rough high-level execution plan if required. SRE highlights what they will check and along with soak/review time and developers can confirm. If there are no further questions or blockers, the issue can be moved into "Ready". |
|
Proposal is complete and the issue is waiting to be picked up for work. |
|
Issue is assigned and work has started. While in progress, the issue should be updated to include steps for verification that will be followed at a later stage. |
|
Issue has an MR in review. |
|
MR was merged and we are waiting to see the impact of the change to confirm that the initial problem is resolved. |
|
Issue is updated with the latest graphs and measurements, this label is applied and issue can be closed. |
There are three other workflow labels of importance:
| State Label | Description |
|---|---|
 |
Work in the issue is being abandoned due to external factors or decision to not resolve the issue. After applying this label, issue will be closed. |
 |
Work is not abandoned but other work has higher priority. After applying this label, team Engineering Manager is mentioned in the issue to either change the priority or find more help. |
 |
Work is blocked due external dependencies or other external factors. Where possible, a blocking issue should also be set. After applying this label, issue will be regularly triaged by the team until the label can be removed. |
The Scalability team uses priority labels as a means to indicate order under which work is next to be picked up. Priorities are roughly defined as:
| Priority level | Definition |
|---|---|
| Scalability::P1 | Issue is blocking other team-members, or blocking other work. As soon as possible after completing ongoing task unless directly communicated otherwise. |
| Scalability::P2 | Issue has a large impact, or will create additional work. |
| Scalability::P3 | Issue should be completed once other urgent work is done. |
| Scalability::P4 | Default priority. A nice-to-have improvement, non-blocking technical debt, or a discussion issue. |
Stage groups use throughput labels to label merge requests in projects in the gitlab-org group. The Scalability team is not a part of the stage groups, and labels of importance for the team are explained above. When submitting work in gitlab-org group, we apply ~"team::Scalability" and ~"feature::maintenance" to merge requests by default. The latter label is describing work towards refinement of existing functionality which describes majority of the work the team is contributing.
We have automated triage policies defined in the triage-ops project. These perform tasks such as automatically labelling issues, asking the author to add labels, and creating weekly triage issues.
We rotate the triage ownership each month, with the current triage owner responsible for picking the next one (a reminder is added to their last triage issue).
When issues arrive on our backlog, we should consider how they align with our vision, mission, and current OKRs.
We need to effectively triage these issues so that they can be handled appropriately. This means:
When handing over an issue to the new owner, provide as much information as you can from your assessment of the issue.
The Infradev process aims to highlight SaaS availability and reliability improvements with the Stage Groups.
Where issues marked as infradev are found to be scaling problems, the team::Scalability label should be added.
Our commitment to this process, in line with the team's vision, is to provide guidance and assistance to the stage groups who are responsible for resolving these issues. We proactively assist them to determine how to resolve a problem, and then we contribute to reviewing the changes that they make.
workflow-infra::Triage issues should be checked to make sure the idea is valid. Move the conversation forward
and ask for clarification where needed. Do what you can to add more details to develop a proposal.workflow-infra::Proposal issues should be checked to make sure that the proposal is complete. The benefit of the
work should be clear, it should contain a breakdown of tasks, and include steps to validate the outcome.workflow-infra::Blocked issues should be checked to see if blockers have been removed.workflow-infra::Ready issues should be moved to the Build Board using the following guidelines:
Scalability::P1 or Scalability::P2 - then move it to the build board, elseService::Unknown refinement - go through issues marked Service::Unknown and add a defined service, where possible.workflow-infra::In Progress, either
through picking them up directly, or asking on our team channel if any one else is able.team::Scalability and infradev labels so we can help the stage groups move those forward.We work from our main epic: Scaling GitLab on GitLab.com.
Most of our work happens on the current in-progress sub epic. This is always prominently visible from the main epic's description.
When choosing something new to work on you can either:
Ready column
according to the priority labels.or
The Scalability team became a reality during the fourth organizational iteration in the Infrastructure department on 2019-08-22, although it only became a reality once the first team member joined the team on 2019-11-29.
Even though it might not look like it at first glance, the Scalability team has its origin connected to the Delivery team. Namely, the first two backend engineers with Infrastructure specialisation were a part of the Delivery team, a specialisation that previously did not fit into the organizational structure. They had a focus on reliability improvements for GitLab.com, often working on features that had many scaling considerations. A milestone, that will prove to be a case for the Scalability team, was Continous Delivery on GitLab.com.
Throughout July, August and September 2019, GitLab.com experienced a higher than normal amount of customer facing events. Mirroring delays, slowdowns, vertical node scaling issues (to name a few) all contributed to general need to improve stability. This placed higher expectations on the Infrastructure department and with the organization at the time, this was harder to meet. To accelerate the timelines, "infradev" and "rapid action" processes were created, as a connection point between Infrastructure and Development departments to help Product prioritise higher impact issues. This approach was starting to yield results, but the process was there as a reaction to an (ongoing) event with the focus on resolving that specific need.
The background processing architectural proposal clearly illustrated the need to stay ahead of the growing needs of the platform and approach the growth strategically as well as tactically. With a clear case and approvals in hand, the team mission, vision, and goals were set and the team buildout could commence. While that was in motion, we had another confirmation through a performance retrospective that the need for the team is real.
As the team was taking shape, the background processing architectural changes were the first changes delivered by the team with a large impact on GitLab.com, with many more incremental changes throughout 2020 that followed. Measuring that impact reliably, and predicting the future challenges remains one of the team focuses at the time of the writing of this history summary.
The team impact overview is logged in issues:
