Delivery Team

1. You are here:
2. Engineering
3. Infrastructure
4. Team
5. Delivery Team

Maintained by:

On this page

• Common Links
• Mission
• Vision
  • Short-term
  • Mid-term
  • Long-term
• Strategy
• Top-level Responsibilities
• Team
  • Team Members
• Team counterparts
• Performance indicators
• How we work
  • Project Management
  • Epics
  • Issue Boards
  • Labels
    • Workflow
    • Priority Labels
    • Other Labels
  • Choosing something to work on
• Project demos
  • Team training
• History

Common Links

Mission

The Delivery Team enables GitLab Engineering to deliver features in a safe, scalable and efficient fashion to both GitLab.com and self-managed customers. The team ensures that GitLab's monthly, security, and patch releases are deployed to GitLab.com and publicly released in a timely fashion.

Vision

By its own nature, the Delivery team is a backstage, non-user feature facing team whose product and output has a direct impact on Infrastructure's primary goals of availability, reliability, performance, and scalability of all of GitLab's user-facing services as well as self-managed customers. The team creates the workflows, frameworks, architecture and automation for Engineering teams to see their work reach production effectively and efficiently.

The Delivery team is focused on our CI/CD blueprint by driving the necessary changes in our software development processes and workflows, as well as infrastructure changes, to embrace the benefits of CI/CD.

Short-term

• Automate release generation, removing most of the manual work
• Automate deployment process, managing and limiting impact on production
• Simplify security releases
• Streamline the process of limiting feature impact in production environments
• Enable feature testing at production-environment scale
• Create detailed architecture blueprints and design for CD on GitLab.com
• Develop and track KPIs to measure the team's impact on GitLab product delivery

Mid-term

• Drive the implementation of infrastructure changes to prepare GitLab.com for CD
• Eliminate the need for feature-freeze blackouts during the development cycle
• Shorten build times to allow for faster release times

Long-term

• Drive necessary changes that will lead to Kuberentes-based infrastructure on GitLab.com
• Fully automated releases for self-managed users

Strategy

The Delivery team significantly contributes to the Infrastructure department direction for FY22 in the following ways:

1. Performance
   • By the end of Q2FY22: Complete the migration of all stateless services to the Kubernetes platform.
   • By the end of Q4FY22: Define, plan and execute on migration of stateful services to the Kubernetes platform.
   • By the end of Q2FY22: Define deployment SLO, and evaluate its impact on Infrastructure performance indicators.
2. Reliability
   • By the end of Q3FY22: Ensure predictable and reliable rollback of stateless services.
3. External contributions
   • By the end of Q4FY22: Convert at least four release workloads to GitLab features, thus enabling external contributions to the features.

Top-level Responsibilities

The team regularly works on the following tasks, in the order of priority:

1. Ensuring continuous delivery of GitLab application software to GitLab SaaS.(e.g. GitLab SaaS auto-deploy)
2. Coordination and preparation of GitLab releases for self-managed users for the monthly patch and security releases.
3. Participating in incident resolution and acting on corrective actions for SaaS and self-managed software delivery.
4. Increasing velocity of both SaaS and self-managed software delivery through foundational project work (e.g. Running GitLab SaaS on Kubernetes).
5. Improving the robustness of SaaS software delivery by creating and improving tooling (e.g. Deployment rollback).
6. Minimizing the use of custom tooling by building or enhancing features within GitLab (e.g. Create a Changelog feature).
7. Support other teams' needs related to software delivery on GitLab SaaS (e.g. New Container Registry deployment).

Team

Each member of the Delivery team is part of this vision:

• Each team member is able to work on all team projects
• The team is able to reach a conclusion independently all the time, consensus most of the time
• Career development paths are clear
• The team creates a database of knowledge through documentation, training sessions and outreach

Team Members

The following people are members of the Delivery Team:

Team counterparts

The following members of other functional teams are our stable counterparts:

Performance indicators

Delivery team contributes to Engineering function performance indicators through Infrastructure department performance indicators. The team's main performance indicator is Mean Time To Production (MTTP), which serves to show how quickly a change introduced through a Merge Request is reaching production environment (GitLab.com). At the moment of writing, the target for this PI is defined in this key result epic.

MTTP is further broken down into charts and tables at the Delivery Team Performance Indicators Sisense dashboard.

How we work

Project Management

The Delivery team work is tracked through number of epics, issues, and issue boards.

Epics and issue boards are complementary to each other, and we always strive to have a 1-1 mapping between a working epic and an issue board. Epics describe the work and allows for general discussions, while the issue board is there to describe order of progress in any given epic.

Epics

Two tracking epics related to the team mission are:

1. GitLab.com on Kubernetes
2. Release Velocity

Any working epic that the team creates should be directly added as a child to one of these two top level tracking epics.

Working epic should always have:

1. A Problem Statement.
2. Directly responsible individuals responsible for the project completion.
3. Defined exit criteria
4. Issue admin section to provide the issue priority, labels and epic for quick actions. Example.
5. Status YYYY-MM-DD to indicate what is being worked on, why, and planned next steps. The DRI is responsible for updating the epic status each week. Note, this must be the last heading in the epic to support automated epic summary updates
6. Start date and estimated due date
7. Labels:
   • 
   • Label used as part of the project scope (eg. kubernetes, security-release).
   • Epic status label using the 'workflow-infra::triage', 'workflow-infra::proposal', 'workflow-infra::in-progress', 'workflow-infra::done'

In cases where the work is tracked in a project in a different group outside of our canonical project location, we will create two epics for the same topic and state in the epic description which one is the working epic.

Issue Boards

Each working epic should be accompanied by an issue board. Issue boards should be tailored to the specific project needs, but at minimum it should contain the workflow labels shown on the workflow diagram.

Labels

The canonical issue tracker for the Delivery team is at gl-infra/delivery. Issues are automatically labeled if no labels are applied using the triage ops project. The default labels defined in the labeling library.

By default, an issue needs to have a:

1. Workflow Label - Default: workflow-infra::Triage
2. Priority Label - Default: Delivery::P4
3. Other Label - project or team management related label.

Workflow

The Delivery team leverages scoped workflow-infra labels to track different stages of work.

On the Planning board problems are discussed and scoped so that we have enough information to prioritise and implement the solution.

The typical workflow is described below:

Not every issue will be prioritised for building as soon as it is ready. Instead we manage a Build board with all workflow-infra::In Progress, and workflow-infra::Ready issues focused on the team's current goals.

The standard progression of workflow is described below:

There are three other workflow labels of importance omitted from the diagram above:

1. workflow-infra::Cancelled:
   • Work in the issue is being abandoned due to external factors or decision to not resolve the issue. After applying this label, issue will be closed.
2. workflow-infra::Stalled
   • Work is not abandoned but other work has higher priority. After applying this label, team Engineering Manager is mentioned in the issue to either change the priority or find more help.
3. workflow-infra::Blocked
   • Work is blocked due external dependencies or other external factors. After applying this label, issue will be regularly triaged by the team until the label can be removed.

Label workflow-infra::Done is applied to signify completion of work, but its sole purpose is to ensure that issues are closed when the work is completed, ensuring issue hygiene.

Priority Labels

The Delivery team uses priority labels to indicate order under which work is next to be picked up. Meaning attached to priorities can be seen below:

The team uses priority labels differently to the general issue triage priority definition in order to avoid ambiguity that comes with difference in timelines between Stage teams and Infrastructure teams. We have different timelines (release brings different expectations for Delivery), different DRI's (no PM for Delivery), and different importance (Blocked release means that no one can ship anything).

Other Labels

Some of the labels related to the team management are defined as:

1. onboarding - issues are related to granting access to team resources.
2. team-tasks - issues related to general team topics.
3. Discussion - meta issues that are likely to be promoted to a working epic or generate separate implementation issues.
4. Announcements - issues used to announce important changes to wider audience.

Project labels are defined as needed, but they are required unless the issue describes a team management task.

Choosing something to work on

The Delivery team generally has working epics assigned to specific owners who are responsible for bringing tasks into the Planning board and Build board to keep the project on track. However, anyone is welcome to pick up any tasks from the Build board regardless of which project it belongs to. If you want to work on something outside of your current project, feel free to contribute to any of the mid to lower priority labeled issues.

Project demos

As part of the project, we might decide to organize project demo's. The decision on creating a demo depends on the expected longevity of the project, but also on the complexity of it.

The purpose of the demo is to ensure that everyone who participates in the project has a way of sharing their findings and challenges they might be encountering outside of the regular async workflow. The demo's do not have presentations attached to it, and they require no prior preparation. The demoer shouldn't feel like they have to excuse themselves for being unprepared, and expect that their explanation without faults. In fact, if what is being demoed is showing off no weaknesses, we might have not cut scope in time.

It is encouraged to show and discuss:

1. The imperfectness of the specific code implementation.
2. How broken the used tool might be.
3. How an estabilished process breaks down.
4. How challenging a problem being resolved might be.

Team training

Every Delivery team member is responsible for sharing skills either through creating a training session for the rest of the team, or through paired work. See the page on team training for details.

History

The Delivery team officially came into existence on 2018-10-23. This was the culmination of a larger alignment that was happening throughout that year, exposed by the need to streamline releases for self-managed users and creating a better experience for GitLab.com users.

All throughout GitLab's existence, Release Management had been a monthly rotating role served by developers. The idea behind it was to keep developers close to the whole lifecycle of the software they create, and ensure that they automate their work. This worked well until the number of application changes, and developer tasks grew too large for anyone to handle as a secondary task. The event that indicated the need for a change was a near miss event near the end of 2017, when the first Release Candidate was deployed to GitLab.com just 2 days before the 22nd. That whole month was riddled with challenges, from release managers struggling to deliver their day to day development tasks and RM tasks, to multiple unsuccessful deployments to GitLab.com. Most importantly, this was a first indication that the company was growing and that the processes that worked previously, might need to change to accommodate the larger growth that was planned.

After some internal discussions, we entered 2018 with an attempt to work on process improvements, rather than changing everything in one go. We went from a monthly rotation to two month release manager rotation, started noting down spent time. Over the next several months we'd seen general stabilization of the process but it became apparent that spending 4 engineers time in Release Manager rotation was not getting us anywhere closer to improving the deployment process for GitLab.com, and with each developer we hired the task list grew bigger.

The initial discussion on what is in front of us to achieve Continuous Delivery on GitLab.com exposed a clear need for a team focused on this specific task.

With the team created, we set out to work on the release we designed, change the way we deploy to GitLab.com, merge GitLab Rails codebases and many more other tasks.

After a successful team onsite (aka Fast boot) where we executed on our tasks while being in the same room together for the first time, we announced the first step towards Continous Delivery on GitLab.com. This was a very large change that changed the deployment frequency from deploying from the default branch once per month (for the total of 4-6 deploys to include bug fixes), to taking commits from the default branch once per week.

The team focus then shifted to getting deployment time measured in hours, and migration of GitLab.com to Kubernetes.

Prior to 2020, the team impact overview was created in Slack, and in the years that followed the overview was logged in issues:

1. Year overview for 2020
2. Year overview for 2021