Gitlab hero border pattern left svg Gitlab hero border pattern right svg

GitLab Releases

Epics: gitlab-org/release/epics

Introduction

This document is part of overall engineering release workflow.

Process overviews

Terminology

Going forward, patch release (MAJOR.MINOR.X) would describe release that is being created as a follow up to a set of releases made in the run-up to the 22nd of the month (MAJOR.MINOR.0). All RC and final MAJOR.MINOR.0 will be referred to as releases.

See also terminology overview.

Current release processes

Releases are a mix of manual tasks driven through a checklist and automated processes to create release artifact and deploy to GitLab.com environments.

The process can be broken down into:

The release turnaround time varies between 12 and 24 hours. In cases where an issue is discovered during any of the mentioned steps, we can have days of delay before the change is applied to environments. This overview does not include situations when GitLab environments require an urgent hot-patch, or security patch.

Few conclusions can be drawn from the above:

Proposal

To address the process above without starting from scratch, we should be looking into a plan that would remove most of the manual work first, and open up more time for us to work on the next step.

Immediate step

As an immediate step, we need to remove the need for most manual actions. Engineers tasked with release should only need to trigger the start of the process, and then have control over deployments that are user facing.

To do this, we need to leverage GitLab as a tool together with ChatOps.

Release would look as follows:

  1. Release is started by using a ChatOps command
  2. Command creates a pipeline that runs the following:
  3. Automatically creates the preparation MR for stable branches
  4. Automatically merges the MR if pipeline passed; If pipeline fails, pings the person that triggered the command
  5. Continues with syncing all remotes
  6. Creates and pushes tags to required projects
  7. Pipelines in the omnibus-gitlab create the package used for deploying to GitLab.com environments and push to a package repository; If pipeline fails, pings the person that triggered the command
  8. Queries the package server for availability of the package and automatically deploys to the staging server
  9. Creates the QA issue 1. Pings the person that triggered the command to inform that canary and production can be deployed
  10. Deployment can be done from the same pipeline as manual action

Follow up

Once the above is achieved, we will expand on each of the steps.

Few items worth considering:

  1. Collecting and displaying metrics related to release
  2. Full test suite runtime in GitLab product projects need to complete in maximum 30 of minutes
  3. CE to EE merges need to be automatically merged in 95% of the cases, requiring manual intervention in rest
  4. Package build needs to complete in maximum of 30 minutes
  5. Deployments on staging needs to be done within 5 minutes from package being available
  6. Deployment on staging needs fully deployed in maximum of 20 minutes. At first this time target should exclude the database migrations, and later expanded to include them
  7. Infrastructure needs to support items as defined in CICD Pipeline on GitLab.com