| Description | Location |
|---|---|
| Release orchestration | Link |
| Release documentation | Link |
| Release related tasks issue tracker | Link |
| Delivery team issue tracker | Link |
| Release manager schedule | Link |
| Maintenance Policy | Link |
The main priority of both types of releases is GitLab availability & security as an application running on both GitLab.com and for customers running GitLab in their own infrastructure.
With these two types of releases, GitLab Inc. has to balance at the same time the workflows normally found in SaaS companies with the ones found in companies that publish packaged software.
We follow a Continuous Deployment approach with the goal of deploying as frequently as possible to keep batch sizes small. This means that GitLab.com lives in the now with changes tracked using commit SHAs rather than semver versioning as used by self-managed installations.
The self-managed release is a collection of many successfully deployed auto-deploy
releases on GitLab.com. Users on GitLab.com therefore receive features and bug fixes earlier than
users of self-managed installations.
GitLab.com receives updates multiple times a day with new deployment branches currently created at 03:00, 06:00, 10:00, 14:00, 17:00 and 20:00 UTC from Monday through Friday.
Once a new branch is created, only commits that pass the CI tests are eligible for deployments ("green build"). This means that if specs are failing in gitlab-org/gitlab, the deployments cannot progress further.
Automated tasks in the release-tools project are setup to drive the next steps:
~"Pick into auto-deploy" (See Labels of importance).When a new package is built, it is automatically deployed to the canary stage of staging.gitlab.com and a set of automated QA end-to-end/integration tests are run. Note that two sets of blocking QA tests are executed, one targeting Staging-Canary and the other targeting Staging. This is designed to assist with exposing issues arising in mixed deployment environments, where multiple versions of GitLab components are deployed that share services such as the database.
After passing, the package is automatically deployed to the canary stage of gitlab.com, where canary testing can take place. This means that specific projects (like gitlab-org/gitlab) as well as a small amount of end user traffic will be using the new package. Like with deployment to staging-canary above, two sets of automated QA end-to-end/integration tests are run. One targets the canary stage of production, the other targets the main stage (to ensure the new and old code are still functional).
After 30 minutes in the canary stage of gitlab.com, and provided no new exceptions or alerts are reported, the package is considered to be ready for deployment to gitlab.com and staging.gitlab.com
The promotion to gitlab.com and staging.gitlab.com is triggered manually by the release managers. Ongoing S1 or S2 incidents, as well as C1 or C2 changes, will block this promotion
Each deployment will trigger a notification in the Slack channel #announcements. As part of the deployments, a QA issue is created in release/tasks issue tracker to inform the people who are involved in the process that their change is going through environments.
The self-managed release is a semver versioned package of features that
are already released on GitLab.com through auto-deploys.
This means that the official published version of GitLab is a historical
snapshot of items that are released to users on GitLab.com. As such,
the self-managed release is created from a backport branch named by the
targeted semver version with a stable suffix, eg. 12-3-stable.
Our maintenance policy describes in detail the cadence of our major, minor and patch releases for self-managed users. The major release yearly cadence was defined after an all stakeholder discussion.
The self-managed release timelines are concentrated around the 22nd. One week
before the release date, the release managers will start preparing for the release by ensuring that the GitLab.com releases are in a consistent state. This means
that a self-managed release will only contain code that is successfully running
on GitLab.com at the time the release manager decides to finalise the release.
In the days leading up to the 22nd, the release manager will post announcements in #releases, #development, #backend, and #frontend update on candidate and guaranteed commits for the release.
Merge Requests that have been included in the monthly release will receive a label indicating inclusion.
The security of the software we ship must be everyone's responsibility as is the quality. It can be a daunting task to deliver quality software securely. GitLab has processes in place to help with application and code security reviews through the Security Departments' Application Security (AppSec) team. If you're unsure about the security implications of an addition to a code-base, the AppSec team can be included (@gitlab-com/gl-security/appsec) in the review to help.
The only guaranteed date throughout the release cycle is the 22nd. On this date,
the self-managed release will be published together with the release announcement.
All other dates are a guideline only and cannot be considered a deadline when it comes to what will be included into any type of release. This includes the development month and the dates defined there as well as any promises given to customers. This is strictly because there are a lot of moving parts to be considered when preparing a release which includes working on highest priority and severity issues as well as security related issues.
If it is absolutely necessary to get a certain feature
ready for a specific version, merge the feature early in the development cycle.
Merges closer to the release date are absolutely not guaranteed to be included
in that specific monthly self-managed release.
For GitLab.com releases, timelines are different and described below.
Anyone can block or halt a deployment by:
In addition, automated deployments to any production environment (including canary), are halted during the change lock period. Currently, the change lock period is:
During the change lock period, manual deployment can be triggered through GitLab ChatOps if the deployment fixes a severity::1 availability or security issue.
Deployments to production will be blocked by the following events:
Release Managers may decide, with input from the EOC to override a block and continue with the deployment.
For both types of releases, there are a few labels of specific importance.
For code that needs to be deployed to GitLab.com with higher priority than the
regular cadence, we have ~"Pick into auto-deploy" label.
The automated systems that create a new GitLab.com release will look for this label specifically, and any merge request with this label and a severity::1/severity::2 severity label will be automatically cherry-picked into the active auto-deploy branch. In case the merge request cannot be picked, which can happen if there is a conflict in the files being picked, the message will be posted in the merge request asking the author to create a new merge request targeting the currently active release branch.
The label should be only used under the following circumstances, when the merge request:
If this label is added because a merge request is blocking further deploys, consider leaving a note in #releases Slack channel to raise awareness of the status.
For new features or non-urgent fixes, the label should not be used because the new release is only days or hours away.
Directions on how to know whether a MR is deployed to GitLab.com are in the release/docs.
Similar to the above mentioned label, each self-managed release has a label
to highlight that a certain merge request should be backported to the targeted
release. For example, releases in the 12.3 series will have ~"Pick into 12.3"
label which will signal that this merge request should be included in one of the
next patch releases being created. The patch releases are created as needed but only according to our maintenance policy.
The label should be applied in the following situations:
The label should not be applied to merge requests:
As a merge request is included in monthly self-managed release candidates (RC), it will receive the
released::candidate label when the release candidate it is included in is deployed to
pre.gitlab.com. See "What is a release candidate and when are they created?"
for more information on release candidates. This label indicates that the MR will most
likely be included in the upcoming Self-managed release.
A merge request will receive the released::published label (which replaces the released::candidate label)
when included in a packaged release, such as 13.6.0 or 13.5.2, and deployed to
release.gitlab.net for both automated and manual testing. This label indicates that the
MR will be published with the Self-managed release.
You can find this out by taking a look at the GitLab Release Managers schedule.
A release candidate (RC) is a GitLab package that includes the changes that will make it into the final self-managed release, except for the rare case where a change may need to be reverted. RCs are only created for the monthly self-managed release, not patch releases. The amount of RCs created per month will vary per release.
There is no fixed point in time where a release manager creates a release candidate. Instead, this is based entirely on how the release process has been going, what the state is of GitLab.com, etc.
Release candidates are created whenever possible, and as such there are no guarantees on creation timing. This will depend on factors such as:
In other words, if you want to know when a release candidate is created your best option is to join one of the following Slack channels:
Release candidates are deployed to pre.gitlab.com for both automated and manual testing.
It is up to a release manager to decide when to create a release candidate, taking into account the state of deployments and GitLab.com.
Please do not message a release manager in private about release related questions or requests. Instead, post your request/question in the #releases channel.
When we create release candidates, and when we create a new release package, the Release Tools project will add a label to the merge requests included.
For more information, refer to the Auto-deploy status command.
In the runup to the 22nd, release managers will also start announcing what commit will at the very least make it into the release. Such notifications are shared in Slack #releases channel and will look something like this (format is defined in the release-tools monthly template):
This is the candidate commit to be released on the 22nd. https://gitlab.com/gitlab-org/gitlab/commits/4033cb0ed66de77a4b5c1936e33de918edef558e
The last commit to make it into the release will have a message similar to this:
:mega: Barring any show-stopping issues, this is the final commit to be released on the 22nd. https://gitlab.com/gitlab-org/gitlab/-/commits/13-1-stable-ee
The earlier in the monthly cycle your MR is merged, the higher the chances are for it to be included in that month's release.
There is no guaranteed "cut-off", "freeze" or any other date defined under which the MR will be included.
Availability, security and performance of GitLab.com is a pre-requisite for any monthly self-managed release. If GitLab.com is not experiencing any issues, MR's merged as late as the 20th of the month were included in the release. On the opposite side, when GitLab.com stability was lower, MR's merged as early as 15th of the month were not included.
In other words:
The quality and stability of what is delivered by everyone defines the final MR that will be included in the monthly release.
For more detailed answer, see self-managed release timelines.
We currently create an auto-deployment branch at a specific timeline. The current Mean Time To Production time and target can be seen on https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#mean-time-to-production-mttp
For a merge request with a specific label, the process is different.
See this guide for more information.
If you've found a regression with a potentially high severity, immediately follow the steps in Deployment blockers to halt the deployment.
If a regression is found in a new feature, and only that feature is affected, follow the directions in the QA issue created in the release/tasks project for a regular regression.
For high severity bugs found in the lead up to the 22nd of the month please also alert the Release Managers in #releases.
The different processes are documented here:
See the Security Release process as Developer documentation for more information.
Also, make sure to see Security Releases How to video for a broad explanation about all the steps required as a Developer when working on a security fix.
Security issues created on GitLab Security need to be associated with the Security Release Tracking issue for them to be included on the Security Release. Make sure to use the security issue template and follow the steps there.
Besides the merge request targeting master, three backports will be needed targeting the last two monthly releases and the current release.
For more information, see security release backports.
Any high severity issue should start with an issue labelled with the appropriate bug and severity labels.
Depending on the bug details, follow one of the following processes:
Once a security merge request has been merged, it's not advisable to revert it for multiple reasons:
If a security vulnerability introduced a bug, in most cases, the appropriate path forward is to fix the issue in the canonical repository (after the security release has been published).
If a security vulnerability introduced a high severity bug, engage with AppSec and release managers to coordinate next steps.
If you need any additional help please ask the Release Managers in the #releases Slack channel.