Blog Engineering GitLab's guide to safe deployment practices
Published on: July 23, 2020
3 min read

GitLab's guide to safe deployment practices

It's important to safeguard your deployment process. Here's our best advice to protect your environments.

safe_deploy.jpg

Here at GitLab we understand the importance of safe deployment practices.

Progressive delivery is continuous delivery with fine-grained control over who sees the change. This ensures that all code and configuration updates go through the CI/CD stages to catch any regressions or bugs before they reach customers. If something does make it past those gates, progressive delivery makes sure any negative impact is as small as possible.

We have recently added several features that add safeguards to your deployment process, which we will review in this blog post.

Protected Environments

It is important that deploy jobs are restricted to only those who are authorized to deploy in that environment, and we call this restriction by roles "protected". While this feature has been around for a while, it is important to remember that this should be the first step to take when thinking about safe deployments.

Take a deeper dive into protected environments.

Sequential Deployment (or Safe Continuous Deployment)

If your project follows the Continuous Deployment practice that deploys the master branch to the production environment with GitLab CI/CD pipelines, you may encounter the following problems due to the asynchronous nature of pipeline jobs:

  • Multiple deployment jobs run concurrently, targeting the same environment. This can make the environment unstable because the deployment script could conflict and finish in an incomplete state.
  • An older deployment job could overwrite the latest deployment, resulting in an unintentional rollback. Some users could be exposed to old feature sets on the production website even though the pipeline shows that the latest deployment job successfully finished.
  • A pipeline might deploy to production at the worst time, such as on a holiday or over the weekend, when there is limited staff available to solve potential problems.

To address these problems, GitLab provides the following options:

Limit job concurrency

You can limit deployment concurrency by adding a resource_group to any .gitlab-ci.yml jobs that should run one at a time. For example:

  • Pipeline-A starts running with SHA-A
  • Pipeline-B starts running with SHA-B (newer)
  • Pipeline-A starts a deployment
  • Pipeline-B waits for Pipeline-A's deployment to finish

Prevent deployment of old versions

The execution order of pipeline jobs can vary from run to run, which could cause undesired behavior. For example, a deployment job in a newer pipeline could finish before a deployment job in an older pipeline. This creates a race condition where the older deployment finishes later, overwriting the "newer" deployment.

You can ensure that older deployment jobs are cancelled automatically when a newer deployment runs by enabling the prevent deployment of old versions feature.

  • Pipeline-A starts running with SHA-A
  • Pipeline-B starts running with SHA-B (newer)
  • Pipeline-B finishes. Now SHA-B is on the production environment
  • Pipeline-A is canceled automatically because it was going to deploy SHA-A to production

Prevent deployment of old versions

Deployment Freeze

To prevent deployments for a particular period, such as during a planned holiday when most employees are out, you can set up a deploy freeze. During a deploy freeze, no deployments can be executed. This is helpful to ensure that deployments do not happen unexpectedly.

Find more detailed information about deployment safety.

Read more about GitLab and safety:

Cover image by Mathew Schwartz on Unsplash

We want to hear from you

Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum. Share your feedback

Ready to get started?

See what your team could do with a unified DevSecOps Platform.

Get free trial

Find out which plan works best for your team

Learn about pricing

Learn about what GitLab can do for your team

Talk to an expert