Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Infrastructure Git Workflow

On this page

Issue: gitlab-com/gl-infra/infrastructure#5276

Idea/Problem Statement

In order for CI/CD to automatically deploy the GitLab.com application and infrastructure to staging and production environments, there need to be controls in place to ensure that every state change is recorded, repeatable, and reversible. Luckily git gives us all of the tools that we need, if we design our workflow sensibly.

The problems that we need to solve are:

Design

This design is similar to Environment branches with GitLab flow. GitLab Flow

In the proposed design, master refers to the branch running in the staging environment, and production refers to the stable branch containing a tagged snapshot of master.

This model can be used for infrastructure (terraform/chef) changes, as well as application changes. The workflow should be usable both for omnibus and kubernetes deployments of the application.

Operational Considerations

Automation

CI/CD targets must be set up to deploy to the staging environment for the master branch, and the production environment for the production branch.

The process of deploying from staging to production should look like this:

git checkout production
git pull origin production
git merge master
git push origin production

It may be advantageous for the scripting to check and only deploy tagged versions to production.

The canary environment can be treated as a separate branch, or as the target of the CI/CD for production, with a manual step to propagate approved tags to production.

Monitoring

GitLab Jobs are used to push new tags and merges to master and production. Alerts are emailed out when these jobs fail.

It may be useful to set up additional alerting.

If possible, we should tag production monitoring tools with the time that each new version becomes active. This makes it much easier to see which change to the deployed verions caused a change in monitored statistics.

Additional Considerations

Any configuration files that are checked in to source control need to take into account that they cannot be specific to the production environment. Currently, environments are separated by putting their config files into a separate directory and manually changing to that directory before running the scripts. Any CI/CD scripts need to take into account which environment the script is running for, and which config files they are using.

Alternatives