Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Change Management

Context and Objectives

Change Management has traditionally referred to the processes, procedures, tools and techniques applied in IT environments to carefully manage changes in an operational environment: change tickets and plans, approvals, change review meetings, scheduling, and other red tape.

In our context, Change Management refers to the guidelines we apply to manage changes in the operational environment with the aim of doing so (in order of highest to lowest priority) safely, effectively and efficiently. In some cases, this will require the use of elements from traditional change management; in most cases, we aim to build automation that removes those traditional aspects of change management to increase our speed in a safe manner.

Our overriding objective is maximize changes that avoid traditional aspects of change management, which is an iterative process that will evolve over time. Success is measured by our ability to safely execute changes at the speed required by our business needs.

Changes

Changes are defined as any voluntary, scheduled modification to the operational environment.

Deployments are a special change metatype depending on their scope and the effect they may have on the environment, as defined above. As we make progress towards CI/CD, we aim to turn all deployments into simple service changes.

Changes that need to be performed during the resolution of an Incident fall under Incident Management.

Operational Environments are currently defined as GitLab.com and ops.GitLab.net as well as supporting systems such as Prometheus.

Trust

Change Management is underpinned by trust: we trust ourselves to act responsibly when working in the operational environment and do everything in our power to safeguard its integrity and maintain its availability and performance.

To that end, we must develop a practice through automation that allows for safe changes. Our highest priority is always the integrity and reliability of the operational environment, which entails appropriate risk evaluation, quantifiable validation and verification, extensive communication, detailed auditing, and a focus on defensive coding.

Anatomy of a Change

At its core, changes implement the transitions that take the operational environment from its current state to a new state that yields the desired configuration. It is critical that we have a full and concrete understanding of both states, as they determine how the transition is planned and executed.

Change always begins with the current state of the environment, which is defined by both its actual state and the assumptions made about it. Predicting the ways in which a change will fail or yield undesirable results is, at best, an extremely difficult undertaking (if at all possible); it is outright impossible when our assumptions do not match reality, which is one of the primary sources of change failure, as the change will behave in unexpected ways under uncertain conditions, likely resulting in an incident.

All changes, without exception, regarless of complexity and whether they require a change plan or not, must outline, implement and codify pre-flight checks and post-verification checks. The automation's primary goal is to protect the environment. When exceptions are necessary, the author should document why this is the case, and the reviewer should approve said exception.

In general, three rules are critical to the successful execution of a change:

Pre-flight Checks

Pre-flight checks protect the operational environment by validating assumptions about the current state of the environment and provide the necessary gates to the execution of any change. If any assumption is proven to be incorrect through these checks, a change should either be halted or take corrective action: under no circumstances should a change proceed as originally planned when pre-flight checks are not successful.

Pre-flight checks should be coded into the automation and executed as the prerequisite step of a change. In essence, we are asking the question what must be true for the change at hand to execute successfully?

In determining what these checks should entail, we must focus on the assumptions implicit in the change. There are many examples of how to defensily protect the environment:

While pre-flight checks will likely have an effect on the speed of a change (both on our speed to implement changes and on the runtimes), failure has a far greater impact, especially on our users. When working on Infrastructure changes, safety is always prioritized over velocity.

Post-verification Checks

Much in the way we must validte the current state of the environment and the assumptions we make about it before a change, we must always validate the new, post-change state and the assumptions made about it. We must explicitly quantify the desired results of the change.

As examples:

Chage Priorities

Change Management helps us prioritize our resources towards changes that need to be made more resilient through defensive automation. Priorities are driven by two factors:

In these situations, we will focus on developing the necessary automation and safeguards to help teams and services move towards safe service changes in a timely fashion. Until then, all changes that fall under the two aforementioned categories are treated as maintenance changes.

Change Severities

Change severities encapsulate the risk associated with a change in the environment. Said risk entails the potential effects if the change fails and becomes an incident. Change management uses our standarized severity definition, which can be found under out which can be found under issue workflow documentation.

Change Plans

All changes should have change plans. Planning is the way the infrastructure department assesses and mitigates the risks changes introduce. They generate awareness and are the focal point for scheduling, communicating, and recording changes.

Change Request Workflows

Plan issues are opened in the production project tracker. Each issue should be opened using an issue template for the corresponding level of criticality: C1, C2, C3, or C4. It must provide a detailed description of the proposed change and include all the requested information in the template. Every plan issue is initially labeled ~"change::unscheduled" until it can be reviewed and scheduled with a Due Date. After the plan is approved and scheduled it should be labeled ~"change::scheduled" for visbility.

Change Criticalities

Criticality 1

These are changes with high impact or high risk. If a change is going to cause downtime to the environment, it is always categorized a C1. Before implementing the change.

Examples of Criticality 1:

  1. Any changes to Postgres hosts that affects DB functionality - quantity of nodes, changes to backup or replication strategy
  2. Architectural changes to Infra as code (IaC)
  3. IaC changes to pets - Postgres, Redis, and other Single Points of Failure
  4. Changes of major vendor - CDN, mail, DNS
  5. Major version upgrades of tooling (HAProxy, Chef)

Approval

  1. Add a Due Date to the issue and to the GitLab Production calendar.
  2. Have the change approved by Reliability Engineering management.
  3. Identify the Engineer On-Call (EOC) scheduled for the time of the change and review the plan with them.
  4. Announce the start of the plan execution in the #production Slack channel and obtain a written approval from the EOC in both the issue and in Slack.
  5. Join The "Situation Room" zoom channel with the EOC and obtain verbal approval to start the plan execution.

The EOC must be engaged for the entire time the execution

Criticality 1 plan template

Criticality 2

These are changes that are not expected to cause downtime, but which still carry some risk of impact if something unexpected happens. For example, reducing the size of a fleet of cattle is usually ok because we've identified over-provisioning, but we need to take care and monitor carefully before and after.

Examples of Criticality 2:

  1. Load Balancer Configuration - major changes to backends or front ends, fundamental to traffic flow
  2. IaC changes to cattle / quantity when there is a decrease
  3. Minor version upgrades of tools or components (HAProxy)
  4. Removing old hosts from IaC (like removals of legacy infrastructure)

Approval

  1. Add a Due Date to the issue and an event to the GitLab Production calendar.
  2. Identify the Engineer On-Call (EOC) scheduled for the time of the change and review the plan with them.
  3. Announce the start of the plan execution in the #production Slack channel and obtain a written approval from the EOC in both the issue and in Slack.

Criticality 2 plan template

Criticality 3

These are changes with either no or very-low risk of negative impact, but where there is still some inherent complexity, or it is not fully automated and hands-off

Examples of Criticality 3:

  1. IaC changes to cattle / quantity when there is an increase (not requiring reboot or destroy/recreate)
  2. Changes in configuration for current systems serving customers related to DNS or CDN

Approval

  1. Add a Due Date to the issue.
  2. Identify the Engineer On-Call (EOC) scheduled for the time of the change and review the plan with them.

Criticality 3 plan template

Criticality 4

These are changes that are exceedingly low risk and commonly executed, or which are fully automated. Often these will be changes that are mainly being recorded for visibility rather than as a substantial control measure.

Examples of Criticality 4:

  1. Any procedural invocation such as a SQL script, a ruby script module, a rake task which is performed on a production console server, either using gitlab-rails or gitlab-rake.
  2. Any invocation of an existing code pathway which ultimately will perform any mutate operation on live data. This is distinguished from diagnostic investigation operations which should typically be limited to read-only operations. It is ostensibly left to the discretion of the engineer whether or not a peer should be included to co-observe the invocation of such diagnostics.

Approval

No approval required.

Criticality 4 plan template

Change Plans Summary

With change plans, we develop a solid library of change procedures. Even more importantly, they provide detailed blueprints for implementation of defensive automation. Adding on to the defensive automation, every change request that uses some sort of a script must have a dry-run capability, the script should be run in the dry-run mode and its output should be provided to the CR for review. Ideally, the planner and the executor should be different individuals.

Change Schedule

Please consider the timezone UTC as the standard for all the changes.

The following table has the original schedule for changes based on the criticality level of the component :

  10 PM - 6 AM 6 AM - 2 PM 2 PM - 10 PM
Criticality 1 ALLOWED NOT ALLOWED NOT ALLOWED
Criticality 2 ALLOWED NOT ALLOWED NOT ALLOWED
Criticality 3 ALLOWED ALLOWED ALLOWED
Criticality 4 ALLOWED ALLOWED ALLOWED

Please consider the time slots on the calendar Production, to add change requests to Criticality 1 and 2. The other criticalities please add direct to the calendar.

Change Execution

If the change is executed by a script, it should be run from the bastion host of the target environment in a terminal multiplexer (e.g. screen or tmux) session. Using a bastion host has the benefit of preventing any unintended actions (e.g. caused by a script bug) from spreading to other environments. A terminal multiplexer guards against the possibility of losing connection to the bastion mid-change and the unpredictable consequences of it.

sudo is disabled on the bastion hosts, so you can copy your Chef PEM file to one of them, if your script requires it, without fearing it being snooped on.

A sequence of actions to run a script could look like this:

your-workstation $ ssh -A bastion-01-inf-gstg
bastion-01-gstg  $ tmux
bastion-01-gstg  $ git clone git@gitlab.com:my-migration/script.git
bastion-01-gstg  $ ./script/migrate

Change Reviews

Maintenance changes require change reviews. The reviews are intended to bring to bear the collective experience of the team while providing a forum for pointing out potential risks for any given change. A minimun quorun of three reviewers is required to approve a ~S1 or ~S2 maintenance change.

Roles

Role Definition and Examples
EMOC Event Manager
  The Event Manager is the tactical leader of the change team. For service changes, the EMOC is the person executing the change. For maintenance changes, the EMOC is the person in the IMOC rotation. ~S1 and ~S2 changes require an EMOC.
CMOC Communications Manager
  The Communications Manager is the communications leader of the change team. The focus of the Change Team is executing the change as safely and quickly as possible. For ~S1 and ~S2 maintenance changes, a CMOC communicates with the appropriate stakeholders. Othersiwe, EMOC can handle communication.
CT Change Team
  The Change Team is primarily composed of technical staff perfoming the change.

Communication Channels

Information is a key asset during any change. Properly managing the flow of information to its intended destination is critical in keeping interested stakeholders apprised of developments in a timely fashion. The awareness that a change is happening is critical in helping stakeholders plan for said changes.

This flow is determined by:

For instance, a large end-user may choose to avoid doing a software release during a maintenance window to avoid any chance that issues may affect their release.

Furthermore, avoiding information overload is necessary to keep every stakeholder’s focus.

To that end, we will have: 

Production Change Lock (PCL)

While changes we make are rigorously tested and carefully deployed, it is a good practice to temporarily halt production changes during certain events such as GitLab LiveStream, GitLab Summit and days where LOA (leave of absence), due to holidays, is high in engineering teams. We categorize these special periods of times into two buckets:

  1. GitLab Events
  2. High LOA

Risks of making a production deployment during the said periods includes immediate customer impact and/or less engineering team coverage in case an incident occurs and has to be resolved immediately. Therefore, we have introduced a mechanism called Production Change Lock (PCL). We see the future of PCL as an automated process which, provided a time range, locks production deployments and releases the lock once the time expires. However, as the first iteration towards this future state we are starting with creating events on our Production Calendar so that teams are aware of the PCL periods.

The following dates are currently scheduled PCLs. Times for the dates below begins at 09:00 UTC and ends at 09:00 UTC.

Dates Type Reason
24 December 2019 Soft Holiday: Christmas Eve
25-26 December 2019 Hard Holiday: Christmas
27-31 December 2019 Soft High Number of SRE Vacations
01 January 2020 Hard Holiday: New Year's Day
21-28 March 2020 Hard GitLab Contribute
22nd of every month Soft Release day

There are 2 types of PCLs: soft and hard.

Soft PCL

Soft PCLs aim to mitigate risk without halting all changes to production. Soft PCLs prohibit infrastructure changes with a criticality level of 2 or higher. In case of an emergency, the EOC should interact with the IMOC for C1 and C2 changes.

During the soft PCL, code deployments to canary are allowed since we have tools to control canary impact. Production deployments are allowed for lower criticality items (C3/C4) in coordination with the EOC. These items include high priority code deployments (impactful bugs, security fixes).

Hard PCL

Hard PCLs include code deploys and infrastructure changes for every criticality level (see change severities). In case of an emergency, the EOC should interact with the IMOC prior to making any decision. It is at EOC and IMOC discretion to make a decision on whether a change should be approved and executed. If the change is approved, IMOC should inform the VPE of this decision.

Questions