1. You are here:
2. Handbook
3. Engineering
4. Infrastructure
5. Change Management

On this page

• Changes
  • Trust
    • Change Severities
  • Change Type
  • Change Plans
    • Criticality 1:
    • Criticality 2:
    • Criticality 3
    • Criticality 4:
    • Change plans summary
  • Change Schedule
  • Change Request Workflow
  • Change Execution
  • Change Reviews
  • Roles
  • Communication Channels
• Production Change Lock (PCL)
  • Questions

Changes

Changes are any modification to the operational environment and are classified into two types:

• Service changes are regular, routine changes executed through well-tested, automated procedures performed with minimal human interaction that may cause predictable and limited performance degradation and no downtime. As such, service changes do not require review or approval except on their very first iteration.
• Maintenance changes are possibly complex changes that require manual intervention beyond initiating the change and that will cause downtime or significant performance degradation by the nature of the change. These changes require strict scheduling, careful planning and review, and approval by the Director of Infrastructure for execution.

Deployments are a special change metatype depending on their scope and the effect they may have on the environment, as defined above. As we make progress towards CI/CD, we aim to turn all deployments into simple service changes.

Operational Environments are currently defined as GitLab.com and ops.GitLab.net.

Trust

GitLab.com is the premier GitLab instance on the planet, and a production instance in every sense of the word. Change Management's primary goal is to safeguard the integrity of the GitLab.com environment through increased predictability by providing a framework to drive all changes towards becoming service changes and to help us achieve an optimal change speed.

Change Management is underpinned by trust: we trust ourselves to act responsibly in the operational environment to maintain its integrity and, by extension, its availability and performance.

To that end, we are not instituting a blanket policy for changes. Rather, we are developing the foundation of what a service change is (risk evaluation, automatic auditing and communication, pre-flight checks, defensive coding, post-change validation) and will help teams with adoption.

Change Management helps us prioritize our resources towards changes that need to be made more resilient through defensive automation. Priorities are driven by two factors:

• changes that cause ~S1 or ~S2 incidents
• changes driven by services that are below a TBD error budget

In these situations, we will focus on developing the necessary automation and safeguards to help teams and services move towards safe service changes in a timely fashion. Until then, all changes that fall under the two aforementioned categories are treated as maintenance changes.

Change Severities

Change severities encapsulate the risk associated with a change in the environment. Said risk entails the potential effects if the change fails and becomes an incident. Change management uses our standarized severity definition, which can be found under out which can be found under issue workflow documentation.

• In order to minimize the number of variables at play, no changes are executed during an active incident.
• ~S1 and ~S2 changes are always serialized and executed exclusively (i.e., never concurrently).
• ~S3 and ~S4 changes are allowed to take place concurrently as long as there is awareness of said concurrency.
• The Infrastructure on-call resource has veto power over any and all changes.

Change Type

The change types can be one of the labels :

• ConfigurationChange - to deploy a configuration change.
• HotFix - to deploy a patch in a software that is production.
• DeploymentNewFeature - to deploy of a new feature.
• Operation - operational tasks, such as running data mutating scripts, one-off cleanup/restarts, and so on.

Change Plans

All changes should have change plans. While this is optional for service changes, it is mandatory for maintenance changes. Change Plans provide detailed description of proposed changes and include the following information depending on the criticality of the service:

Criticality 1:

These are changes with high impact or high risk. If a change is going to cause downtime to the environment, then it is implicitly a C1 (any rare exceptions to this rule will be obvious)

Examples of Criticality 1:

1. Any changes to Postgres hosts that affects DB functionality - quantity of nodes, changes to backup or replication strategy
2. Architectural changes to Infra as code (IaC)
3. IaC changes to pets - Postgres, Redis, and other Single Points of Failure
4. Changes of major vendor - CDN, mail, DNS
5. Major version upgrades of tooling (HAProxy, Chef)

Detailed steps for the change. For each step the following must be considered:

• pre-conditions for execution of the step - how to verify it is safe to proceed
• execution commands for the step - what to do
• post-execution validation for the step - how to verify the step succeeded

It is strongly recommended to:

• Note relevant graphs in grafana to monitor the effect of the change, including how to identify that it has worked, or has caused undue negative effects
• Review alerts that may go off that can be silenced pro-actively

Rollback steps

• As for the original steps. It may be acceptable to reference the change steps as the process, with variations (e.g. Revert commit and run deployment).
• It is acceptable to list a full rollback process, and allow for the applier to select where to start based on how far through they got.

Criticality 2:

These are changes that are not expected to cause downtime, but which still carry some risk of impact if something unexpected happens. E.g. reducing the size of a fleet of cattle is usually ok because we've identified over-provisioning, but we need to take care and monitor carefully before + after.

Examples of Criticality 2:

1. Load Balancer Configuration - major changes to backends or front ends, fundamental to traffic flow
2. IaC changes to cattle / quantity when there is a decrease
3. Minor version upgrades of tools or components (HAProxy)
4. Removing old hosts from IaC (like removals of legacy infrastructure)

Detailed steps for the change. For each step the following must be considered:

• pre-conditions for execution of the step - how to verify it is safe to proceed
• execution commands for the step - what to do
• post-execution validation for the step - how to verify the step succeeded

It is strongly recommended to:

• Note relevant graphs in grafana to monitor the effect of the change, including how to identify that it has worked, or has caused undue negative effects
• Review alerts that may go off that can be silenced pro-actively

Rollback steps

• As for the original steps. It may be acceptable to reference the change steps as the process, with variations (e.g. Revert commit and run deployment).
• It is acceptable to list a full rollback process, and allow for the applier to select where to start based on how far through they got.

Criticality 3

These are changes with either no or very-low risk of negative impact, but where there is still some inherent complexity, or it is not fully automated and hands-off

Examples of Criticality 3:

1. IaC changes to cattle / quantity when there is an increase (not requiring reboot or destroy/recreate)
2. Changes in configuration for current systems serving customers related to DNS or CDN

Detailed steps for the change. For each step the following must be considered:

• pre-conditions for execution of the step - how to verify it is safe to proceed
• execution commands for the step - what to do
• post-execution validation for the step - how to verify the step succeeded

It is strongly recommended to:

• Note relevant graphs in grafana to monitor the effect of the change, including how to identify that it has worked, or has caused undue negative effects
• Review alerts that may go off that can be silenced pro-actively

Rollback steps

• As for the original steps. It may be acceptable to reference the change steps as the process, with variations (e.g. Revert commit and run deployment).
• It is acceptable to list a full rollback process, and allow for the applier to select where to start based on how far through they got.

Criticality 4:

These are changes that are exceedingly low risk and commonly executed, or which are fully automated. Often these will be changes that are mainly being recorded for visibility rather than as a substantial control measure.

Examples of Criticality 4:

1. Any procedural invocation such as a SQL script, a ruby script module, a rake task which is performed on a production console server, either using gitlab-rails or gitlab-rake.
2. Any invocation of an existing code pathway which ultimately will perform any mutate operation on live data. This is distinguished from diagnostic investigation operations which should typically be limited to read-only operations. It is ostensibly left to the discretion of the engineer whether or not a peer should be included to co-observe the invocation of such diagnostics.

Detailed steps for the change. For each step the following must be considered:

• pre-conditions for execution of the step - how to verify it is safe to proceed
• execution commands for the step - what to do
• post-execution validation for the step - how to verify the step succeeded

It is strongly recommended to:

• Note relevant graphs in grafana to monitor the effect of the change, including how to identify that it has worked, or has caused undue negative effects
• Review alerts that may go off that can be silenced pro-actively

Rollback steps

• As for the original steps. It may be acceptable to reference the change steps as the process, with variations (e.g. Revert commit and run deployment).
• It is acceptable to list a full rollback process, and allow for the applier to select where to start based on how far through they got.

Change plans summary

With change plans, we develop a solid library of change procedures. Even more importantly, they provide detailed blueprints for implementation of defensive automation. Adding on to the defensive automation, every change request that uses some sort of a script must have a dry-run capability, the script should be run in the dry-run mode and its output should be provided to the CR for review. Ideally, the planner and the executor should be different individuals.

Change Schedule

Please consider the timezone UTC as the standard for all the changes.

The following table has the original schedule for changes based on the criticality level of the component :

Please consider the time slots on the calendar Production, to add change requests to Criticality 1 and 2. The other criticalities please add direct to the calendar.

Change Request Workflow

To drive a change request to be executed on production :

• Create an issue in the production project and select a template with the criticality of the change
• Add to the Production calendar in the timeslot desired or in the production calendar.
• The issue will reviewed by SRE team management and Staff.
• After getting the approval the change can be executed.

Change Execution

If the change is executed by a script, it should be run from the bastion host of the target environment in a terminal multiplexer (e.g. screen or tmux) session. Using a bastion host has the benefit of preventing any unintended actions (e.g. caused by a script bug) from spreading to other environments. A terminal multiplexer guards against the possibility of losing connection to the bastion mid-change and the unpredictable consequences of it.

sudo is disabled on the bastion hosts, so you can copy your Chef PEM file to one of them, if your script requires it, without fearing it being snooped on.

A sequence of actions to run a script could look like this:

your-workstation $ ssh -A bastion-01-inf-gstg
bastion-01-gstg  $ tmux
bastion-01-gstg  $ git clone git@gitlab.com:my-migration/script.git
bastion-01-gstg  $ ./script/migrate

Change Reviews

Maintenance changes require change reviews. The reviews are intended to bring to bear the collective experience of the team while providing a forum for pointing out potential risks for any given change. A minimun quorun of three reviewers is required to approve a ~S1 or ~S2 maintenance change.

Roles

Communication Channels

Information is a key asset during any change. Properly managing the flow of information to its intended destination is critical in keeping interested stakeholders apprised of developments in a timely fashion. The awareness that a change is happening is critical in helping stakeholders plan for said changes.

This flow is determined by:

• the type of information,
• its intended audience,
• and timing sensitivity.

For instance, a large end-user may choose to avoid doing a software release during a maintenance window to avoid any chance that issues may affect their release.

Furthermore, avoiding information overload is necessary to keep every stakeholder’s focus.

To that end, we will have: 

• a dedicated change bridge (zoom call) for S1 and S2 changes.
• a dedicated #change channel, since #production contains sizeable amounts of information and it takes effort to filter out non-relevant items. This is particularly important for the change team, which must be focused on technical information to perform the change. While #change is an open channel and anyone is free to join, we will encourage people to use other channels to communicate with the EMOC.
• periodic updates intended to the various audiences at place (CMOC handles this):
  • End-users (Twitter)
  • eStaff
  • Support staff
  • Employees at large
• a dedicated repo for issues related to Production separate from the queue that holds Infrastructures’s workload: namely, issues for incidents and changes. This is useful because there may be other teams, over time, that need to do work in the production environment.
• After the maintenance is complete - handoff notes to other region on call team members should be left. Including items like:
  • state / success of the maintenance
  • any alerts that can have been silenced and may go handoff
  • links to specific graphs to watch for areas of concern

Production Change Lock (PCL)

While changes we make are rigorously tested and carefully deployed, it is a good practice to temporarily halt production changes during certain events such as GitLab LiveStream, GitLab Summit and days where LOA (leave of absence), due to holidays, is high in engineering teams. We categorize these special periods of times into two buckets:

1. GitLab Events
2. High LOA

Risks of making a production deployment during the said periods includes immediate customer impact and/or less engineering team coverage in case an incident occurs and has to be resolved immediately. Therefore, we have introduced a mechanism called Production Change Lock (PCL). We see the future of PCL as an automated process which, provided a time range, locks production deployments and releases the lock once the time expires. However, as the first iteration towards this future state we are starting with creating events on our Production Calendar so that teams are aware of the PCL periods.

Questions

• Does production above include canary?

  Yes.

• Does this apply only to production environment?

  Yes. Only production environment. This means you can still make changes and deployments to pre-prod environments.

• What is the exact scope of the changes that are enforced under PCL? (infrastructure, software, handbook…etc)

  Any production change that would make it to gitlab.com.

• What if I still want to deploy during the PCL period?

  VPE will need to approve your change to be deployed to production during PCL period.

• Does this apply to our monthly release which happens on the 22nd?

  No.

• Why did you select these dates only?

  The above listed periods are immediate times we know we will have less engineering teams' coverages (because of holidays, vacations and events). This was a data-driven decision based on our engineering population density around the globe. We hope to have made a progress towards the automated, future state of PCL beyond May, 2019 and if we achieve our goal we would be managing these via systems rather than manual communications and calendar events.

• We have a question that is not answered here?

  Please raise an issue to Infrastructure team's queue and we will be happy to get back to you as soon as we can.