Changes

Changes are any modification to the operational environment and are classified into two types:

Service changes are regular, routine changes executed through well-tested, automated procedures performed with minimal human interaction that may cause predictable and limited performance degradation and no downtime. As such, service changes do not require review or approval except on their very first iteration.
Maintenance changes are possibly complex changes that require manual intervention beyond initiating the change and that will cause downtime or significant performance degradation by the nature of the change. These changes require strict scheduling, careful planning and review, and approval by the Director of Infrastructure for execution.

Deployments are a special change metatype depending on their scope and the effect they may have on the environment, as defined above. As we make progress towards CI/CD, we aim to turn all deployments into simple service changes.

Trust

GitLab.com is the premier GitLab instance on the planet, and a production instance in every sense of the word. Change Management's primary goal is to safeguard the integrity of the GitLab.com environment through increased predictability by providing a framework to drive all changes towards becoming service changes and to help us achieve an optimal change speed.

Change Management is underpinned by trust: we trust ourselves to act responsibly in the operational environment to maintain its integrity and, by extension, its availability and performance.

To that end, we are not instituting a blanket policy for changes. Rather, we are developing the foundation of what a service change is (risk evalation, automatic auditing and communication, pre-flight checks, defensive coding, post-change validation) and will help teams with adoption.

Change Management helps us prioritize our resources towards changes that need to be made more resilient through defensive automation. Priorities are driven by two factors:

changes that cause ~S1 or ~S2 incidents
changes driven by services that are below a TBD error budget

In these situations, we will focus on developing the necessary automation and safeguards to help teams and services move towards safe service changes in a timely fashion. Until then, all changes that fall under the two abovementioned categories are treated as maintenance changes.

Change Severities

Change severities encapsulate the risk associated with a change in the environment. Said risk entails the potential effects if the change fails and becomes an incident. Change management uses our standarized severity definition, which can be found under out which can be found under issue workflow documentation.

In order to minimize the number of variables at play, no changes are executed during an active incident.
~S1 and ~S2 changes are always serialized and executed exclusively (i.e., never concurrently).
~S3 and ~S4 changes are allowed to take place concurrently as long as there is awareness of said concurrency.
The Infrastructure on-call resource has veto power over any and all changes.

Change Type

The change types can be one of the labels :

ConfigurationChange - to deploy a configuration change.
HotFix - to deploy a patch in a software that is production.
DeploymentNewFeature - to deploy of a new feature.

Change Plans

All changes should have change plans. While this is optional for service changes, it is mandatory for maintenance changes. Change Plans provide detailed descriptios of proposed changes and include the following information depending on the criticallity of the service :

Criticality 1:

Examples of Criticality 1:

Any changes to Postgres hosts that affects DB functionality - quantity of nodes, changes to backup or replication strategy
Architectural changes to Infra as code (IaC)
IaC changes to pets - Postgres, Redis, and other Single Points of Failure
Changes of major vendor - CDN, mail, DNS
Major version upgrades of tooling (HAProxy, Chef)

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer	A colleague will review the change
Tested in staging	The change was tested on staging environment
Dry-run output	If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Downtime Component	if yes how many minutes
Detailed steps for the change. Each step must include:	* pre-conditions for execution of the step, * execution commands for the step, * post-execution validation for the step , * rollback of the step, * review of changed graphs in grafana, * review of alerts to disable and enable

Criticality 2:

Examples of Criticality 2:

Load Balancer Configuration - major changes to backends or front ends, fundamental to traffic flow
IaC changes to cattle / quantity when there is a decrease
Minor version upgrades of tools or components (HAProxy)
Removing old hosts from IaC (like removals of legacy infrastructure)

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer	A colleague will review the change
Tested in staging	The change was tested on staging environment
Dry-run output	If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step, - review of changed graphs in grafana, - review of alerts to disable and enable

Criticality 3:

Examples of Criticality 3:

IaC changes to cattle / quantity when there is an increase (not requiring reboot or destroy/recreate)
Changes in configuration for current systems serving customers related to DNS or CDN

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer or tested in staging	A colleague will review the change or The change was tested on staging environment
Dry-run output	If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step, , - review of changed graphs in grafana, - review of alerts to disable and enable

With change plans, we develop a solid library of change procedures. Even more importantly, they provide detailed blueprints for implentation of defensive automation. Adding on to the defensive automation, every change request that uses some sort of a script must have a dry-run capability, the script should be ran in the dry-run mode and its output should be provided to the CR for review. Ideally, the planner and the executor should be different individuals.

Change Schedule

Please consider the timezone UTC as the standard for all the changes.

The following table has the original schedule for changes based on the criticality level of the component :

	10 PM - 6 AM	6 AM - 2 PM	2 PM - 10 PM
Criticality 1	ALLOWED	NOT ALLOWED	NOT ALLOWED
Criticality 2	ALLOWED	NOT ALLOWED	NOT ALLOWED
Criticality 3	ALLOWED	ALLOWED	ALLOWED
Criticality 4	ALLOWED	ALLOWED	ALLOWED

Please consider the time slots on the calendar Production, to add change requests to Criticality 1 and 2. The other criticalities please add direct to the calendar.

Change Request Workflow

To drive a change request to be executed on production :

Create an issue with the template of the criticality of the change
Add to the Production calendar in the timeslot desired or in the production calendar.
The issue will reviewed by SRE team management and Staff.
After getting the approval the change can be executed.

Change Reviews

Maintenance changes require change reviews. The reviews are intended to bring to bear the collective experience of the team while providing a forum for pointing out potential risks for any given change. A minimun quorun of three reviewers is required to approve a ~S1 or ~S2 maintenance change.

Roles

Role	Definition and Examples
`EMOC`	Event Manager
	The Event Manager is the tactical leader of the change team. For service changes, the EMOC is the person executing the change. For maintenance changes, the EMOC is the person in the IMOC rotation. ~S1 and ~S2 changes require an EMOC.
`CMOC`	Communications Manager
	The Communications Manager is the communications leader of the change team. The focus of the Change Team is executing the change as safely and quickly as possible. For ~S1 and ~S2 maintenance changes, a CMOC communicates with the appropriate stakeholders. Othersiwe, EMOC can handle communication.
`CT`	Change Team
	The Change Team is primarily composed of technical staff perfoming the change.

Communication Channels

Information is a key asset during any change. Properly managing the flow of information to its intended destination is critical in keeping interested stakeholders apprised of developments in a timely fashion. The awareness that a change is happening is critical in helping stakeholders plan for said changes.

This flow is determined by:

the type of information,
its intended audience,
and timing sensitivity.

For instance, a large end-user may choose to avoid doing a software release during a maintenance window to avoid any chance that issues may affect their release.

Furthermore, avoiding information overload is necessary to keep every stakeholder’s focus.

To that end, we will have:

a dedicated change bridge (zoom call) for S1 and S2 changes.
a dedicated #change channel, since #production contains sizeable amounts of information and it takes effort to filter out non-relevant items. This is particularly important for the change team, which must be focused on technical information to perform the change. While #change is an open channel and anyone is free to join, we will encourage people to use other channels to communicate with the EMOC.
periodic updates intended to the various audiences at place (CMOC handles this):
- End-users (Twitter)
- eStaff
- Support staff
- Employees at large
a dedicated repo for issues related to Production separate from the queue that holds Infrastructures’s workload: namely, issues for incidents and changes. This is useful because there may be other teams, over time, that need to do work in the production environment.
After the maintenance is complete - handoff notes to other region on call team members should be left. Including items like:
- state / success of the maintenance
- any alerts that can have been silenced and may go handoff
- links to specific graphs to watch for areas of concern

Production Change Lock (PCL)

While changes we make are rigorously tested and carefully deployed, it is a good practice to temporarily halt production changes during certain events such as GitLab LiveStream, GitLab Summit and days where LOA (leave of absence), due to holidays, is high in engineering teams. We categorize these special periods of times into two buckets:

GitLab Events
High LOA

Risks of making a production deployment during the said periods includes immediate customer impact and/or less engineering team coverage in case an incident occurs and has to be resolved immediately. Therefore, we have introduced a mechanism called Production Change Lock (PCL). We see the future of PCL as an automated process which, provided a time range, locks production deployments and releases the lock once the time expires. However, as the first iteration towards this future state we are starting with creating events on our Production Calendar so that teams are aware of the PCL periods.

Start Date/Time	End Date/Time	Timezone Region	Description
05/08/19 12:00AM	05/15/19 12:00	NA	GitLab Contribute 2019

Questions

Does production above include canary?

Yes.
Does this apply only to production environment?

Yes. Only production environment. This means you can still make changes and deployments to pre-prod environments.
What is the exact scope of the changes that are enforced under PCL? (infrastructure, software, handbook…etc)

Any production change that would make it to gitlab.com.
What if I still want to deploy during the PCL period?

VPE will need to approve your change to be deployed to production during PCL period.
Does this apply to our monthly release which happens on the 22nd?

No.
Why did you select these dates only?

The above listed periods are immediate times we know we will have less engineering teams' coverages (because of holidays, vacations and events). This was a data-driven decision based on our engineering population density around the globe. We hope to have made a progress towards the automated, future state of PCL beyond May, 2019 and if we achieve our goal we would be managing these via systems rather than manual communications and calendar events.
We have a question that is not answered here?

Please raise an issue to Infrastructure team's queue and we will be happy to get back to you as soon as we can.

Change Management

On this page