Changes

Changes are any modification to the operational environment and are classified into two types:

Service changes are regular, routine changes executed through well-tested, automated procedures performed with minimal human interaction that may cause predictable and limited performance degradation and no downtime. As such, service changes do not require review or approval except on their very first iteration.
Maintenance changes are possibly complex changes that require manual intervention beyond initiating the change and that will cause downtime or significant performance degradation by the nature of the change. These changes require strict scheduling, careful planning and review, and approval by the Director of Infrastructure for execution.

Deployments are a special change metatype depending on their scope and the effect they may have on the environment, as defined above. As we make progress towards CI/CD, we aim to turn all deployments into simple service changes.

Trust

GitLab.com is the premier GitLab instance on the planet, and a production instance in every sense of the word. Change Management's primary goal is to safeguard the integrity of the GitLab.com environment through increased predictability by providing a framework to drive all changes towards becoming service changes and to help us achieve an optimal change speed.

Change Management is underpinned by trust: we trust ourselves to act responsibly in the operational environment to maintain its integrity and, by extension, its availability and performance.

To that end, we are not instituting a blanket policy for changes. Rather, we are developing the foundation of what a service change is (risk evalation, automatic auditing and communication, pre-flight checks, defensive coding, post-change validation) and will help teams with adoption.

Change Management helps us prioritize our resources towards changes that need to be made more resilient through defensive automation. Priorities are driven by two factors:

changes that cause ~S1 or ~S2 incidents
changes driven by services that are below a TBD error budget

In these situations, we will focus on developing the necessary automation and safeguards to help teams and services move towards safe service changes in a timely fashion. Until then, all changes that fall under the two abovementioned categories are treated as maintenance changes.

Change Severities

Change severities encapsulate the risk associated with a change in the environment. Said risk entails the potential effects if the change fails and becomes an incident. Change management uses our standarized severity definition, which can be found under out which can be found under CONTRIBUTION.MD.

In order to minimize the number of variables at play, no changes are executed during an active incident.
~S1 and ~S2 changes are always serialized and executed exclusively (i.e., never concurrently).
~S3 and ~S4 changes are allowed to take place concurrently as long as there is awareness of said concurrency.
The Infrastructure on-call resource has veto power over any and all changes.

Change Type

The change types can be one of the labels :

ConfigurationChange - to deploy a configuration change.
HotFix - to deploy a patch in a software that is production.
DeploymentNewFeature - to deploy of a new feature.

Change Plans

All changes should have change plans. While this is optional for service changes, it is mandatory for maintenance changes. Change Plans provide detailed descriptios of proposed changes and include the following information depending on the criticallity of the service :

Criticality 1:

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer	A colleague will review the change
Tested in staging	The change was tested on staging environment
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Downtime Component	if yes how many minutes
Detailed steps for the change. Each step must include:	* pre-conditions for execution of the step, * execution commands for the step, * post-execution validation for the step , * rollback of the step

Criticality 2:

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer	A colleague will review the change
Tested in staging	The change was tested on staging environment
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step

Criticality 3:

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer or tested in staging	A colleague will review the change or The change was tested on staging environment
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step

Criticality 4:

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	List services
Change Team Members	Name of the involved in the change
Change Severity	How critical is the change
Change Reviewer or tested in staging	A colleague will review the change or The change was tested on staging environment
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of who is executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step

With change plans, we develop a solid library of change procedures. Even more importantly, they provide detailed blueprints for implentation of defensive automation. Ideally, the planner and the executor should be different individuals.

Change Schedule

Please consider the timezone UTC as the standard for all the changes.

The following table has the original schedule for changes based on the criticality level of the component :

	10 PM - 6 AM	6 AM - 2 PM	2 PM - 10 PM
Criticality 1	ALLOWED	NOT ALLOWED	NOT ALLOWED
Criticality 2	ALLOWED	NOT ALLOWED	NOT ALLOWED
Criticality 3	ALLOWED	ALLOWED	ALLOWED
Criticality 4	ALLOWED	ALLOWED	ALLOWED

Please consider the time slots on the calendar Production, to add change requests to Criticality 1 and 2. The other criticalities please add direct to the calendar.

Change Request Workflow

To drive a change request to be executed on production :

Create an issue with the template of the severity needed
Add to the calendar xxxxxx in the timeslot desired or in the calendar directly.
The issue will reviewed by SRE team management and Staff.
After getting the approval the change can be executed.

Change Reviews

Maintenance changes require change reviews. The reviews are intended to bring to bear the collective experience of the team while providing a forum for pointing out potential risks for any given change. A minimun quorun of three reviewers is required to approve a ~S1 or ~S2 maintenance change.

Roles

Role	Definition and Examples
`EMOC`	Event Manager
	The Event Manager is the tactical leader of the change team. For service changes, the EMOC is the person executing the change. For maintenance changes, the EMOC is the person in the IMOC rotation. ~S1 and ~S2 changes require an EMOC.
`CMOC`	Communications Manager
	The Communications Manager is the communications leader of the change team. The focus of the Change Team is executing the change as safely and quickly as possible. For ~S1 and ~S2 maintenance changes, a CMOC communicates with the appropriate stakeholders. Othersiwe, EMOC can handle communication.
`CT`	Change Team
	The Change Team is primarily composed of technical staff perfoming the change.

Communication Channels

Information is a key asset during any change. Properly managing the flow of information to its intended destination is critical in keeping interested stakeholders apprised of developments in a timely fashion. The awareness that a change is happening is critical in helping stakeholders plan for said changes.

This flow is determined by:

the type of information,
its intended audience,
and timing sensitivity.

For instance, a large end-user may choose to avoid doing a software release during a maintenance window to avoid any chance that issues may affect their release.

Furthermore, avoiding information overload is necessary to keep every stakeholder’s focus.

To that end, we will have:

a dedicated change bridge (zoom call) for S1 and S2 changes.
a dedicated #change channel, since #production contains sizeable amounts of information and it takes effort to filter out non-relevant items. This is particularly important for the change team, which must be focused on technical information to perform the change. While #change is an open channel and anyone is free to join, we will encourage people to use other channels to communicate with the EMOC.
periodic updates intended to the various audiences at place (CMOC handles this):
- End-users (Twitter)
- eStaff
- Support staff
- Employees at large
a dedicated repo for issues related to Production separate from the queue that holds Infrastructures’s workload: namely, issues for incidents and changes. This is useful because there may be other teams, over time, that need to do work in the production environment.

Change Management

On this page