Change Management has traditionally referred to the processes, procedures, tools and techniques applied in IT environments to carefully manage changes in an operational environment: change tickets and plans, approvals, change review meetings, scheduling, and other red tape.
In our context, Change Management refers to the guidelines we apply to manage changes in the operational environment with the aim of doing so (in order of highest to lowest priority) safely, effectively and efficiently. In some cases, this will require the use of elements from traditional change management; in most cases, we aim to build automation that removes those traditional aspects of change management to increase our speed in a safe manner.
Our overriding objective is maximize changes that avoid traditional aspects of change management, which is an iterative process that will evolve over time. Success is measured by our ability to safely execute changes at the speed required by our business needs.
Changes are defined as modifications to the operational environment, including configuration changes, adding or removing components or services to the environment and cloud infrastructure changes. Application deployments, while technically being changes, are excluded from the change management process, as are most, but not all, feature flag toggles.
Service changes are regular, routine changes executed through well-tested, automated procedures performed with minimal human interaction that may cause predictable and limited performance degradation and no downtime. A service change is implemented such that it protects the environment while executing the desired change. As such, mature service changes do not require review or approval except on their very first iteration.
Maintenance changes are complex changes that require manual intervention and will cause downtime or significant performance degradation. These changes require strict scheduling, careful planning and review, and approval by the Director of Reliability.
Deployments are a special change metatype depending on their scope and the effect they may have on the environment, as defined above. As we make progress towards CI/CD, we aim to turn all deployments into simple service changes.
Changes that need to be performed during the resolution of an Incident fall under Incident Management.
Operational Environments are currently defined as GitLab.com and ops.GitLab.net as well as supporting systems such as Prometheus.
Change Management is underpinned by trust: we trust ourselves to act responsibly when working in the operational environment and do everything in our power to safeguard its integrity and maintain its availability and performance.
To that end, we must develop a practice through automation that allows for safe changes. Our highest priority is always the integrity and reliability of the operational environment, which entails appropriate risk evaluation, quantifiable validation and verification, extensive communication, detailed auditing, and a focus on defensive coding.
At its core, changes implement the transitions that take the operational environment from its current state to a new state that yields the desired configuration. It is critical that we have a full and concrete understanding of both states, as they determine how the transition is planned and executed.
Change always begins with the current state of the environment, which is defined by both its actual state and the assumptions made about it. Predicting the ways in which a change will fail or yield undesirable results is, at best, an extremely difficult undertaking (if at all possible); it is outright impossible when our assumptions do not match reality, which is one of the primary sources of change failure, as the change will behave in unexpected ways under uncertain conditions, likely resulting in an incident.
All changes, without exception, regardless of complexity and whether they require a change plan or not, must outline, implement and codify pre-flight checks and post-verification checks. The automation's primary goal is to protect the environment. When exceptions are necessary, the author should document why this is the case, and the reviewer should approve said exception.
In general, three rules are critical to the successful execution of a change:
Pre-flight checks protect the operational environment by validating assumptions about the current state of the environment and provide the necessary gates to the execution of any change. If any assumption is proven to be incorrect through these checks, a change should either be halted or take corrective action: under no circumstances should a change proceed as originally planned when pre-flight checks are not successful.
Pre-flight checks should be coded into the automation and executed as the prerequisite step of a change. In essence, we are asking the question what must be true for the change at hand to execute successfully?
In determining what these checks should entail, we must focus on the assumptions implicit in the change. There are many examples of how to defensily protect the environment:
While pre-flight checks will likely have an effect on the speed of a change (both on our speed to implement changes and on the runtimes), failure has a far greater impact, especially on our users. When working on Infrastructure changes, safety is always prioritized over velocity.
Much in the way we must validte the current state of the environment and the assumptions we make about it before a change, we must always validate the new, post-change state and the assumptions made about it. We must explicitly quantify the desired results of the change.
As examples:
Key to all pre-flight and post-verification steps is our monitoring stack, based primarily on Prometheus, Thanos and AlertManager (the monitoring service).
Additionally, most incidents are the direct result of change, either in the application, through deployments, or infrastructure changes. It is therefore imperitive that any elective change (ie, one not related to an urgent ongoing incident) requires the availability of our metrics platform.
If the monitoring platform is not functioning correctly, no change should be executed, unless it is being made to resolve an ongoing critical incident.
"no metrics, no changes."
Change Management helps us prioritize our resources towards changes that need to be made more resilient through defensive automation. Priorities are driven by two factors:
In these situations, we will focus on developing the necessary automation and safeguards to help teams and services move towards safe service changes in a timely fashion. Until then, all changes that fall under the two aforementioned categories are treated as maintenance changes.
Change severities encapsulate the risk associated with a change in the environment. Said risk entails the potential effects if the change fails and becomes an incident. Change management uses our standarized severity definition, which can be found under out which can be found under issue workflow documentation.
All changes should have change plans. Planning is the way the infrastructure department assesses and mitigates the risks changes introduce. They generate awareness and are the focal point for scheduling, communicating, and recording changes.
Plan issues are opened in the production project tracker via the change management issue template. Each issue should be opened using an issue template for the corresponding level of criticality: C1, C2, C3, or C4. It must provide a detailed description of the proposed change and include all the requested information in the template. Every plan issue is initially labeled ~"change::unscheduled" until it can be reviewed and scheduled with a Due Date. After the plan is approved and scheduled it should be labeled ~"change::scheduled" for visibility.
All change requests, regardless of the change criticality follow the same change management issue template.
These are changes with high impact or high risk. If a change is going to cause downtime to the environment, it is always categorized a C1. Before implementing the change.
Examples of Criticality 1:
Creating and deploying a custom build generated by ourselves is generally not recommended. There are use cases where it could be necessary due to a security or stability patch, to apply before it is part of the official upstream release. It is only allowed through Criticality 1 guidelines.
#production Slack channel and obtain a written approval from the EOC in both the issue and in Slack.The EOC must be engaged for the entire execution of the change.
These are changes that are not expected to cause downtime, but which still carry some risk of impact if something unexpected happens. For example, reducing the size of a fleet of cattle is usually ok because we've identified over-provisioning, but we need to take care and monitor carefully before and after.
Examples of Criticality 2:
#production Slack channel and obtain a written approval from the EOC in both the issue and in Slack.These are changes with either no or very-low risk of negative impact, but where there is still some inherent complexity, or it is not fully automated and hands-off
Examples of Criticality 3:
These are changes that are exceedingly low risk and commonly executed, or which are fully automated. Often these will be changes that are mainly being recorded for visibility rather than as a substantial control measure.
Examples of Criticality 4:
gitlab-rails or gitlab-rake.No approval required.
With change plans, we develop a solid library of change procedures. Even more importantly, they provide detailed blueprints for implementation of defensive automation. Adding on to the defensive automation, every change request that uses some sort of a script must have a dry-run capability, the script should be run in the dry-run mode and its output should be provided to the CR for review. Ideally, the planner and the executor should be different individuals.
Please consider the timezone UTC as the standard for all the changes.
The following table has the original schedule for changes based on the criticality level of the component :
Weekdays:
| 10 PM - 6 AM | 6 AM - 2 PM | 2 PM - 10 PM | |
|---|---|---|---|
| Criticality 1 | ALLOWED | NOT ALLOWED | NOT ALLOWED |
| Criticality 2 | ALLOWED | NOT ALLOWED | NOT ALLOWED |
| Criticality 3 | ALLOWED | ALLOWED | ALLOWED |
| Criticality 4 | ALLOWED | ALLOWED | ALLOWED |
Weekends:
No specific restriction about times. If the change risks are high or requires downtime, please schedule it at the least impactful time for our customers (like Sunday 1am UTC, for example)
Please consider the above time slots when posting criticalities 1 and 2 to the Production calendar, for all other criticalities, please add them directly to the calendar.
If the change is executed by a script, it should be run from the bastion host of the target environment in a terminal multiplexer (e.g. screen or tmux) session. Using a bastion host has the benefit of preventing any unintended actions (e.g. caused by a script bug) from spreading to other environments. A terminal multiplexer guards against the possibility of losing connection to the bastion mid-change and the unpredictable consequences of it.
sudo is disabled on the bastion hosts, so you can copy your Chef PEM file to
one of them, if your script requires it, without fearing it being snooped on.
A sequence of actions to run a script could look like this:
your-workstation $ ssh -A bastion-01-inf-gstg
bastion-01-gstg $ tmux
bastion-01-gstg $ git clone git@gitlab.com:my-migration/script.git
bastion-01-gstg $ ./script/migrate
Maintenance changes require change reviews. The reviews are intended to bring to bear the collective experience of the team while providing a forum for pointing out potential risks for any given change. A minimun quorun of three reviewers is required to approve a ~S1 or ~S2 maintenance change.
| Role | Definition and Examples |
|---|---|
EMOC |
Event Manager |
| The Event Manager is the tactical leader of the change team. For service changes, the EMOC is the person executing the change. For maintenance changes, the EMOC is the person in the IMOC rotation. ~S1 and ~S2 changes require an EMOC. | |
CMOC |
Communications Manager |
| The Communications Manager is the communications leader of the change team. The focus of the Change Team is executing the change as safely and quickly as possible. For ~S1 and ~S2 maintenance changes, a CMOC communicates with the appropriate stakeholders. Othersiwe, EMOC can handle communication. | |
CT |
Change Team |
| The Change Team is primarily composed of technical staff perfoming the change. |
Information is a key asset during any change. Properly managing the flow of information to its intended destination is critical in keeping interested stakeholders apprised of developments in a timely fashion. The awareness that a change is happening is critical in helping stakeholders plan for said changes.
This flow is determined by:
For instance, a large end-user may choose to avoid doing a software release during a maintenance window to avoid any chance that issues may affect their release.
Furthermore, avoiding information overload is necessary to keep every stakeholder’s focus.
To that end, we will have:
From time to time we will need to run a production change that requires downtime, affecting customers and our SLO. This section covers how to successfully manage communications in these type of situations.
As a reference, we should communicate 5-6 weeks before the change, for a C1 that does not carry a significant architecture change. Longer preparation time is adviced if the change involves a large migration or a significant architecture change.
Steps:
Confidential until the day we publish it in status.io, when we will set it to Not Confidential.#whats-happening-at-gitlab slack channelWhile changes we make are rigorously tested and carefully deployed, it is a good practice to temporarily halt production changes during certain events such as GitLab LiveStream, GitLab Summit and days where LOA (leave of absence), due to holidays, is high in engineering teams. We categorize these special periods of times into two buckets:
Risks of making a production deployment during the said periods includes immediate customer impact and/or less engineering team coverage in case an incident occurs and has to be resolved immediately. Therefore, we have introduced a mechanism called Production Change Lock (PCL). We see the future of PCL as an automated process which, provided a time range, locks production deployments and releases the lock once the time expires. However, as the first iteration towards this future state we are starting with creating events on our Production Calendar so that teams are aware of the PCL periods.
The following dates are currently scheduled PCLs. Times for the dates below begins at 09:00 UTC and ends at 09:00 UTC.
| Dates | Type | Reason |
|---|---|---|
| 24 December 2019 | Soft | Holiday: Christmas Eve |
| 25-26 December 2019 | Hard | Holiday: Christmas |
| 27-31 December 2019 | Soft | High Number of SRE Vacations |
| 01 January 2020 | Hard | Holiday: New Year's Day |
| 21-28 March 2020 | Hard | GitLab Contribute |
| 22nd of every month | Soft | Release day |
There are 2 types of PCLs: soft and hard.
Soft PCLs aim to mitigate risk without halting all changes to production. Soft PCLs prohibit infrastructure changes with a criticality level of 2 or higher. In case of an emergency, the EOC should interact with the IMOC for C1 and C2 changes.
During the soft PCL, code deployments to canary are allowed since we have tools to control canary impact. Production deployments are allowed for lower criticality items (C3/C4) in coordination with the EOC. These items include high priority code deployments (impactful bugs, security fixes).
Hard PCLs include code deploys and infrastructure changes for every criticality level (see change severities). In case of an emergency, the EOC should interact with the IMOC prior to making any decision. It is at EOC and IMOC discretion to make a decision on whether a change should be approved and executed. If the change is approved, IMOC should inform the VPE of this decision.
Feature flags reduce risk by allowing application changes to be easily tested in production. Since they can be gradually or selectively enabled, and quickly turned off, their use is encouraged whenever appropriate.
However, as the company and the number of developers working with feature flags continues to grow, it becomes important to manage risk associated with these changes too. Developers follow the process defined in the developers documentation for feature flags.
On any given day, dozens of feature flag changes may occur. Many of these are trivial, allowing low risk changes – sometimes just changes to UI appearance – to be tested. However, some feature flag changes can have a major impact on the operation of GitLab.com, negatively affecting our service level agreements. This in turn can have a financial and reputational risk for the company. Without clear communication between the application developers toggling features and the engineer-on-call (EOC), it can be difficult for the EOC to assess which feature flag toggles are high risk and which are not.
Additionally, during an incident investigation, knowing which high-risk features have recently been enabled, and documentation on how to assess their operation, is important.
For this reason, feature flag toggles which meet any of the below criteria, should be accompanied by a change management issue.:
Does production above include canary?
Yes.
Does this apply only to production environment?
Yes. Only production environment. This means you can still make changes and deployments to environments other than production.
What is the exact scope of the changes that are enforced under PCL? (infrastructure, software, handbook…etc)
Any production change that has a potential of making gitlab.com unavailable. For example, configuration changes, setup of new libraries, introducing new code.
What if I still want to make a change during the PCL period?
VPE will need to approve your change production rollout during PCL period in non-urgent situations.
Does this apply to our monthly release which happens on the 22nd?
No. If 22nd falls under PCL period, additional coordination is necessary to ensure uninterrupted monthly release.
Why did you select these dates only?
The above listed periods are immediate times we know we will have less engineering teams' coverages (because of holidays, vacations and events). This was a data-driven decision based on our engineering population density around the globe. We hope to have made a progress towards the automated, future state of PCL beyond May, 2019 and if we achieve our goal we would be managing these via systems rather than manual communications and calendar events.
We have a question that is not answered here?
Please raise an issue to Infrastructure team's queue and we will be happy to get back to you as soon as we can.
