NOTE: The content below covers feature flags used by GitLab to deploy its own features, which is not the same as the feature flags offered as part of the product.
This page defines the process for the feature flags lifecycle at GitLab. Technical information for GitLab developers and code contributors can be found in the developer documentation.
The deployment of features behind feature flags is required for changes that could affect GitLab's availability of existing functionalities. If your changes only affect the new feature you're creating, use your judgment to determine if you need one or not. Such changes include:
Situations where you might consider not using a feature flag:
In all cases, if you're working on the changes like these, ask yourself:
Why do I need to add a feature flag? If I don't add one, what options do I have to control the impact on application reliability and user experience?
For perspective on why we limit our use of feature flags please watch the video Feature flags only when needed.
In case you are uncertain whether feature flag is necessary, please ask about this early in your merge request review process, and reviewers will likely provide you with an answer.
It may seem like feature flags are configuration, which goes against our convention-over-configuration principle. However, configuration is by definition something that is user-manageable. Feature flags are not intended to be user-editable. Instead, they are intended as a tool for Engineers and Site Reliability Engineers to use to de-risk their changes. Feature flags are the shim that gets us to Continuous Delivery with our monorepo and without having to deploy the entire codebase on every change. Feature flags are created to ensure that we can safely rollout our work on our terms. If we use Feature Flags as a configuration, we are doing it wrong and are indeed in violation of our principles. If something needs to be configured, we should intentionally make it configuration from the first moment.
Some of the benefits of using development-type feature flags are:
When reading about the process, you can think that deploying a feature behind a feature flag adds a lot of work. Fortunately, this is not the case, and we'll show you why. For this example, we'll specify the cost of the work to do as a number, ranging from 0 to infinity. The greater the number, the more expensive the work is. The cost does not translate to time, it's just a way of measuring complexity of one change relative to another.
Let's say we are building a new feature, and we have determined that the cost of this is 10. We have also determined that the cost of adding a feature flag check in a variety of places is 1. If we do not use feature flags, and our feature works as intended, our total cost is 10. This, however, is the best-case scenario. Optimizing for the best-case scenario is guaranteed to lead to trouble, whereas optimizing for the worst-case scenario is almost always better.
To illustrate this, let's say our feature causes an outage, and there's no immediate way to resolve it. This means we'd have to take the following steps to resolve the outage:
As history has shown, these steps are time-consuming, complex, often involve many developers, and, the worst of all: our users will have a bad experience using GitLab.com until the problem is resolved.
Now let's say that all of this has an associated cost of 10. This means that, in the worst-case scenario, which we should optimize for, our total cost is now 20.
If we had used a feature flag, things would have been very different. We don't need to revert a release, and because feature flags are disabled by default, we don't need to revert and pick any Git commits. In fact, all we have to do is disable the feature, and in the worst case, perform a cleanup. Let's say that the cost of this is 2. In this case, our best case cost is 11: 10 to build the feature, and 1 to add the feature flag. The worst case cost is now 13:
Here we can see that, in the best-case scenario, the work necessary to resolve the issue is only a tiny bit bigger when compared to not using a feature flag. Meanwhile, the process of reverting our changes was significantly and reliably cheaper.
In other words, feature flags do not slow down the development process. Instead, they speed up the process as managing incidents now becomes much easier. As continuous deployments are easier to perform, the time to iterate on a feature is reduced even further, as you no longer need to wait weeks before your changes are available on GitLab.com.
For an overview about starting with feature flags in GitLab development, follow the training template.
Before using feature flags, make sure to read the information on this page and the following development guides:
For information about how the user will interact with features behind flags, see:
In order to build a final release and present the feature for self-managed users, the feature flag should be at least defaulted to on. If the feature is deemed stable and there is confidence that removing the feature flag is safe, consider removing the feature flag altogether. It's strongly recommended that the feature flag is enabled globally on production for at least one day before making this decision. Unexpected bugs are sometimes discovered during this period.
The process for enabling features that are disabled by default can take 5-6 days from when the merge request is first reviewed to when the change is deployed to GitLab.com. However, it is recommended to allow 10-14 days for this activity to account for unforeseen problems.
Feature flags must be documented according to their state (enabled/disabled), and when the state changes, docs must be updated accordingly.
Changing the default state or removing the feature flag has to be done before the 22nd of the month, at least 3-4 working days before, in order for the change to be included in the final self-managed release.
Note: consider that, by removing the flag, the feature will become available on GitLab.com shortly after the merge request was merged.
In addition to this, the feature behind feature flag should:
While rare, release managers may decide to reject picking or revert a change in a stable branch, even when feature flags are used. This might be necessary if the changes are deemed problematic, too invasive, or there simply isn't enough time to properly measure how the changes behave on GitLab.com.
Feature flag rollouts require access to internal GitLab systems. Because of this restriction, a community contribution will require assistance from a GitLab team member to rollout the feature flag.
During the planning phase it is essential that the EM and PM are pinged by the MR Coach or a GitLab Team member, who executes the review, on the feature flag rollout issue to get it scheduled/planned/refined. Also they should tag the feature flag rollout issue in the final MR using the feature flag, so that the EM and PM are aware that the rollout will be required soon.
For more information on the lifecycle of a feature flag, such as how many are introduced per release, how long they exist in the application, or how long they've been enabled, see this dashboard (internal only). We also carry 7 days worth of data in Kibana (internal only) for feature flags that become disabled or enabled.
