Published on: September 20, 2023
24 min read
Follow this comprehensive tutorial to learn how to create and use feature flags in your software development environment.
Developers typically use advanced techniques like canary, blue/green, and incremental deployments to reduce risk when practicing progressive delivery, a facet of continuous delivery (CD). In this tutorial, we will show you how to use feature flags, another progressive delivery option developers can use to test while in production.
Progressive delivery is the ability to test in production while controlling your audience of who can exercise or see updates to an application with a high level of granularity. This approach can also be thought of as developer experimentation.
Feature flags enable you to choose what to deploy and who to deploy to in production. They allow you to define the audience for your application updates as well as the fashion in which they will be served.
Feature flags help stakeholders reduce risk, allowing them to do controlled testing of features and separate feature delivery from customer launch.
The following are benefits of GitLab's feature flags.
Lower risk. Feature flags prevent unscheduled outages, control your audience in a fine-grained fashion, and can be optionally used in conjunction with canary deployments.
Ease of use. Feature flags have simple configurability and instrumentation, support user lists, and offer built-in service.
Language agnostic. Our feature flag implementation supports all of the main programming languages.
Better compliance and audit capabilities. The GitLab platform automatically records all feature flags actions.
This is what you need for this tutorial:
A GitLab account on gitlab.com SaaS
Flux CLI installed on your local desktop (on my Mac, I installed it by
executing brew install fluxcd/tap/flux
)
A running Kubernetes cluster, i.e. a GKE cluster with 3 e2-medium nodes
kubectl
connectivity to your Kubernetes cluster from a local Terminal
window on your desktop
This tutorial is based on a fictitious application, which is a simplified inventory system. The goal of this tutorial is to show you how to create, configure, and implement a feature flag using GitLab.
Note: This tutorial is for learning purposes and not meant to deploy a production-ready architecture. Also, to keep the number of steps low, masked variables and sealed secrets are not being used throughout this tutorial.
Here is how to install Flux and GitLab agent for Kubernetes.
Log on to your GitLab workspace.
Create a personal access token (PAT) from your GitLab account by
navigating to User settings > Preferences > Access tokens. In the
Personal Access Tokens section, click on the Add new token button on
the righthand side of the section. For Token name, enter pat-for-flux
.
Leave the expiration date with its default (it should be 30 days from its
creation) and select the API scope for your PAT. Click on the
Create personal access token button to create your PAT. Copy and save
the value of your PAT; you will need it at a later step.
{:
.shadow.medium.center}
Creating a personal access token
Head back to your GitLab workspace main page.
Create a group named “hn” by clicking the button New group (or New subgroup if you are creating this group inside an existing group) on the top right hand side of your screen, and then clicking on the Create group tile. Enter "hn" for your Group name and click on the Create group button to create it. Leave the rest of the fields with their defaults.
{:
.shadow.medium.center}
Creating group "hn"
{:
.shadow.medium.center}
Creating project "flux-config"
kubectl
access to your Kubernetes cluster,
export your PAT by entering the following command:export GITLAB_TOKEN=
<replace with your PAT value>
kubectl
access to your Kubernetes cluster,
bootstrap Flux by executing the following command:Note: Make sure to replace <your path>
with whatever precedes your
group “hn”. For example, it could be --owner=tech-marketing/sandbox/hn
, or
if your group “hn” is at the very top level of your GitLab workspace, it
would be --owner=hn
.
flux bootstrap gitlab \
--owner=<your path>/hn \
--repository=flux-config \
--branch=main \
--path=clusters/my-cluster \
--deploy-token-auth
{:
.shadow.medium.center.}
Flux bootstrap output
The “flux-config” project should now contain new directories and files as shown below:
{:
.shadow.medium.center}
Project flux-config post flux bootstrap process
Note: Make sure to replace <your path>
with whatever precedes your
group “hn”. For example, it could be - id: tech-marketing/sandbox/hn
or if
your group “hn” is at the very top level of your GitLab workspace, it would
be - id: hn
.
ci_access:
groups:
- id: <your path>/hn
Commit this file to main by clicking on the Commit changes button and ensuring that the target branch is “main”.
{:
.shadow.medium.center}
Creating the GitLab agent for Kubernetes configuration manifest
{:
.shadow.medium.center}
Registering the GitLab agent for Kubernetes
{:
.shadow.medium.center}
The agent access token to save
At this moment, you will see the agent listed and its Connection status will be “Never connected”.
{:
.shadow.medium.center}
Agent registered but not connected yet
apiVersion: v1
kind: Namespace
metadata:
name: gitlab
{:
.shadow.medium.center}
Manifest for the gitlab namespace
Commit this file to main by clicking on the Commit changes button and ensuring that the target branch is “main”.
Note: You can check that the namespace was created in your cluster by
executing this command from a Terminal:
kubectl get ns
{:
.shadow.medium.center}
Flux created gitlab namespace
Note: Make sure to replace <your-agent-access-token-here>
with your
Agent access token you saved earlier.
apiVersion: v1
kind: Secret
metadata:
name: gitlab-agent-token-initial
type: Opaque
stringData:
values.yaml: |-
config:
token: "<your-agent-access-token-here>"
{:
.shadow.medium.center.}
Manifest for agent token secret created on local desktop
kubectl apply -f secret.yaml -n gitlab
Note: You can check that the secret was created in your cluster by executing
this command from a Terminal:
kubectl get secrets -n gitlab
{:
.shadow.medium.center}
Applying the agent token secret to the Kubernetes cluster
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
labels:
app.kubernetes.io/component: agentk
app.kubernetes.io/created-by: gitlab
app.kubernetes.io/name: agentk
app.kubernetes.io/part-of: gitlab
name: gitlab-agent
namespace: gitlab
spec:
interval: 1h0m0s
url: https://charts.gitlab.io
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: gitlab-agent
namespace: gitlab
spec:
chart:
spec:
chart: gitlab-agent
sourceRef:
kind: HelmRepository
name: gitlab-agent
namespace: gitlab
interval: 1h0m0s
values:
replicas: 1
config:
kasAddress: "wss://kas.gitlab.com"
valuesFrom:
- kind: Secret
name: gitlab-agent-token-initial
valuesKey: values.yaml
{:
.shadow.medium.center}
Creating the manifest for the GitLab agent for Kubernetes
Commit this file to main by clicking on the Commit changes button and ensuring that the target branch is “main”.
Note: In a few seconds, you can check that the GitLab agent for Kubernetes
was created in your cluster by executing this command from a Terminal (the
pod name should start with “gitlab-agent”):
kubectl get pods -n gitlab
{:
.shadow.medium.center}
Agentk running in the Kubernetes cluster
Using the breadcrumb at the top of your window, head to group “hn” and create a new project by clicking on the New project button. On the Create new project window, click on the Import project tile.
At the Import project window, click on the Repository by URL button. The window will display fields to enter the URL of the repository you would like to import. In the text field Git repository URL, enter the following:
https://gitlab.com/tech-marketing/sandbox/mysql.git
Leave the rest of the fields with their defaults.
{:
.shadow.medium.center}
Importing mysql project into group "hn"
Click on the Create project button at the bottom of the screen. You will see an "Importing in progress" message temporarily on your screen.
Now we need to create a deploy token for this project so that Flux can interact with it. While in project “mysql”, select Settings > Repository and scroll down to the Deploy tokens section. Click on the Expand button to the right of the Deploy tokens section. Then click on the Add token button, which will expand the section to include fields to start entering information for the deploy token to be created.
Give the deploy token the name “mysql-flux-deploy-token” and check the checkbox read_repository for it. Then click on the button Create deploy token to create the token.
{:
.shadow.medium.center}
Creating the deploy token for "mysql" project for Flux to interact with it
Copy and save the username and password for the newly created deploy token; you will need them at a later step.
{:
.shadow.medium.center}
Creating the deploy token for "mysql" project for Flux to interact with it
Note: Make sure to replace <your path>
with the missing partial path
to the project “mysql”,
flux create secret git mysql-flux-deploy-authentication \
--url=https://gitlab.com/<your path>/hn/mysql \
--namespace=default \
--username=<your-deploy-token-username> \
--password=<your-deploy-token-password>
Note: You can check that the secret was created in your cluster by executing
this command from a Terminal:
kubectl -n default get secrets mysql-flux-deploy-authentication
{:
.shadow.medium.center}
Creating secret for the deploy token for "mysql" project in the Kubernetes cluster
{:
.shadow.medium.center}
Selecting Web IDE from the dropdown menu
{:
.shadow.medium.center}
Navigate to directory "clusters/my-cluster" in the Web IDE
Note: Replace <your path>
with the missing partial path to the project
“mysql”
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: mysql
namespace: default
spec:
interval: 1m0s
ref:
branch: main
secretRef:
name: mysql-flux-deploy-authentication
url: https://gitlab.com/<your path>/hn/mysql
{:
.shadow.medium.center}
Creating mysql-manifests-source.yaml file in the Web IDE
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: mysql-source-kustomization
namespace: default
spec:
interval: 1m0s
path: ./
prune: true
sourceRef:
kind: GitRepository
name: mysql
namespace: default
targetNamespace: default
{:
.shadow.medium.center}
Creating mysql-manifests-kustomization.yaml file in the Web IDE
{:
.shadow.medium.center}
Clicking on the Source Control icon and committing to main in the Web IDE
Then press the Continue button to confirm that you want to commit your changes to the default branch:
{:
.shadow.medium.center}
Clicking on the Source Control icon and committing to main in the Web IDE
Note: You can check that the GitLab agent for Kubernetes was created in your
cluster by executing this command from a Terminal:
kubectl get pods -l app=mysql
You can check the persistent volume by executing this command from a
Terminal:
kubectl describe pvc mysql-pv-claim
{:
.shadow.center}
Verifying that mysql pod and its associated persitent volume claim are up and ready
{:
.shadow.medium.center}
Head to "mysql" project and select Build > Pipelines from the left vertical navigation menu
{:
.shadow.medium.center}
Clicking on the Run pipeline button to run the project "mysql" pipeline
{:
.shadow.medium.center}
The "mysql" pipeline and its two manual jobs
Note: The clear_db job should only be used if you’d like to erase all of the database resources created by the create_and_load_db job. The clear_db should only be used AFTER a failed run of the create_and_load_db job.
Now that we have the database ready to go, let’s set up the project that we will use for the creation of the feature flags.
Note 1: Make sure to create and set the KUBE_CONTEXT and KUBE_NAMESPACE variable in group “hn” and to these values:
| variable | value |
| --- | --- |
| KUBE_CONTEXT | <your path>
/hn/flux-config:k8s-agent |
| KUBE_NAMESPACE | my-apps |
For example, in my case <your path>
was
“tech-marketing/sandbox/hn/flux-config:k8s-agent”. In your case, it will be
different. If <your path>
is at the root of your GitLab workspace, then it
would be empty so the value of KUBE_CONTEXT would be
“hn/flux-config:k8s-agent”.
{:
.shadow.medium.center}
Adding variable KUBE_CONTEXT in group "hn"
{:
.shadow.medium.center}
Adding variable KUBE_NAMESPACE in group "hn"
Note 2: As an FYI, when uncommenting the GitLab managed apps in the “helmfile.yaml” file, there will not be one for Prometheus. So, you will only uncomment the lines for ingress and cert-manager.
{:
.shadow.medium.center}
Uncommenting lines for ingress and cert-manager in file "helmfile.yaml"
Note 3: When the pipeline for project “cluster-management” runs, you will notice that the job “sync” is a manual job. You will need to click on its Play (right arrow next to its name) button to run it. Wait until the “sync” job completes successfully before continuing.
{:
.shadow.medium.center}
Job "sync" is manual so you need to press on the Play button next to its name
Note 4: Once the pipeline finishes, for your convenience, here is the command you need to run from a Terminal window to get the external IP address of your cluster:
kubectl --namespace gitlab-managed-apps get services -o wide -w
ingress-ingress-nginx-controller
{:
.shadow.medium.center}
Running kubectl
command to get the ingress IP address to the cluster
Create and set a variable KUBE_INGRESS_BASE_DOMAIN
in group “hn” and set
it to the external IP address of your cluster and append the suffix
“.nip.io” to it.
{:
.shadow.medium.center}
Addding variable KUBE_INGRESS_BASE_DOMAIN in group "hn"
Inside group “hn”, create a new project. Click on the New project button. On the Create new project window, click on the Import project tile and then click on the Repository by URL button.
This will expand the window and show fields to enter the URL of the repository you would like to import. In the field Git repository URL, enter the following:
https://gitlab.com/tech-marketing/sandbox/prodmgr.git
Leave the rest of the fields with their defaults.
{:
.shadow.medium.center}
Importing project "prodmgr" into group "hn"
Click on the Create project button at the bottom of the screen. You will see an Importing in progress message temporarily on your screen.
In project “prodmgr”, create a pipeline file and make sure to name it “.gitlab-ci.yml”. Paste the following code block into the empty file:
include:
template: Auto-DevOps.gitlab-ci.yml
variables:
K8S_SECRET_TF_VAR_dbusername: "sasha"
K8S_SECRET_TF_VAR_dbpassword: "password"
TEST_DISABLED: "true"
CODE_QUALITY_DISABLED: "true"
LICENSE_MANAGEMENT_DISABLED: "true"
BROWSER_PERFORMANCE_DISABLED: "true"
LOAD_PERFORMANCE_DISABLED: "true"
SAST_DISABLED: "true"
SECRET_DETECTION_DISABLED: "true"
DEPENDENCY_SCANNING_DISABLED: "true"
CONTAINER_SCANNING_DISABLED: "true"
DAST_DISABLED: "true"
REVIEW_DISABLED: "true"
CODE_INTELLIGENCE_DISABLED: "true"
CLUSTER_IMAGE_SCANNING_DISABLED: "true"
POSTGRES_ENABLED: "false"
STAGING_ENABLED: "true"
INCREMENTAL_ROLLOUT_MODE: "manual"
Click on the Commit changes button ensuring that the Target branch is main.
{:
.shadow.medium.center}
Creating an Auto-DevOps-based pipeline for project "prodmgr"
{:
.shadow.medium.center}
To deploy the application to production, click on the rollout 100% Play button
At this point, you have a running application in the staging and production environments in your Kubernetes cluster. Let’s start creating a feature flag.
Click on the link View user lists on the top right hand side of your screen.
Click on the New user list button on the top right hand side of your screen.
In the Name field of the user list, enter “prods-in-alphabetical-order-userlist” and then click on the Create button.
{:
.shadow.medium.center.}
Creating user list named "prods-in-alphabetical-order-userlist”
On the next screen, click on the Add Users button on the top right hand side of your screen.
In the User IDs text field, enter the following two email addresses and then click on the Add button:
{:
.shadow.medium.center}
Adding users to user list "prods-in-alphabetical-order-userlist”
Click on the New feature flag button on the top right hand side of your screen.
In the New feature flag window, enter “prods-in-alphabetical-order-ff”.
In the Strategies section of the New feature flag window, there should already be sub-sections for Type and Environments.
For Type, select Percent rollout from the dropdown menu.
For Percentage, enter 50 in the field.
For Based on, ensure that Available ID is selected from the popdown menu.
For Environments, click on the + sign and select the production environment.
Click on the Add strategy button on the right hand side of the Strategies section. A new sub-section for another strategy will appear.
For Type, select User List from the dropdown menu.
For User List, select the user list prods-in-alphabetical-order-userlist.
For Environments, click on the + sign and select the staging environment.
Click on Create feature flag button at the bottom of your screen to complete the creation of the feature flag.
{:
.shadow.medium.center}
Defining the feature flag with its strategies for strating and production environments
In order for developers to instrument their code for this feature flag, you need to share with them the following information:
On the Feature flags window, click on the Configure button on the top right hand side of your screen.
Copy and save the values of API URL (URL where the client application connects to get a list of feature flags) and Instance ID (unique token that authorizes the retrieval of the feature flags). These are the two values that you will need for feature flag instrumentation.
{:
.shadow.medium.center}
Copy and save the values for the feature flag API URL and Instance ID
| Variable Key | Variable Value | Variable Type | Environment Scope | Flag - Protect variable | Flag - Mask variable
----------- | ----------- | ----------- | ----------- | ----------- |
---|
| K8S_SECRET_UNLEASH_URL | <saved API URL value> | Variable | All (default) | unchecked | unchecked
| K8S_SECRET_UNLEASH_INSTANCE_ID | <saved Instance ID value> | Variable | All (default) | unchecked | unchecked
{:
.shadow.medium.center}
Adding variable K8S_SECRET_UNLEASH_URL to project "prodmgr"
{:
.shadow.medium.center}
Adding variable K8S_SECRET_UNLEASH_INSTANCE_ID to project "prodmgr"
These two variables contain values that will be passed to your application (via the K8S_SECRET_ keyword) so that it can make use of the feature flags defined and managed by GitLab.
In order for your application to be able to use feature flags, you need to instrument your application with our Feature Flags framework. Let's see how you do this in the sample Java application.
In this example, we are using the Java client for Unleash but if you’re using a different programming language then you need to use the client library for your language. To get all the supported languages, refer to the Unleash documentation or Unleash open source project.
In project “prodmgr”, navigate to the directory
src/main/java/csaa/jspring/ProductManager
.
Click on the file name “AppController.java” to view its contents and then click on the Edit button to enter edit mode.
You will see a few code blocks that have been commented out and are preceded by the line:
// Uncomment block below to instrument Feature Flag
Uncomment all the code blocks under each of the lines indicated above.
{:
.shadow.medium.center}
Partial view of AppController.java file with uncommented code blocks
Commit the changes to the main branch.
The commit starts a pipeline that deploys the application to the staging environment. Head to Build > Pipelines and click on the most recently executed pipeline (should be the first one in the list). Click on the pipeline to display it and wait until the staging job finishes. Then deploy the application to production by clicking on “rollout 100%” job.
Now that the application is running in the staging and production environments, let’s see the feature flag in action.
Now let's check how the feature flag is working.
In project “prodmgr”, click on Operate > Environments to see the list of all environments. Then click on the "Open live environment" button for the staging environment.
A new browser tab will appear and will display a login screen. If your browser complains about the connection being insecure, accept the risk and open the browser tab.
Remember that the feature flag strategy for staging is based on the user list containing michael and mary in it. Let’s try logging in as each of them.
Enter credentials [email protected] with password p33sw0rd. Verify that Michael gets a product list sorted in alphabetical order. Log out and close the browser tab to ensure that his session closes.
{:
.shadow.medium.center}
Michael gets the feature flag that orders the list of product names in alphabetical order
From the Environments window, click on the "Open live environment" button for the staging environment. Enter credentials "[email protected]" with password "p33sw0rd". Verify that mary gets a product list sorted in alphabetical order. Log out and close the browser tab to ensure that her session closes.
From the Environments window, click on the "Open live environment" button for the staging environment. This time, enter credentials for "[email protected]" with password "p33sw0rd". Verify that thomas does not get a product list sorted in alphabetical order. Log out and close the browser tab to ensure that his session closes.
{:
.shadow.medium.center}
Thomas does not get the feature flag because the product names are not ordered in alphabetical order
The steps above demonstrate that the feature flag strategy for staging successfully worked.
Click on Operate > Environments to see the list of all environments. Then click on the "Open live environment" button for the production environment.
A new browser tab will appear and will display a login screen. If your browser complains about the connection being insecure, accept the risk and open the browser tab.
Remember that the strategy in production is that the feature will be served to 50% of the users. Try logging into the web application as each of the following users keeping track of who gets the list of products sorted in alphabetical order by name and who does not:
Note: Remember to click on the "Open live environment" button for the production environment. Once you log out from each user, remember to close the browser tab to ensure that the session closes.
| Username | Password
| ----------- | ----------- |
| [email protected] | pa33w0rd
| [email protected] | pa33w0rd
| [email protected] | pa33w0rd
| [email protected] | pa33w0rd
| [email protected] | pa33w0rd
| [email protected] | pa33w0rd
Your final count should consist of three users being served the feature and three not, matching the strategy that was set for the production environment.
As changes are made to feature flags, you can track them from the audit events window.
Note: A Premium GitLab subscription is needed for viewing Audit events.
In project “prodmgr”, select Secure > Audit events from the left vertical navigation menu.
This displays all the events that have occurred in GitLab for the last thirty days. You will see that events related to updates to feature flags are listed.
{:
.shadow.medium.center}
Audit events is an auditable list of actions that have been taken againt resources
This auditing allows you to identify when and who made changes to feature flags. It can also help preempt out-of-compliance scenarios and streamline audits to avoid penalties, providing an opportunity to optimize cost, and lower risk of unscheduled production outages.
Now you know how to create and use feature flags to lower your deployment risk.
Photo by Liam Desic on Unsplash