There is a lot of flexibility when it comes to defining and using variables for CI/CD. Variables are extremely useful for controlling jobs and pipelines, and they help you avoid hard-coding values in your .gitlab-ci.yml configuration file. The information in this post should weave a larger picture by bringing together all (or most) of the information around defining and handling variables, making it easier to understand the scope and capabilities. Relevant documentation is linked throughout the post.
In GitLab CI/CD, variables can be used to customize jobs by defining and storing values. When using variables there is no need to hard code values. In GitLab, CI/CD variables can be defined by going to Settings >> CI/CD >> Variables, or by simply defining them in the .gitlab-ci.yml file.
Variables are useful for configuring third-party services for different deployment environments, such as testing, staging, production, etc. Modify the services attached to those environments by simply changing the variable that points to the API endpoint the services need to use. Also use variables to configure jobs and then make them available as environment variables within the jobs when they run.
The relationship between variables and environments
Software development as a process includes stages to test a product before rolling it out to users. Environments are used to define what those stages look like and it may differ between teams and organizations.
On the other hand, variables are data values that are likely to change as a result of user interaction with a product. For example, their age, preference, or any input you could possibly think of that might determine their next step in the product task-flow.
We often hear the term environment variable. These are variables that are defined in a given environment, but outside the application. GitLab CI/CD variables provide developers with the ability to configure values in their code. Using variables is helpful because it ensures that the code is flexible. GitLab CI/CD variables allow users to modify an application deployed to a certain environment without making any change to code. It is simple to run tests or even integrate third-party services by changing a configuration environment variable outside the application.
The scope of variables for CI/CD
.gitlab-ci.yml defined variables
Variables that need to be available in the job environment can be added to GitLab. These CI/CD variables are meant to store non-sensitive project configuration, like the database URL in the .gitlab-ci.yml file. Reuse this variable in multiple jobs or scripts, wherever the value is needed. If the value changes, you only need to update the variable once, and the change is reflected everywhere the variable is used.
Project CI/CD variables
Moving a step above the repository-specific requirements, you can define CI/CD variables in project settings, which makes them available to CI/CD pipelines. These are stored out of the repository (not in the .gitlab-ci.yml file), but are still available to use in the CI/CD configuration and scripts. Storing the variables outside the .gitlab-ci.yml file keeps these values limited to a project-only scope, and not saved in plain text in the project.
Group and instance CI/CD variables
Some variables are relevant at the group level, or even instance level, and could be useful to all projects in a group or instance. Define the variables in the group or instance settings so all projects within those scopes can use the variables without actually needing to know the value or having to create the variables for the lower scope. For example, a common value that needs to be updated in multiple projects can be easily managed if it stays up-to-date in a single place. Alternatively, multiple projects could use a specific password without actually needing to know the value of the password itself.
Jobs and pipelines as environments
GitLab CI/CD variables, besides being used as environment variables, also work in the scope of the .gitlab-ci.yml configuration file to configure pipeline behavior, unrelated to any environment. The variables can be stored in the project/group/instance settings and be made available to jobs in pipelines.
For example:
job:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
script:
- echo "This job ran on the $CI_COMMIT_BRANCH branch."
The variable ($CI_COMMIT_BRANCH) in the script section runs in the scope of the job in which it was defined. This scope is the "job environment" – meaning, when the job starts, the GitLab runner starts up a Docker container and runs the job in that environment. The runner will make that variable (and all other predefined or custom variables) available to the job, and it can display their value in the log output if needed.
But the variable is also used in the if: section to determine when the job should run. That in itself is not an environment, which is why we call these CI/CD variables. They can be used to dynamically configure your CI/CD jobs, as well as be used as environment variables when the job is running.
Predefined variables
A number of variables are predefined when a GitLab CI/CD pipeline starts. A user can immediately access values for things like commit, project, or pipeline details without needing to define the variables themselves.
Custom CI/CD variables
When creating a CI/CD variable in the settings, GitLab gives the user more configuration options for the variable. Use these extra configuration options for stricter control over more sensitive variables:
Environment scope: If a variable only ever needs to be used in one specific environment, set it to only ever be available in that environment. For example, you can set a deploy token to only be available in the production environment.
Protected variables: Similar to the environment scope, you can set a variable to be available only when the pipeline runs on a protected branch, like your default branch.
Variable type: A few applications require configuration to be passed to it in the form of a file. If a user has an application that requires this configuration, just set the type of variable as a "File". Configuring the CI/CD variable this way means that when the runner makes the variable available in the environment, it actually writes it out to a temporary file, and stores the path to the file as the value. Next, a user can pass the path to the file to any applications that need it.
Along with the listed ways of defining and using variables, GitLab introduced a feature that generates pre-filled variables when there's a need to run a pipeline manually. Prefilled variables reduce the chances of running into an error and makes running the pipeline easier.
Masked variables: Masked variables are CI variables that have been hidden in job logs to prevent the variable’s value from being displayed.
Masked and hidden variables: Introduced in GitLab 17.4, Masked and hidden variables provide the same masking feature from job logs and keep the value hidden in the Settings UI. We do not recommend using either of these variables for sensitive data (e.g. secrets) as they can be inadvertently exposed.
Secrets
A secret is a sensitive credential that should be kept confidential. Examples of a secret include:
- Passwords
- SSH keys
- Access tokens
- Any other types of credentials where exposure would be harmful to an organization
GitLab currently enables its users to use external secrets in CI, by leveraging HashiCorp Vault, Google Cloud Secret Manager, and Azure Key Vault to securely manage keys, tokens, and other secrets at the project level. This allows users to separate these secrets from other CI/CD variables for security reasons.
GitLab Secrets Manager
Besides providing support for external secrets in CI, GitLab is also working on introducing a native solution to secrets management to securely and conveniently store secrets within GitLab. This solution will also help customers use the stored secrets in GitLab specific components and environments, and easily manage access at namespace groups and projects level.
Read more
Disclaimer: This blog contains information related to upcoming products, features, and functionality. It is important to note that the information in this blog post is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. As with all projects, the items mentioned in this blog and linked pages are subject to change or delay. The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab.
