Some of our environments in use at GitLab are split into two separate logical
deployments of components, called "stages". The two stages are "main" and "canary"
(sometimes shortened to cny).
The reason for the canary stages is to allow us to test new releases of the GitLab software in a controlled and isolated fashion, before rolling it out to all users.
The Canary stage contains code functional elements like web, container registry and git servers while sharing data elements such as sidekiq, database and file storage with the main stage of the environment. This allows UX code and most application logic code to be consumed by a smaller subset of users under real world scenarios before being made available to all users on the environment.
Currently a canary stage covers the following services
Currently a canary stage does not cover the following services (when using canary for these services, you are using the version running in the "main" stage)
As Gitaly and Praefect nodes are randomly (with weighting) chosen to store Git data, it's not entirely possible to have a clean separation of Gitaly/Praefect nodes between stages.
What this means is that when using an environment, the Gitaly/Praefect code used (depending on the repository in question, see below) may be from either stage (main or canary), regardless of what stage you are using through the web frontend, or what you have selected in next.gitlab.com. What Gitaly/Prafect node is used follows the following rules
In the production environment, the following well known groups/repos are run of
Gitaly/Praefect nodes in the canary stage exclusively
The credentials you use and permissions you have in the "canary" stage of an environment is identical to the "main" stage of the environment.
At a technical level, access to the canary stage of an environment is accomplished in one of the following ways
gitlab_canary cookie to true in whatever browser/tool
you are using to talk to the environmentThe canary stage is also automatically used in the following circumstances (with caveats)
gitlab_canary is not set which is the default for most users. If you wish to ensure your requests are never sent to
canary, manually set the gitlab_canary cookie to false in whatever
browser/tool you are using to talk to the environment or ensure that "current" is toggled on https://next.gitlab.comThe best method when using the UI is to enable the performance bar
and look at the name of the Kubernetes pod service your page. If it starts
with gitlab-cny (and has a baby chicken next to it), you are using the canary
stage. There will also be the word "next" in a green box next to the Gitlab logo
in the top left.
When looking at logs for an environment in Elasticsearch, you can add a filter
for json.stage.keyword is cny and that will only show you logs from the
canary stage of the service in the environment.
When looking at a metrics dashboard in dashboards.gitlab.net
for an environment, choose cny from the drop down for stage at the top.
As the database (where feature flags are stored) is shared between main and canary stage, enabling a feature flag following the normal chatops process) for the environment will change it for both main and canary stages of an environment.
Some examples for the most commonly used environments are as follows
Feature flags on staging and staging-canary:
/chatops run feature set feature_flag_name true --staging/chatops run feature set feature_flag_name false --stagingFeature flags on production and production-canary:
/chatops run feature set feature_flag_name true/chatops run feature set feature_flag_name falseCurrently the canary stage has no console access, you can standard console access process to access a console server in the environment running the main stage of the code only. Note that as the database is shared between stages, depending on what actions your perform in the console, it will affect the "canary" stage as well as the "main" stage.
The Infrastructure department teams can globally disable use of canary in an environment if it is deemed necessary. As this is very disruptive to the testing and deployment process, the typical process for doing starts with declaring an incident.
The chatops command to disable canary in an environment is as follows
# Disable production-canary
/chatops run canary --disable --production
# Disable staging-canary
/chatops run canary --disable
The chatops command to re-enable canary in an environment is as follows
# Disable production-canary
/chatops run canary --enable --production
# Disable staging-canary
/chatops run canary --enable