As GitLab begins it's journey to deploying GitLab.com into Kubernetes, it'll be important that we discover the best ways towards implementing the infrastructure and organizing all components and methods of deploying application configurations into this platform. This Design Documents' goal is to provide a framework for which we operate Kubernetes for GitLab.com
This document encapsulates details that are captured from the Kubernetes Deployment as well as some conversations that were had at our recent 2019 GitLab Contribute event.
Not to be confused with how we store cluster configurations as we may rely on existing terraform capabilities to build a cluster. Instead we'll have multiple types of kubernetes configurations each of which we'll need a way to determine the best way to store configurations. The types of configurations of concern:
For GitLab.com and supporting services we're going to utilize a group k8s-workloads. This group will serve as the central location for all generated or manually created configurations will be housed under here. All repos should use a shared configuration provided by the k8s-workloads. Example look of this inside of GitLab.com (or ops.gitlab.com):
gitlab-com └── gl-infra └── k8s-workloads ├── about-gitlab-com ├── forum-gitlab-com ├── gitlab-com └── monitoring-stuff
We eliminate the need for a directory per environment and rely on environment variables of CI/CD for each environment configuration, otherwise known as Environments and Deploys. We'll utilize various branches for environments following GitLab Flow as discussed in a prior design document: Infrastructure GitLab Flow
.gitlab-ci.yml being one of the key factors of this configuration. It
will allow us to define how we want things deployed. We are using simple shell
commands to run
kubectl apply on the
testapp directory. We can iterate and
make this more compliant with our needs in the future. But using this setup in
tandem with GitLab Flow, we'll be able to target the branches for the
environment as necessary. In our examples, the
master branch would deploy
items to the
staging environment, while the
production branch would deploy
production environment. We also don't automate the generation of these
files, so they are all hand crafted, or generated using some other means. This
allows us to reduce the scope of what this repo does, allows for an easier
time reviewing configuration file changes, and provides a means of iterating
towards a more desired way of generating configurations at a later time.
The more important configuration is going to be within the GitLab application itself. This is where we'll define Environments and all the associated variables assigned to those environments that will allow GitLab CI ito act upon the kubernetes configuration appropriately.
We are relying on the Kuberentes API and the
kubectl client to perform the work
we deem necessary. We can expound upon this in the future and iterate as we see
fit. But initially, we should minimally build a way to determine if an upgrade
was successful or not such that we can report a failure to the CI job. Upgrades
will happen by updating the yaml file for the associated application.
Downgrades would operate in a similar fashion.
With using simplistic
kubectl commands to apply configurations, we'll need
some way to apply changes when files are removed from a repo. This would
require further thought and discussion in order to come up with a concise
solution that works with our workflow. For the current moment in time, I would
suggest that we simply manually remove items. For this we'll need to come up
with at least a runbook such that engineers know the correct way to remove items
since this is a manual task.
Out of scope of this design document. If you want to use helm to deploy an
application, generate the files and plop them into a repo such that they can be
kubectl apply. If one chooses to hand generate the files, the same
There are certain considerations when doing so, however. Let's make ourselves aware of this in Operational Considerations
There is a strong desire to shift from our current GKMS and chef vault to using Hashicorp Vault. Existing issue infrastructure/6119 is there for discussion. Until Vault is in place, however, since we are starting with one of our minor applications, we should have a minimal set of secrets. As as method of preventing ourselves from being blocked in getting started with Kubernetes, we'll simply plop our secret configuration files in 1Password for the time being. This will stress us in times of disaster recovery, so we'll need to ensure we have the appropriate run books set up until we have this fully automated.
Doing this sounds terrible, however:
We should automate what we can from the start. The later we wait the more
difficulty we'll have with introducing automation. We should strongly consider
the use of GitLab CI/CD where we see fit and leverage scripts to minimize any
manual interaction with a Kubernetes cluster. We should limit any engineer's
ability to manually run
kubectl from their workstation as much as possible.
Depending on the above solution, we can come up with a few ways to assist with automation:
kubediffperiodically to ensure consistency between repo and what's in the cluster
The k8s-workloads takes advantage of using Environments and Deploys to handle
connecting to clusters securely and not exposing information in CI job logs. We
can start off very simple and as our needs increase, we can start to script
items that become more difficult from a quick