Published on: November 18, 2021
13 min read
In our third article in our GitOps series, learn how to connect a Kubernetes cluster with GitLab for pull and push-based deployments.
It is possible to use GitLab as a best-in-class GitOps tool, and this blog post series is going to show you how. These easy-to-follow tutorials will focus on different user problems, including provisioning, managing a base infrastructure, and deploying various third-party or custom applications on top of them. You can find the entire "Ultimate guide to GitOps with GitLab" tutorial series here.
This GitOps with GitLab post shows how to connect a Kubernetes cluster with GitLab for pull and push based deployments and easy security integrations. In order to do so, the following elements are required:
A Kubernetes cluster that you can access and can create new resources,
including Role
and RoleBinding
in it.
You will need kubectl
and your local environment configured to access
the beforementioned cluster.
(Optional, recommended) Terraform and a Terraform project set up as shown in the previous article to retrieve an agent registration token from GitLab.
(Optional, recommended) kpt
and kustomize
to install the Agent into
your cluster.
(Optional, quickstart) If you prefer a less "gitopsy" approach, you will
need docker
(Docker Desktop is not needed). This is simpler to follow, but
provides less control to you.
There are many ways how one can connect a cluster to GitLab:
you can set up a $KUBECONTEXT
variable manually, manage all the related
connections and use GitLab CI/CD to push changes into your cluster
you can use a 3rd party tool, like ArgoCD or Flux to get pull based deployments
you can use the legacy, certificate-based cluster integration within
GitLab in which case GitLab will manage the $KUBECONTEXT
for you and you
can get easy metrics, log and monitoring integrations
or you can use the recommended approach, the GitLab Agent for Kubernetes, to have pull and push based deployment support, network security policy integrations and the possibility of metrics and monitoring too
We are going to focus on the Agent-based setup here as we believe that it serves and will serve our users best, hopefully you included.
The Agent has a component that needs to be installed into your cluster. We
call this component agentk
. Once agentk
is installed it reaches out to
GitLab, and authenticates itself with an access token. So, the first step is
to get a token from GitLab. We call this step "the Agent registration." If
the authentication succeeds, agentk
sets up a bidirectional GRPC channel
between itself and GitLab. The emphasis here is on "bidirectional." This
enables requests and messages to be sent by either side and provides the
possibility of much deeper integrations than the other approaches while
still being a nice citizen within your cluster.
Once the connection is established, the Agent retrieves its own
configuration from GitLab. This configuration is a config.yaml
file under
a repository, and you actually register the location of this configuration
file when you register a new Agent. The configuration describes the various
capabilities enabled of an Agent.
On the GitLab side, agentk
communicates with - what we call - the
Kubernetes Agent Server, or kas
. As most users do not have to deal with
setting up kas
, I won't write about it here. You need to be a GitLab
administrator to set up and manage
kas
. If you
are on gitlab.com, kas
is available to you at kas.gitlab.com
, thanks to
our amazing SRE team.
So the steps we are going to take in this article are the following:
Create a configuration file for the Agent
Register the Agent and retrieve its authentication token
Install agentk
into the cluster together with the token
Finally, we will set up an example pull-based deployment just to test that everything worked as expected. Let's get started!
We recommend having a separate Agent registered at least against each of
your environments. If you have multiple clusters, have at least one agent
registered with each cluster. While it is possible to have many agentk
deployments with the same authentication token and thus configuration file,
this is not supported and might lead to syncronization problems!
The different agent configurations can use the same Kubernetes manifests for deployments. So maintaining a multi-region cluster where all the clusters should be identical does not require much effort.
We designed agentk
to be very lightweight so you should not worry about
deploying multiple instances of it into a cluster.
We know users who use separate agentk
instances by squad for example. In
these situations, the squad
owns some namespaces in the cluster and each
Agent can access only the namespaces available for their squad. This way
agentk
is not just a good citizen in your cluster, but is like a team
member in your squad.
Note:
You can use either the Terraform project from the previous step or start with a new project. I will assume that we build on top of the Terraform setup from the previous article, linked above, that will come in handy when we want to register the Agent using Terraform. I won't go through setting up all the environment variables here for local Terraform run.
Decide about your agent name, and create an empty file in your project under
.gitlab/agents/<your agent name>/config.yaml
. Nota bene, that the
extension is yaml
not yml
and your agent name must follow the DNS label
standard from RFC
1123.
I'll call my agent demo-agent
, so the file is under
.gitlab/demo-agent/config.yaml
.
The next step is to register the Agent with GitLab. You can do this either through the GitLab UI or using Terraform. I will show you both approaches.
Once the configuration file is in place, visit Infrastructure/Kubernetes
and add a new cluster using the Agent. A dialog will pop up where you can
select your agent.
Once you hit "next," you will see the registration token and a docker
command for easy installation. The docker
command includes the token too
and you can run it to quickly set up an agentk
inside of your cluster.
(You might need to create a namespace first!) Feel free to run the command
for a quickstart or follow the tutorial for a truly code-based approach.
We will use Terraform to register the Agent through code. Let's create the following files:
terraform/gitlab-agent/main.tf
terraform {
backend "http" {
}
required_version = ">= 0.13"
required_providers {
gitlab = {
source = "gitlabhq/gitlab"
version = "~>3.6.0"
}
}
}
provider "gitlab" {
token = var.gitlab_password
}
module "gitlab_kubernetes_agent_registration" {
source = "gitlab.com/gitlab-org/kubernetes-agent-terraform-register-agent/local"
version = "0.0.2"
gitlab_project_id = var.gitlab_project_id
gitlab_username = var.gitlab_username
gitlab_password = var.gitlab_password
gitlab_graphql_api_url = var.gitlab_graphql_api_url
agent_name = var.agent_name
token_name = var.token_name
token_description = var.token_description
}
As you can see we will use a module here. The module is hosted using the Terraform registry provided by GitLab. You can check out the module source code here. You might have guessed correctly that under the hood the module uses the GitLab GraphQL API to register the agent and retrieve a token. We will need to set up variables for it to work.
terraform/gitlab-agent/variables.tf
variable "gitlab_project_id" {
type = string
}
variable "gitlab_username" {
type = string
}
variable "gitlab_password" {
type = string
}
variable "agent_name" {
type = string
}
variable "token_name" {
type = string
default = "kas-token"
}
variable "token_description" {
type = string
default = "Token for KAS Agent Authentication"
}
variable "gitlab_graphql_api_url" {
type = string
default = "https://gitlab.com/api/graphql"
}
terraform/gitlab-agent/outputs.tf
output "agent_id" {
value = module.gitlab_kubernetes_agent_registration.agent_id
}
output "token_secret" {
value = module.gitlab_kubernetes_agent_registration.token_secret
sensitive = true
}
Once the registration is over, you'll be able to retrieve the agent ID and the token using these Terraform outputs.
Once the above code is in place, we need to run it to actually register the Agent. Here, I am going to extend the setup from the previous article.
terraform/gitlab-agent/.envrc
as you did for the network
project.
export TF_STATE_NAME=${PWD##*terraform/}
source_env ../../.main.env
Now run Terraform
terraform init
terraform plan
terraform apply
Extend the .gitlab-ci.yml
file with the following 3 jobs:
gitlab-agent:init:
extends: .terraform:init
stage: init
variables:
TF_ROOT: terraform/gitlab-agent
TF_STATE_NAME: gitlab-agent
only:
changes:
- "terraform/gitlab-agent/*"
gitlab-agent:review:
extends: .terraform:build
stage: build
variables:
TF_ROOT: terraform/gitlab-agent
TF_STATE_NAME: gitlab-agent
resource_group: tf:gitlab-agent
only:
changes:
- "terraform/gitlab-agent/*"
gitlab-agent:deploy:
extends: .terraform:deploy
stage: deploy
variables:
TF_ROOT: terraform/gitlab-agent
TF_STATE_NAME: gitlab-agent
resource_group: tf:gitlab-agent
environment:
name: demo-agent
when: manual
only:
changes:
- "terraform/gitlab-agent/*"
variables:
- $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
As you can see these are the same jobs that we saw already, they are just
parameterized for the gitlab-agent
terraform project.
Nota bene, even if you use GitLab to register the Agent, you will need your
command line to install agentk
for the first time! As a result, you can
not avoid a local setup as you will need to run at least terraform output
to retrieve the token!
agentk
In this tutorial we are going to follow the advanced installation
instructions
from the GitLab documentation. This approach is highly customizable using
kustomize
and kpt
.
First, let's retrieve the basic Kubernetes resource definitions using kpt
:
Create a directory packages
using mkdir packages
Run kpt pkg get https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent.git/build/deployment/gitlab-agent packages/gitlab-agent
This will retrieve the most recent version of the agentk
installation
resources. You can request a tagged version with the well-known @
syntax,
for example by running kpt pkg get https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent.git/build/deployment/[email protected] packages/gitlab-agent
. You can see all the available versions
here.
kpt
- could we make this a box?
The choice of kpt
is because it allows sane upstream package management to
you. With kpt
you will be able to regularly update your packages using
something like kpt pkg update packages/gitlab-agent@<new version> --strategy=resource-merge
. It basically allows you to modify your package
locally, and will try to merge upstream changes into it. Read the kpt pkg update -h
output for more information and alternative merge strategies.
The kpt
packages you retrieved are actually a set up kustomize
overlays.
The base
defines only the agentk
deployment and namespace; the cluster
defines some default RBAC around the deployment. Feel free to add your own
overlays and use those. We will extend this package with custom overlays in
a part 6 of the series.
To configure the package, see the available configuration options using:
kustomize cfg list-setters packages/gitlab-agent
NAME VALUE SET BY DESCRIPTION COUNT REQUIRED IS SET
agent-version stable package-default Image tag for agentk container 1 No No
kas-address wss://kas.gitlab.com package-default kas address. Use 1 No No
grpc://host.docker.internal:8150
if connecting from within Docker
e.g. from kind.
name-prefix Prefix for resource names 1 No No
namespace gitlab-agent package-default Namespace to install GitLab 2 No No
Kubernetes Agent into
prometheus-scrape true package-default Enable or disable Prometheus 1 No No
scraping of agentk metrics.
The package default will be different if you used a tagged version for
getting the package. Let's set the version as using stable
is not
recommended.
kustomize cfg set packages/gitlab-agent agent-version v14.4.1
set 1 field(s) of setter "agent-version" to value "v14.4.1"
Feel free to adjust the other configuration options too or add you own overlays if that is needed.
If possible the version of agentk
should match the major and minor version
of your GitLab instance. You can find our the version of your GitLab
instance under the Help menu on the UI.
If there is no agent version with your major and minor version, then pick the agent with the highest major and minor below the version of your GitLab.
Warning:
Before the next step, I want to warn you about never, ever committing unencrypted secrets into git, and the agent registration token is a secret!
Let's retrieve the agent registration token from our Terraform project. Run
the following command in the terraform/gitlab-agent
directory:
terraform output -raw token_secret >
../../packages/gitlab-agent/base/secrets/agent.token
This writes the registration token to a file on your local computer. Do not commit these changes to git!
At this point, we are ready to deploy agentk
into the cluster, so run:
kustomize build packages/gitlab-agent/cluster | kubectl apply -f -
Let's get rid of the secret:
echo "Invalid token" > packages/gitlab-agent/base/secrets/agent.token
You are good to commit your changes to git
now!
We have installed the Agent, now what? How can we start using it? In the
next article we will see in detail how to deploy a more serious application
into the cluster. Still, to check that cluster syncronization actually
works, let's deploy a ConfigMap
.
kubernetes/test_config.yaml
with the following content:
apiVersion: v1
kind: ConfigMap
metadata:
name: gitlab-gitops
namespace: default
data:
key: It works!
.gitlab/demo-agent/config.yaml
, and add the following to it:
gitops:
# Manifest projects are watched by the agent. Whenever a project changes,
# GitLab deploys the changes using the agent.
manifest_projects:
- id: path/to/your/project
default_namespace: gitlab-agent
# Paths inside of the repository to scan for manifest files.
# Directories with names starting with a dot are ignored.
paths:
- glob: 'kubernetes/test_config.yaml'
#- glob: 'kubernetes/**/*.yaml'
Change the - id: path/to/your/project
line above to point to your
project's path!
The above configuration tells the Agent to kepp the
kubernetes/test_config.yaml
file in sync with the cluster. I've left a
commented line at the end to show how you could use wildcards. This will
come handy in future steps of this article. Thedefault_namespace
is used
if no namespace is provided in the Kuberentes manifests. There are many
other options to configure as well even for the gitops
use case. You can
read more about these in the configuration file reference
documentation.
Once you commit the above changes, GitLab notifies agentk
about the
changed files. First, agentk
updates its configuration; second, it
retrieves the ConfigMap
.
Wait a few seconds, and run kubectl describe configmap gitlab-gitops
to
check that the changes got appliedd to your cluster. You should see
something similar:
Name: gitlab-gitops
Namespace: default
Labels: <none>
Annotations: config.k8s.io/owning-inventory: 502-28431043
k8s-agent.gitlab.com/managed-object: managed
Data
====
key: