It is possible to use GitLab as a best-in-class GitOps tool, and this blog post series is going to show you how. These easy-to-follow tutorials will focus on different user problems, including provisioning, managing a base infrastructure, and deploying various third-party or custom applications on top of them. You can find the entire "Ultimate guide to GitOps with GitLab" tutorial series here.
In this article, we will build upon the first few articles, and will turn a GitLab agent for Kubernetes installation to manage itself. This is highly recommended for production usage as it puts your
agentk deployment under your GitOps project, and enables flawless and simple upgrades.
This article builds on a few previous articles from this series and makes the following assumptions:
- You have an agent connection set up using the
- You have set up Bitnami's Sealed secrets.
- You understand how to use
kustomizewith the agent.
The goal of this tutorial is to manage a GitLab agent for Kubernetes deployment using that given agent. This has several benefits, including:
- By turning the agent to manage itself, the agent configuration and deployment is managed in code. As a result, all the code-oriented tools, including Merge Requests, Approvals, and branching are there to support your processes and policies.
- Managing a fleet of agent installations in code enables simple upgrades of the deployments.
Upgrading GitLab and the GitLab agent for Kubernetes
A single GitLab instance might have dozens of agent connections. How should you upgrade all these deployments in a coordinated way? Turning everything into code simplifies the upgrade process a lot.
We have the GitLab - Agent version compatibility documented. The recommended approach is to first upgrade GitLab together with
KAS, the GitLab-side component of the connection, and then upgrade all the
If you manage the
agentk deployments in code, the upgrade requires only bumping the version number in code and the
agentk instances will take care of upgrading themselves.
Turning an agent installation to manage itself
Let's do a quick recap and an overview how we wil use the tools.
kpt to check out tagged
agentk deployment manifests. As the manifests are a set of
kustomize layers, we can extend them with our own overlays if needed, or just customize the setup per our requirements. The agent connection requires a token to authenticate with GitLab. We can use Bitnami's Sealed Secrets to store an encrypted sycret in the repo.
All the above code can be put under version control safely. Moreover, we can use GitLab CI/CD to dehydrate the
kustomize package into vanilla Kubernetes manifests that the agent can deal with.
Let's see the above in action!
Kustomize layer with encrypted secret
Based on the previous articles, we have the
kpt package checked out under
packages/gitlab-agent. We would like to store the vanilla Kubernetes manifests in the repository. We can run
kustomize build packages/gitlab-agent/cluster > kubernetes/gitlab-agent.yaml to get the manifests, but this will include the unencrypted authentication token too.
To never output the unencrypted token, we should turn it into a sealed secret.
Navigate to the
gitlab-agent Terraform project, and create a Kubernetes secret from the token
terraform output -raw token_secret | kubectl create secret generic gitlab-agent-token -n gitlab-agent --dry-run=client --type=Opaque --from-file=token=/dev/stdin -o yaml > ../../ignored/gitlab-agent-token.yaml. If you followed the instructions in the previous articles, the files under the
ignored directory are never committed to
We will turn this unencrypted secret into a sealed secret. As the secret will already exist in the cluster, we should instruct the Bitnami Sealed Secret controller to pull it under its management. Moreover, as kustomize applies a random hash to every secret name, we should enable renaming the secret within the namespace. We can achieve these by adding two annotations to the unencrypted secrets object.
Add the following annotations to
annotations: sealedsecrets.bitnami.com/managed: "true" sealedsecrets.bitnami.com/namespace-wide: "true"
Next, we should create an encrypred secret from the ignored, unencrypted one running
bin/seal-secret ignored/gitlab-agent-token.yaml > packages/gitlab-agent/sealed-secret in the root of our project. This creates the encrypted secret under
packages/gitlab-agent/sealed-secret/SealedSecret.gitlab-agent-token.yaml. Now, we need a kustomize layer that will use this secret instead of the original one that came with
kpt. Let's create the following files around the encrypted secret:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - SealedSecret.gitlab-agent-token.yaml components: - ../cluster/components/gitops-read-all - ../cluster/components/gitops-write-all - ../cluster/components/cilium-alert-read configurations: - configuration/sealed-secret-config.yaml secretGenerator: - name: gitlab-agent-token behavior: replace type: Opaque namespace: gitlab-agent options: annotations: sealedsecrets.bitnami.com/managed: "true" sealedsecrets.bitnami.com/namespace-wide: "true"
nameReference: - kind: Secret fieldSpecs: - kind: SealedSecret path: metadata/name - kind: SealedSecret path: spec/template/metadata/name
This configuration enables us to reference the name of the Sealed Secret in the
We created a new
kustomize overlay that builds on the
cluster layers, but will use the sealed secret. We can hydrate this into vanilla manifests using
kustomize build packages/gitlab-agent/sealed-secret > kubernetes/gitlab-agent.yaml. This configuration does not include any unencrypted, sensitive data. As a result, we can commit it freely using
Adopt the agent by the agent
Right now the agent configuration file looks similar to:
gitops: # Manifest projects are watched by the agent. Whenever a project changes, # GitLab deploys the changes using the agent. manifest_projects: - id: path/to/your/project default_namespace: gitlab-agent # Paths inside of the repository to scan for manifest files. # Directories with names starting with a dot are ignored. paths: - glob: 'kubernetes/test_config.yaml' - glob: 'kubernetes/**/*.yaml'
If we would push the previously hydrated manifests,
agentk would fail applying them complaining about missing inventories. We can easily fix this by temporarily setting a looser inventory policy:
gitops: # Manifest projects are watched by the agent. Whenever a project changes, # GitLab deploys the changes using the agent. manifest_projects: - id: path/to/your/project default_namespace: gitlab-agent inventory_policy: adopt_all # Paths inside of the repository to scan for manifest files. # Directories with names starting with a dot are ignored. paths: - glob: 'kubernetes/test_config.yaml' - glob: 'kubernetes/**/*.yaml'
With the inventory policy configured, we can commit and push our changes to GitLab. The agent will see the new configuration and resources, and will apply them into the cluster. From now on, you can change the code in the repository, push it to git, and the changes will be automatically applied into your cluster.
What are inventory policies?
The GitLab agent for Kubernetes knows about the managed resources using so-called inventory objects. In technical terms, an inventory object is just a
ConfigMap with a unique label. Whenever the agent sees an object that it should manage, it applies the same label. This way, every agent can easily find the resources that it manages.
You can read more about the possible inventory policy configurations in the documentation.
A word about RBAC
Depending on the authorization rights given to the
agentk deployment, not every change might be possible. For example, if you would like to create new
ClusterRoleBinding in a new
kustomize overlay, and apply that with the Agent, that might fail. It will fail, if your current role-based access control (RBAC) does not allow your
agentk deployment to create these resources. In this case, you should either provide higher rights to your
agentk service account first or you should apply the changes manually from your command line.
Now, if you want to change something in your agent deployment, you need to take two actions:
- change the code in the
kustomize buildto hydrate the results
Let's automate the second step so you can focus on your main job only. Following the setup of a GitOps-style Auto DevOps pipeline, we need to extend the
hydrate-packages: ... script: - mkdir -p new_manifests ... - kustomize build packages/gitlab-agent/sealed-secret > new_manifests/gitlab-agent.yaml
We can re-use all the other automation as presented in the previous articles.
How to upgrade
Just to provide a practical example, let's see how we can use the above setup to easily upgrade an
agentk deployment to a newer version.
kustomize cfg set packages/gitlab-agent agent-version v14.9.1 we set the intended
agentk version to be version
14.9.1. You can commit and push this change to git, and lay back in your chair to see how the changes are being rolled out across your clusters. You can point several agent configurations at the same
kubernetes/gitlab-agent.yaml manifest, and upgrade all of them at once.
In this article we have seen:
- how to turn an Agent deployment to manage itself
- how to extend the default
kptproject with a custom
kustomizeoverlay to customize the
- how to easily upgrade a set of
- how to pull already existing objects to be managed by the Agent using inventory policies
Note: This is the final installment in this series of how to do GitOps with GitLab.