GitOps with GitLab: Connecting GitLab with a Kubernetes cluster for GitOps-style application delivery

It is possible to use GitLab as a best-in-class GitOps tool, and this blog post series is going to show you how. These easy-to-follow tutorials will focus on different user problems, including provisioning, managing a base infrastructure, and deploying various third-party or custom applications on top of them. You can find the entire "Ultimate guide to GitOps with GitLab" tutorial series here.

In this article, we will look at how to connect an application project to a manifest project for controlled, GitOps-style deployments.

Prerequisites

This article builds upon the previous tutorials in this series. We will assume that you have a Kubernetes cluster connected to GitLab using the GitLab Agent for Kubernetes, and that you understand the basics of GitLab CI/CD.

If this is not the case, I recommend following the previous articles to have a similar setup from where we will start today.

A common setup

Many users prefer to separate application code from infrastructure code, and the manifests describing the application deployments are considered infrastructure code. As mentioned above, this tutorial shows how to connect an application repository to a manifest repository to achieve GitOps-style deployments with GitLab.

The plan

The plan for this article is to build and deploy a minimal application. The focus will be on the deployment aspect.

We will deploy a simple "Hello World" application. Our pipeline will build a Docker container and the deployment will span two environments, integrated into GitLab.

You can see the final repository for the application and the manifest repository.

The application

In this section, we will create our super simple "Hello World" application and put a Dockerfile beside it. If you read the previous article on how to use Auto DevOps with the GitLab Agent for Kubernetes, this setup will be very familiar to you.

1. Start a new project.
2. Add src/main.py with the following content:

   # From https://gist.github.com/davidbgk/b10113c3779b8388e96e6d0c44e03a74
   import http.server
   import socketserver
   from http import HTTPStatus
   
   class Handler(http.server.SimpleHTTPRequestHandler):
       def do_GET(self):
           self.send_response(HTTPStatus.OK)
           self.end_headers()
           self.wfile.write(b'Hello world')
   
   
   httpd = socketserver.TCPServer(('', 5000), Handler)
   httpd.serve_forever()
   

3. Create the Dockerfile with:

   FROM python:3.9.10-slim-bullseye
   
   WORKDIR /app
   
   COPY ./src .
   
   EXPOSE 5000
   
   CMD [ "python", "main.py" ]
   

4. Let's build the Docker container using GitLab CI/CD. Extend your .gitlab-ci.yml with the following job:

   kaniko-build:
      image:
         # For latest releases see https://github.com/GoogleContainerTools/kaniko/releases
         # Only debug/*-debug versions of the Kaniko image are known to work within Gitlab CI
         name: gcr.io/kaniko-project/executor:debug
         entrypoint: [""]
      variables:
         # The Dockerfile to build
         DOCKERFILE: Dockerfile
         IMAGE_NAME: "hello-world"
         VERSION: $CI_COMMIT_SHORT_SHA
         KANIKO_ARGS: ""
      script:
         - mkdir -p /kaniko/.docker
         # Write credentials to access Gitlab Container Registry within the runner/ci
         - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
         # Build and push the container. To disable push add --no-push
         - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE/${IMAGE_NAME}:$VERSION $KANIKO_ARGS
   

   Note the KANIKO_ARGS variable above. It's worth reading the Kaniko docs to further optimize the build. For example, large builds can be much faster using something like --cache=true --cache-copy-layers --cache-repo ${CI_REGISTRY_IMAGE}/caches.
5. Commit the change to the repository.

Configure cluster to access GitLab Registry images

If you followed the previous tutorials, the manifests repository should already exist. We will expand that repository.

As we want to deploy containers into the cluster, we should instruct the cluster how to authenticate with the GitLab container registry. This can be achieved by setting up some secrets within the cluster.

This section builds on the fourth article in the series, where we set up Bitnami's Sealed Secrets to manage Kubernetes Secrets in a GitOps way.

1. Create a Personal access token or a Project access token with read registry rights. Take note of the token.
2. Run the following command to create a Secret manifest:

   kubectl create secret docker-registry gitlab-credentials --docker-server=registry.gitlab.com --docker-username=<GitLab username> --docker-password=<token from the previous step> --docker-email=<your e-mail address> -n gitlab-agent --dry-run=client -o yaml > ignored/gitlab-credentials.yaml
   

3. Encrypt the secret:

   kubeseal --format=yaml --cert=sealed-secrets.pub.pem < ignored/gitlab-credentials.yaml > kubernetes/
   

4. Commit the changes.

As a quick recap, note that we specified the namespace when we created the Secret manifest. Bitnami's Sealed Secrets are namespace scoped. Feel free to change the namespace in the unencrypted Secret manifest, but do not change it in the encrypted one.

Now, we are ready to orchestrate the application deployment.

Setting up manifests

We will use Kustomize to describe the deployments. You can use Helm as well, if you prefer, with minor modifications to the code.

1. Start a new Kustomize base:

   mkdir -p packages/hello-world/base
   cd packages/hello-world/base
   

2. Create a deployment.yaml under packages/hello-world/base:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
   name: hello-world
   spec:
   selector:
     matchLabels:
     app: hello-world
   template:
     metadata:
     labels:
        app: hello-world
     spec:
     imagePullSecrets:
        - name: gitlab-credentials
     containers:
        - name: hello-world
          image: registry.gitlab.com/gitlab-examples/ops/gitops-demo/hello-world-service-gitops/hello-world:38adc854
          imagePullPolicy: Always
          resources: {}
          # limits:
          #   memory: "128Mi"
          #   cpu: "500m"
          livenessProbe:
          httpGet:
            path: /
            port: 5000
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 10
          ports:
            - containerPort: 5000
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
            drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
     securityContext: {}
   

3. Create the Kustomization definition file:

   kustomize init --autodetect
   

The above files create a Kustomize based, now we want to create a production and a staging overlay. The staging overlay will not modify the base, while the production overlay with have more replicas.

1. Create the Kustomize overlay for the "staging" environment.

   cd ..
   mkdir staging
   cd staging
   kustomize init --namesuffix -staging --resources ../base
   

2. Create the Kustomize overlay for the "production" environment.

   cd ..
   mkdir production
   cd production
   kustomize init --namesuffix -prod --resources ../base
   cat <<EOF >> ./kustomization.yaml
   replicas:
   name: hello-world
     count: 3
   EOF
   

3. Commit and push the changes.

These files describe the deployments into the "staging" and "production" environments. We will have to render them into vanilla Kubernetes manifests for the agent.

Hydrating the manifests

To hydrate the Kustomize overlays into Kubernetes manifests we need to call the kustomize build ... command. We can do this locally and commit the changes or we can do it in GitLab CI/CD and commit the changes automatically. The latter approach works great as it minimizes human error and still provides great flexibility. Let's see how to do it!

1. Extend your .gitlab-ci.yml with the following job:

   This job hydrates the manifests and stores the hydrated manifests as artifacts. The next job will pick up these artifacts and commit them back to the repository, but first we need some preparations.

2. Create a Project access token for the Manifest project with read_repository, write_repository rights and save it as a "masked" and "protected" Environment variable under the manifest project's "Settings > CI/CD" page. Name the variable GITLAB_TOKEN.

3. Add a job to Git commit and push the changes. Add the following file under .gitlab/ci-templates .

4. Reference the downloaded file at the top of .gitlab-ci.yml and add the "commit&push" job:

The update-packages job will grab the hydrated manifests from the hydrate-packages job and push them back to the repository. Once there, the GitLab agent for Kubernetes can pick them up and deploy them into your repository.

Connecting the two repositories

Now, we have the Docker containers waiting in the application repository and the manifests waiting for those containers. Let's connect the two!

First, we should update the CI/CD jobs of the applications repository. Head over to the application repository for the following changes and edit the .gitlab-ci.yml file.

1. Extend the kaniko-build job:

   kaniko-build:
     ...
     script:
        ...
        - |
          cat << EOF  > deploy.env
          IMAGE_REF=$CI_REGISTRY_IMAGE/${IMAGE_NAME}:$VERSION
          EOF
     artifacts:
       reports:
         dotenv: deploy.env
   

   This added one more command to the scripts to create a deploy.env file and takes that file as an artifact.

2. Create the deploy:staging job by adding the following to your file:

   Replace <Here comes your manifest project ID> with the ID of your manifest project. You can find that ID on the project homepage. Note that this should be the ID of the manifest project, not the application project!

3. Create a similar job for the production deployments:

Next, we need to handle the triggered pipeline in the manifest repository. Head over to the manifest repository for the following changes:

1. Extend .gitlab-ci.yml with the three jobs from:

As you can see, these jobs will run only when the pipeline is triggered by another pipeline. The first job takes the variables passed from the application repository and converts them to a format that's easy to consume for the other jobs. The second job updates the respective Kustomize overlay. The third job commits and pushes the changes back to the repository.

It's interesting to note how these changes integrate into the previous jobs in the manifest CI/CD pipeline. Once the image tag is updated and the changes are pushed back to the repository, a new pipeline will be triggered that hydrated the changes and pushes the hydrated manifests back to the repo.

Recap

This article showed how to connect an application repository with a manifest repository. In the process, we used Kustomize to have different variants for a "staging" and a "production" deployment. One can use Helm or any other tool following the same logic.

What is next

In the final article, I will show you how to use the techniques already presented to manage a deployment of the GitLab agent for Kubernetes using the deployment of that same agent.

Click here for the next tutorial.