Simple Kubernetes management with GitLab

Noah Ing ยท
Nov 15, 2022 ยท 6 min read ยท Leave a comment
Noah Ing GitLab profile

Kubernetes can be very complex and has dozens of tutorials out there on how to provision and manage a cluster. This tutorial aims to provide a simple, lightweight solution to provision a Kubernetes cluster and manage it with infrastructure as code (IaC) using Terraform and Helm in 20 minutes or less.

The final product of this tutorial will be two IaC repositories with fully functional CI/CD pipelines:

  1. gitlab-terraform-k8s - A single source of truth to provision, configure, and manage your Kubernetes infrastructure using Terraform
  2. cluster-management - A single source of truth to define the desired state of your Kubernetes cluster using the GitLab Agent for Kubernetes and Helm

Final Product

Prerequisites

An overview of this tutorial is as follows:

  1. Set up the GitLab Terraform Kubernetes Template ๐Ÿ—๏ธ
  2. Register the GitLab Agent ๐Ÿ•ต๏ธ
  3. Add in Cloud Credentials โ˜๏ธ๐Ÿ”‘
  4. Set up the Kubernetes Cluster Management Template ๐Ÿšง
  5. Enjoy your Kubernetes Cluster completely managed in code! ๐Ÿ‘

Set up the GitLab Terraform Kubernetes Template

Start by importing the example project by URL - https://gitlab.com/projects/new#import_project

To import the project:

  1. In GitLab, on the top bar, select Main menu > Projects > View all projects.
  2. On the right of the page, select New project.
  3. Select Import project.
  4. Select Repository by URL.
  5. For the Git repository URL:
  6. Complete the fields and select Create project.

Register the GitLab Agent

With your newly created gitlab-terraform-k8s repo, create a GitLab Agent for Kubernetes:

  1. On the left sidebar, select Infrastructure > Kubernetes clusters. Select Connect a cluster (agent).
  2. From the Select an agent dropdown list, select eks-agent/gke-agent and select **Register an agent.
  3. GitLab generates a registration token for the agent. Securely store this secret token, as you will need it later.
  4. GitLab provides an address for the agent server (KAS). Securely store this as you will also need it later.
  5. Add this to the gitlab-terraform-eks/.gitlab/agents/eks-agent/config.yaml in order to allow the GitLab Agent to have access to your entire group.
ci_access:
  groups:
    - id: your-namespace-here

Register GitLab Agent

Add in your Cloud Credentials to CI/CD variables

AWS EKS

On the left sidebar, select Settings > CI/CD. Expand Variables.

  1. Set the variable AWS_ACCESS_KEY_ID to your AWS access key ID.
  2. Set the variable AWS_SECRET_ACCESS_KEY to your AWS secret access key.
  3. Set the variable TF_VAR_agent_token to the agent token displayed in the previous task.
  4. Set the variable TF_VAR_kas_address to the agent server address displayed in the previous task.

Add in CI/CD variables

GCP GKE

  1. To authenticate GCP with GitLab, create a GCP service account with the following roles: Compute Network Viewer, Kubernetes Engine Admin, Service Account User, and Service Account Admin. Both User and Admin service accounts are necessary. The User role impersonates the default service account when creating the node pool. The Admin role creates a service account in the kube-system namespace.
  2. Download the JSON file with the service account key you created in the previous step.
  3. On your computer, encode the JSON file to base64 (replace /path/to/sa-key.json to the path to your key):
base64 /path/to/sa-key.json | tr -d

On the left sidebar, select Settings > CI/CD. Expand Variables.

  1. Set the variable BASE64_GOOGLE_CREDENTIALS to the base64 encoded JSON file you just created.
  2. Set the variable TF_VAR_gcp_project to your GCPโ€™s project name.
  3. Set the variable TF_VAR_agent_token to the agent token displayed in the previous task.
  4. Set the variable TF_VAR_kas_address to the agent server address displayed in the previous task.

Run GitLab CI to deploy your Kubernetes cluster!

Deploy Kubernetes cluster

When successfully completed, view the cluster in the AWS/GCP console!

AWS EKS

You are halfway done! ๐Ÿ‘ Keep it up!

Set up the Kubernetes Cluster Management Project

Create a project from the cluster management project template - https://gitlab.com/projects/new#create_from_template

  1. In GitLab, on the top bar, select Main menu > Projects > View all projects.
  2. On the right of the page, select New project.
  3. Select Create from template.
  4. From the list of templates, next to GitLab Cluster Management, select Use template.
  5. Enter the project details. Ensure this project is created in the same namespace as the gitlab-terraform-k8s project.
  6. Select Create project.
  7. Once the project is created on the left sidebar, select Settings > CI/CD. Expand Variables.
  8. Set the variable KUBE_CONTEXT to point to the GitLab Agent. For example, noah-ing-demos/infrastructure/gitlab-terraform-eks:eks-agent.

Set Kube Context

Uncomment Applications in helmfile

That will trigger your CI/CD pipeline and it should look like this.

Cluster Management CI/CD

Once completed, go to the AWS/GCP console and check out all the deployed resources!

Deployed EKS applications

Voila! ๐ŸŽ‰

Enjoy your Kubernetes cluster completely defined in code! ๐Ÿ‘๐Ÿ‘๐Ÿ‘

Now with these two repositories you can manage a Kubernetes cluster entirely through code:

โžก๏ธ Bonus: If you'd like to deploy your own application to the Kubernetes cluster, then add to your cluster-management helmfile and see the GitLab Agent for Kubernetes roll it out with ease!

References

โ€œFollow our tutorial to provision a Kubernetes cluster with GitLab and manage it with IAC using Terraform and Helm in 20 minutes or less.โ€ โ€“ Noah Ing

Click to tweet

Open in Web IDE View source