Getting [meta] with GitLab CI/CD: Building build images

An alternative title for this post could have been:

I heard you liked Docker, so I put dind.

Getting started

It should be clear by now that I love building stuff with GitLab CI/CD. From DNS to breakfast GitLab CI/CD offers a pretty wide range. However, past those "fun" use cases, I also like to share some best practices I have acquired during my years of using GitLab CI/CD, both for software and non-software projects alike.

I crossed out "best" above because I don't really like the term "best practices." It implies that there is only one right answer to a given question – which is the opposite of the point of computer science. Sure there are better and worse ways to do something – but like many things in life, you have to find what works for you. "The best camera is the one you have with you" comes to mind when building CI/CD for projects. Something that works is better than something that's pretty.

But, enough of my digression, let's get to the practice I wanted to share in this post: Building build images as part of the build process. Yes, it is precisely as meta as it sounds.

Why?

Often when building a particular project, you may have several unique build dependencies. In many languages, package managers solve for the majority if not all of these dependencies – at least for build time (think npm, RubyGems, Maven). However, when we are building and deploying (CI/CD let's remember) from a machine that is not our own, that may not be enough. There may be a few dependencies we might need from elsewhere.

The language libraries themselves are one such dependency – to build Java I'm going to need the JDK or JRE. To build Node, I'll need... well Node, etc. In a Docker-based environment, those languages and dependencies typically have an official image on Docker Hub (JRE from Oracle or Node from Node.js for instance). Assume, however, that I may need a few other things not included in either those official Docker images or the package manager I'm using. For instance, maybe I need a CLI tool for deploy (AWS, Heroku, Firebase, etc.). We also might need a testing framework or tool like Selenium or headless Chrome. Or I may need other tools for packaging, testing, or deployment.

Sometimes there is a Docker image on Docker Hub for these combinations – or some of them – but not always a maintained version. One easy solution to this could be to just run the install of the tools before every job that needs it. This can even be "automated" using something like the before_script syntax. However, this adds time to our pipeline and seems inefficient: Is there a better way?

Enter the GitLab Docker registry

Since GitLab is a single application for the entire DevOps lifecycle – it actually ships out of the box with a built-in Docker registry. This can be a useful tool when deploying code in a containerized environment. We can build our application into a container and send it off into Kubernetes or some other Docker orchestrator.

However, I also see this registry as an opportunity to save time in my pipeline (and save round trips to Docker hub and back every time). For builds that require some of these extra dependencies, I like to build a "build" Docker image. That way, I have an image with all of those baked right in. Then, as part of my pipeline, I can build the image at the start (only when changes are made or every time). And the rest of the pipeline can consume that image as the base image.

Putting it in practice

For example, let's see what it looks like to build a simple Docker image to use with deploying to Google Firebase.

Firebase is a "backend as a service" tool that provides a database, authentication, and other services across platforms (web, iOS, and Android). It also includes web hosting and several other items that can be deployed through a CLI. This tool makes getting started really easy. You can deploy the whole stack with firebase deploy. Alternatively, you can deploy a part (like serverless functions) with a command like firebase deploy --only functions.

Making this work in a CI/CD world requires a few extra steps though. We'll need a Node Docker image that has the firebase CLI in it, so let's make a simple Dockerfile to do that.

Putting this Dockerfile in .meta/Dockerfile

FROM node:10

RUN npm install -g firebase-tools

Next, I'll add a job to the front of my pipeline.

Added to the front of my .gitlab-ci.yml

meta-build-image:
  image: docker:stable
  services:
    - docker:dind
  stage: prepare
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - cd .meta
    - docker build -t $CI_REGISTRY/group/project/buildimage:latest .
    - docker push $CI_REGISTRY/group/project/buildimage:latest
  only:
    refs:
      - main
    changes:
      - .meta/Dockerfile

Let's break down that job:

1. We use the docker:stable image and a service of docker:dind
2. The stage is my first stage called prepare
3. In the script, we login to the GitLab registry with the built-in variables and build the image. For more details see the GitLab documentation for building Docker images.
4. We only run this on main and only when the .meta/Dockerfile changes. This makes sure we are specific about when we change the Docker image. We could also use the commit hash or other methods here to make the image more fungible.

Now, in further jobs down the pipeline, I can use the latest build of the Docker image like this:

firestore:
  image: registry.gitlab.com/group/project/buildimage
  stage: deploy 🚢🇮🇹
  script:
    - firebase deploy --only firestore
  only:
    changes:
      - .firebase-config/firestore.rules
      - .firebase-config/firestore.indexes.json

In this job, we only run the job if something about the Firestore (the database from Firebase) configuration changes. And when it does, we run the firestore deploy command in CI. I also added a token for deploy as a GitLab CI/CD variable based off the Firebase documentation for using firebase with CI.

Summary

In the end, this helps speed up pipelines by ensuring that you have a custom-built build image that you control. You don't have to rely on unstable or unmaintained Docker Hub images or even have a Docker Hub account yourself to get started.

To learn more about GitLab CI/CD you can read the GitLab website or the CI/CD docs. Also, there's a lot more to learn about the GitLab Docker registry.

Cover image by Hack Capital on Unsplash.