Getting [meta] with GitLab CI/CD: Building build images

An alternative title for this post could have been:

I heard you liked Docker, so I put dind.

Getting started

It should be clear by now that I love building stuff with GitLab CI/CD. From

DNS

to breakfast GitLab CI/CD

offers a pretty wide range. However, past those "fun" use cases, I also like

to share some best practices I have acquired during my years of using [GitLab

CI/CD](/solutions/continuous-integration/), both for software and non-software projects alike.

I crossed out "best" above because I don't really like the term "best practices." It

implies that there is only one right answer to a given question – which is the

opposite of the point of computer science. Sure there are better and worse ways to

do something – but like many things in life, you have to find what works for

you. "The best camera is the one you have with you"

comes to mind when building CI/CD for projects. Something that works is better than something that's pretty.

But, enough of my digression, let's get to the practice I wanted to share in this

post: Building build images as part of the build process. Yes, it is precisely as meta as it sounds.

Why?

Often when building a particular project, you may have several unique build dependencies.

In many languages, package managers solve for the majority if not all of these

dependencies – at least for build time (think npm, RubyGems,

Maven). However, when we are building and

deploying (CI/CD let's remember) from a machine that is not our own, that may not

be enough. There may be a few dependencies we might need from elsewhere.

The language libraries themselves are one such dependency – to build Java I'm going to need

the JDK or JRE. To build Node, I'll need... well Node, etc. In a Docker-based environment,

those languages and dependencies typically have an official image on Docker

Hub (JRE from Oracle or

Node from Node.js for instance). Assume, however, that

I may need a few other things not included in either those official Docker images or

the package manager I'm using. For instance, maybe I need a CLI tool for

deploy (AWS, Heroku,

Firebase, etc.). We also might need a testing

framework or tool like Selenium or

headless Chrome.

Or I may need other tools for packaging, testing, or deployment.

Sometimes there is a Docker image on Docker Hub for these combinations – or some of

them – but not always a maintained version. One easy solution to this could be to

just run the install of the tools before every job that needs it. This can

even be "automated" using something like

the before_script syntax.

However, this adds time to our pipeline and seems inefficient: Is there a better way?

Enter the GitLab Docker registry

Since GitLab is a single application for the entire DevOps lifecycle – it actually

ships out of the box with a built-in

Docker registry.

This can be a useful tool when deploying code in a containerized environment. We can

build our application into a container and send it off into Kubernetes or some

other Docker orchestrator.

However, I also see this registry as an opportunity to save time in my

pipeline (and save round trips to Docker hub and back every time). For builds that require

some of these extra dependencies, I like to build a "build" Docker image.

That way, I have an image with all of those baked right in. Then, as part of my

pipeline, I can build the image at the start (only when changes are made or every time).

And the rest of the pipeline can consume that image as the base image.

Putting it in practice

For example, let's see what it looks like to build a simple Docker image to use with

deploying to Google Firebase.

Firebase is a "backend as a service" tool that provides a database, authentication,

and other services across platforms (web, iOS, and Android). It also includes web hosting

and several other items that can be deployed through a CLI.

This tool makes getting started really easy. You can deploy the whole stack with

firebase deploy. Alternatively, you can deploy a part (like serverless functions)

with a command like firebase deploy --only functions.

Making this work in a CI/CD world requires a few extra steps though. We'll need a Node

Docker image that has the firebase CLI in it, so let's make a simple Dockerfile to do that.

Putting this Dockerfile in .meta/Dockerfile

FROM node:10


RUN npm install -g firebase-tools

Next, I'll add a job to the front of my pipeline.

Added to the front of my .gitlab-ci.yml

meta-build-image:
  image: docker:stable
  services:
    - docker:dind
  stage: prepare
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - cd .meta
    - docker build -t $CI_REGISTRY/group/project/buildimage:latest .
    - docker push $CI_REGISTRY/group/project/buildimage:latest
  only:
    refs:
      - main
    changes:
      - .meta/Dockerfile

Let's break down that job:

1. We use the docker:stable image and a service of docker:dind

2. The stage is my first stage called prepare

3. In the script, we login to the GitLab registry with the built-in variables and build the

image. For more details see the GitLab documentation for building Docker images.

1. We only run this on main and only when the .meta/Dockerfile changes. This makes

sure we are specific about when we change the Docker image. We could also use the

commit hash or other methods here to make the image more fungible.

Now, in further jobs down the pipeline, I can use the latest build of the Docker image like this:

firestore:
  image: registry.gitlab.com/group/project/buildimage
  stage: deploy 🚢🇮🇹
  script:
    - firebase deploy --only firestore
  only:
    changes:
      - .firebase-config/firestore.rules
      - .firebase-config/firestore.indexes.json

In this job, we only run the job if something about

the Firestore (the database from Firebase)

configuration changes. And when it does, we run the firestore deploy command in CI. I

also added a token for deploy as a GitLab CI/CD variable

based off the Firebase documentation

for using firebase with CI.

Summary

In the end, this helps speed up pipelines by ensuring that you have a custom-built build

image that you control. You don't have to rely on unstable or unmaintained Docker Hub

images or even have a Docker Hub account yourself to get started.

To learn more about GitLab CI/CD you can read the GitLab website

or the CI/CD docs. Also, there's a lot more to

learn about the GitLab Docker registry.

Cover image by Hack Capital on Unsplash.