Building a GitLab CI/CD pipeline for a monorepo the easy way

Monorepos allow you to host multiple applications’ code in a single repository. In GitLab, that involves placing disparate application source code in separate directories in one project. While this strategy allows for version controlled storage of your code, it was tricky leveraging the full power of GitLab’s CI/CD pipeline capabilities… until now!

The ideal case: CI/CD in a monorepo

Since you have more than one application’s code living in your repository, you will want to have more than one pipeline configuration. For example, if you have a .NET application and a Spring application in one project, each application may have different build and test jobs to complete. Ideally, you can completely decouple the pipelines and only run each pipeline based on changes to that specific application’s source code.

The technical approach for this would be to have a project-level .gitlab-ci.yml pipeline configuration file that includes a specific YAML file based on changes in a certain directory. The .gitlab-ci.yml pipeline serves as the control plane that triggers the appropriate pipeline based on the changes made to the code.

The legacy approach

Prior to GitLab 16.4, we were not able to include a YAML file based on changes to a directory or file in a project. However, we could accomplish this functionality via a workaround.

In our monorepo project, we have two directories for different applications. In this example, there are java and python directories representing a Java and Python app, respectively. Each directory has an application-specific YAML file to build each app. In the project’s pipeline file, we simply include both application pipeline files, and do the logic handling in those files directly.

.gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

top-level-job:
  stage: build
  script:
    - echo "Hello world..."

include:
  - local: '/java/j.gitlab-ci.yml'
  - local: '/python/py.gitlab-ci.yml'

In each application-specific pipeline file, we create a hidden job named .java-common or .python-common that only runs if there are changes to that app’s directory. Hidden jobs do not run by default, and are often utilized to reuse specific job configurations. Each pipeline extends that hidden job to inherit the rules defining which files to watch for changes, which would then initiate the pipeline job.

j.gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

.java-common:
  rules:
    - changes:
      - '../java/*'

java-build-job:
  extends: .java-common
  stage: build
  script:
    - echo "Building Java"

java-test-job:
  extends: .java-common
  stage: test
  script:
    - echo "Testing Java"

py.gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

.python-common:
  rules:
    - changes:
      - '../python/*'

python-build-job:
  extends: .python-common
  stage: build
  script:
    - echo "Building Python"

python-test-job:
  extends: .python-common
  stage: test
  script:
    - echo "Testing Python"

There are some downsides to this, including having to extend the job for each other job in the YAML file to ensure it complies with the rules, creating a lot of redundant code and room for human error. Additionally, extended jobs cannot have duplicate keys, so you could not define your own rules logic in each job since there would be a collision in the keys and their values are not merged.

This results in a pipeline running that includes the j.gitlab-ci.yml jobs when java/ is updated, and py.gitlab-ci.yml when python/ is updated.

The new approach: Conditionally include pipeline files

In GitLab 16.4, we introduced include with rules:changes for pipelines. Previously, you could include with rules:if, but not rules:changes making this update extremely powerful. Now, you can simply use the include keyword and define the monorepo rules in your project pipeline configuration.

New .gitlab-ci.yml:

stages:
  - build
  - test

top-level-job:
  stage: build
  script:
    - echo "Hello world..."

include:
  - local: '/java/j.gitlab-ci.yml'
    rules:
      - changes:
        - 'java/*'
  - local: '/python/py.gitlab-ci.yml'
    rules:
      - changes:
        - 'python/*'

Then each application’s YAML can just focus on building and testing that application’s code, without extending a hidden job repeatedly. This allows for more flexibility in job definitions and reduces code rewriting for engineers.

New j.gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

java-build-job:
  stage: build
  script:
    - echo "Building Java"

java-test-job:
  stage: test
  script:
    - echo "Testing Java"

New py.gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

python-build-job:
  stage: build
  script:
    - echo "Building Python"

python-test-job:
  stage: test
  script:
    - echo "Testing Python"

This accomplishes the same task of including the Java and Python jobs only when their directories are modified. Something to consider in your implementation is that jobs can run unexpectedly when using changes. The changes rule always evaluates to true when pushing a new branch or a new tag to GitLab, so all jobs included will run upon first push to a branch regardless of the rules:changes definition. You can mitigate this experience by creating your feature branch first and then opening a merge request to begin your development, since the first push to the branch when it is created will force all jobs to run.

Ultimately, monorepos are a strategy that can be used with GitLab and CI/CD, and, with our new include with rules:changes feature, we have a better best practice for using GitLab CI with monorepos. To get started with monorepos, take out a free Gitlab Ultimate trial today.

More CI/CD resources

• 5 tips for managing monorepos in GitLab

• How to learn CI/CD fast