Breaking down CI/CD complexity with parent-child and multi-project pipelines

Feb 22, 2022 · 9 min read · Leave a comment
Fabio Pitino GitLab profile

Software requirements change over time. Customers request more features and the application needs to scale well to meet user demands. As software grows in size, so does its complexity, to the point where we might decide that it's time to split the project up into smaller, cohesive components.

As we proceed to tackle this complexity we want to ensure that our CI/CD pipelines continue to validate that all the pieces work correctly together.

There are two typical paths to splitting up software projects:

When we pick a path for splitting up the project, we should also adapt the CI/CD pipeline to match.

For the first path, GitLab CI/CD provides parent-child pipelines as a feature that helps manage complexity while keeping it all in a monorepo.

For the second path, multi-project pipelines are the glue that helps ensure multiple separate repositories work together.

Let's look into how these two approaches differ, and understand how to best leverage them.

Parent-child pipelines

It can be challenging to maintain complex CI/CD pipeline configurations, especially when you need to coordinate many jobs that may relate to different components, while at the same time keeping the pipeline efficient.

Let's imagine we have an app with all code in the same repository, but split into UI and backend components. A "one-size-fits-all" pipeline for this app probably would have all the jobs grouped into common stages that cover all the components. The default is to use build, test, and deploy stages. Unfortunately, this could be a source of inefficiency because the UI and backend represent two separate tracks of the pipeline. They each have their own independent requirements and structure and likely don't depend on each other. The UI might not need the build stage at all, but it might instead need a system-test stage with jobs that test the app end-to-end. Similarly, the UI jobs from system-test might not need to wait for backend jobs to complete.

Parent-child pipelines help here, enabling you to extract cohesive parts of the pipeline into child pipelines that runs in isolation.

With parent-child pipelines we could break the configurations down into two separate tracks by having two separate jobs trigger child pipelines:

ui:
  trigger:
    include: ui/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes: [ui/*]
backend:
  trigger:
    include: backend/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes: [backend/*]

The modifier strategy: depend, which is also available for multi-project pipelines, makes the trigger job reflect the status of the downstream (child) pipeline and waits for it to complete. Without strategy: depend the trigger job succeeds immediately after creating the downstream pipeline.

Now the frontend and backend teams can manage their CI/CD configurations without impacting each other's pipelines. In addition to that, we can now explicitly visualize the two workflows.

example parent-child pipeline

The two pipelines run in isolation, so we can set variables or configuration in one without affecting the other. For example, we could use rules:changes or workflow:rules inside backend/.gitlab-ci.yml, but use something completely different in ui/.gitlab-ci.yml.

Child pipelines run in the same context of the parent pipeline, which is the combination of project, Git ref and commit SHA. Additionally, the child pipeline inherits some information from the parent pipeline, including Git push data like before_sha, target_sha, the related merge request, etc. Having the same context ensures that the child pipeline can safely run as a sub-pipeline of the parent, but be in complete isolation.

A programming analogy to parent-child pipelines would be to break down long procedural code into smaller, single-purpose functions.

Multi-project pipelines

If our app spans across different repositories, we should instead leverage multi-project pipelines. Each repository defines a pipeline that suits the project's needs. Then, these standalone and independent pipelines can be chained together to create essentially a much bigger pipeline that ensures all the projects are integrated correctly.

There can be endless possibilities and topologies, but let's explore a simple case of asking another project to run a service for our pipeline.

The app is divided into multiple repositories, each hosting an independent component of the app. When one of the components changes, that project's pipeline runs. If the earlier jobs in the pipeline are successful, a final job triggers a pipeline on a different project, which is the project responsible for building, running smoke tests, and deploying the whole app. If the component pipeline fails because of a bug, the process is interrupted and there is no need to trigger a pipeline for the main app project.

The component project's pipeline:

build:
  stage: build
  script: ./build_component.sh

test:
  stage: test
  script: ./test_component.sh

deploy:
  stage: deploy
  trigger:
    project: myorg/app
    strategy: depend

The full app project's pipeline in myorg/app project:

build:
  stage: build
  script: ./build_app.sh  # build all components

qa-test:
  stage: test
  script: ./qa_test.sh

smoke-test:
  stage: test
  script: ./smoke_test.sh

deploy:
  stage: deploy
  script: ./deploy_app.sh

example multi-project pipeline

In our example, the component pipeline (upstream) triggers a downstream multi-project pipeline to perform a service: verify the components work together, then deploy the whole app.

A programming analogy to multi-project pipelines would be like calling an external component or function to either receive a service (using strategy:depend) or to notify it that an event occurred (without strategy:depend).

Key differences between parent-child and multi-project pipelines

As seen above, the most obvious difference between parent-child and multi-project pipelines is the project where the pipelines run, but there are are other differences to be aware of.

Context:

Control:

Side-effects:

Visibility:

Conclusions

Parent-child pipelines inherit a lot of the design from multi-project pipelines, but parent-child pipelines have differences that make them a very unique type of pipeline relationship.

Some of the parent-child pipelines work we at GitLab will be focusing on is about surfacing job reports generated in child pipelines as merge request widgets, cascading cancelation and removal of pipelines as well as passing variables across related pipelines. Some of the parent-child pipeline work we at GitLab plan to focus on relates to:

You can check this issue for planned future developments on parent-child and multi-project pipelines. Leave feedback or let us know how we can help.

Cover image by Ravi Roshan on Unsplash

“Breaking down CI/CD complexity with parent-child and multi-project pipelines” – Fabio Pitino

Click to tweet

Open in Web IDE View source