How we used GitLab CI to build GitLab faster

May 2, 2018 · 4 min read
Rémy Coutable GitLab profile

GitLab is an open source project, but also a commercial project. For historic reasons, we have two Git repositories: gitlab-ce for GitLab Core and gitlab-ee for GitLab Enterprise packages (you can read our recent blog post explaining GitLab self-managed tiers). While we're working on having a single codebase, we still need to regularly merge gitlab-ce into gitlab-ee since most of the development happens on GitLab Core, but we also develop features on top of it for GitLab Starter, Premium, and Ultimate.

How we used to merge GitLab CE into GitLab EE

Until December 2017, the merge of gitlab-ce into gitlab-ee was manual on a daily basis with basically the following commands (see the full documentation):

# the `origin` remote refers to
# the `ce` remote refers to
git fetch origin master
git checkout -b ce-to-ee origin/master
git fetch ce master
git merge --no-ff ce/master

At this point, since we'd merge a day's worth of GitLab Core's new commits, chances were good we'd see conflicts. Most of the time, the person responsible for this process would handle the conflict resolutions, commit them and push the ce-to-ee branch to

There were a few problems with this approach:

The solution

Our plan was to have a single script that would automate the merge, and in the case of conflicts, identify the person best suited to resolve each of them. It would then create the merge request using the GitLab API and a GitLab API Ruby wrapper, and post a message in Slack when a new merge request was created or an existing one was still pending.

Finally, we'd use GitLab's pipeline schedules to run the script every three hours.

Step 1: Write the script

We chose to write the script in our release-tools project, since it already had a strong foundation for working with the relevant Git repositories.

This script was written iteratively as a set of classes over the course of a few months:

  1. Add the ability to find/create a merge request
  2. Move remotes to the Project classes and get rid of the Remotes class
  3. Add head, status, log, fetch, checkout_new_branch, pull, push, and merge to RemoteRepository
  4. Introduce a new CommitAuthor class

The last piece of the puzzle was the new upstream_merge Rake task.

Step 2: Create a pair of SSH keys and add the public key to the gitlab-ee project

Under Repository Settings > Deploy Keys of the gitlab-ee project:

Deploy key in `gitlab-ee`

Step 3: Create secret variables in the release-tools project

Under CI / CD Settings of the release-tools project, create three secret variables:

Secret variable

Step 4: Add a new CI job that runs the upstream_merge Rake task for pipeline schedules only

This was heavily inspired by GitBot – automating boring Git operations with CI.

Create a new upstream-merge CI job that:

`upstream-merge` job

You can check out the task for yourself.

Step 5: Create a pipeline schedule that runs every three hours

Under Schedules of the release-tools project:

Pipeline schedule

Step 6: Let the bot work for us!

The CI job:

CI job

The Slack messages:

Slack messages

The merge request:

Merge request

What are the benefits?

Since we started automating this process in December 2017, our dear @gitlab-bot created no fewer than 229 automatic merges, and we started noticing the benefits immediately:

The last, perhaps least visible, but most important benefit, is that we reduced developer frustration and increased happiness by removing a tedious chore.

Photo by Max Ostrozhinskiy on Unsplash

Edit this page View source