How we solved GitLab's CHANGELOG conflict crisis

Since its very first commit more than six years ago, GitLab has had a changelog

detailing the noteworthy changes in each release. Shortly after [Enterprise

Edition (EE) was introduced], it got a changelog of its own. Whenever anyone

– whether it was a community contributor or a GitLab employee – contributed a

new feature or fix to the project, a changelog entry would be added to let users

know what improved.

As GitLab gained in popularity and started receiving more contributions, we'd

constantly see merge conflicts in the changelog when multiple merge requests

attempted to add an entry to the list. This quickly became a major source of

delays in development, as contributors would have to rebase their branch in order

to resolve the conflicts.

This post outlines how we completely eliminated changelog-related merge

conflicts, removed bottlenecks for contributions, and automated a crucial part

of our release process.

At the beginning, GitLab's CHANGELOG file would look something like this:

v 8.0.0 (unreleased)
  - Prevent anchors from being hidden by header (Stan Hu)
  - Remove satellites
  - Better performance for web editor (switched from satellites to rugged)
  - Faster merge
  - ...
  - Ability to fetch merge requests from refs/merge-requests/:id

v 7.14.1
  - Improve abuse reports management from admin area
  - Ability to enable SSL verification for Webhooks

v 7.14.0
  - Fix bug where non-project members of the target project could set labels on new merge requests.
  - Upgrade gitlab_git to 7.2.14 to ignore CRLFs in .gitmodules (Stan Hu)
  - ...
  - Fix broken code import and display error messages if something went wrong with creating project (Stan Hu)

When a developer made a change in the upcoming release, 8.0.0 in this example,

they would add a changelog entry at the bottom:

diff --git a/CHANGELOG b/CHANGELOG

index de2066f..0fc2c18 100644

--- a/CHANGELOG

+++ b/CHANGELOG

@@ -5,6 +5,7 @@ v 8.0.0 (unreleased)
   - Faster merge
   - ...
   - Ability to fetch merge requests from refs/merge-requests/:id
+  - Made literally everything better. Evvvvverything!

 v 7.14.1
   - Improve abuse reports management from admin area

At the same time, another developer might have made a similar change in their

branch:

diff --git a/CHANGELOG b/CHANGELOG

index de2066f..5f81cfd 100644

--- a/CHANGELOG

+++ b/CHANGELOG

@@ -5,6 +5,7 @@ v 8.0.0 (unreleased)
   - Faster merge
   - ...
   - Ability to fetch merge requests from refs/merge-requests/:id
+  - Made a few things worse. Woops!

 v 7.14.1
   - Improve abuse reports management from admin area

Now when one branch was merged, it'd create a conflict in the other:

diff --cc CHANGELOG

index 5f81cfd,0fc2c18..0000000

--- a/CHANGELOG

+++ b/CHANGELOG

@@@ -5,7 -5,7 +5,11 @@@ v 8.0.0 (unreleased
    - Faster merge
    - ...
    - Ability to fetch merge requests from refs/merge-requests/:id
++<<<<<<< HEAD
 +  - Made a few things worse. Woops!
++=======

+   - Made literally everything better. Evvvvverything!

++>>>>>>> developer-1

  v 7.14.1
    - Improve abuse reports management from admin area

This resulted in a ton of wasted time as something would get merged, and then

every other open branch adding a changelog entry would need to be rebased. The

situation only got worse as the number of contributors to GitLab grew over time.

Our initial, boring solution to the problem was to begin adding empty

placeholder entries at the beginning of each monthly release cycle. The

changelog for the upcoming unreleased version might look like this:

v8.1.0 (unreleased)
  -
  -
  -
  -
  -
  -
  -
  - (and so on)

A developer would make their change and then choose a random spot in the list to

add a changelog entry. This worked for a while, until the placeholders began to

be filled out as we got closer to the release date. Eventually two (or more)

merge requests would attempt to add different entries at the same placeholder,

and one being merged created a conflict in the others.

The problem was lessened, but not solved.

Not only was this a huge waste of time for developers, it created an additional

headache for release managers when they cherry-picked a commit into a stable

branch for a patch release. If the commit included a changelog entry, which any

change intended for a patch release should have, cherry-picking that commit

would bring in the contents of the changelog at the point of that commit, often

including dozens of unrelated changes. The release manager would have to

manually remove the unrelated entries, often doing this multiple times per

release. This was compounded when we had to release multiple patch versions at

once due to a security issue.

Brainstorming solutions

Frustrations with the process finally reached a tipping point, and [an issue was

created] to discuss a solution. Yorick had the original idea that would

ultimately form the foundation of our solution. During a [trip around the

world], myself, Douwe, and Marin were in Brooklyn, NY, and during a walk

around the city one beautiful summer evening we ended up with a proposal to

finally solve the problem.

Each changelog entry would be its own YAML file in a CHANGELOG/unreleased

folder. When a release manager went to cherry-pick a merge into a stable branch

in preparation for a release, they'd use a custom script that would perform the

cherry-pick and then move any changelog entry added by that action to a

version-specific subfolder, such as CHANGELOG/8.9.4. At the time of release,

any entries in the version's subfolder would be compiled into a single Markdown

changelog file, and then deleted.

With an idea of where we wanted to end up but no idea how to get there, I

started with a spike.

A turning point

After a few days of working on the spike, I had a realization that we didn't

need the cherry-picking concept at all:

Cherry picking a merge commit into a stable branch will add that merge's

CHANGELOG/unreleased/whatever-its-called.yml file to the stable branch. Upon

tagging a release with release-tools, we can consider everything in that

stable branch's "unreleased" folder as part of the tagged release. We collect

those files, compile them to Markdown, remove them from the stable branch

and master, and that's our changelog for the release.

This was a major "aha" moment, as it greatly simplified the

workflow for release managers. They could continue their existing workflow, and

the release flow would transparently handle the rest. It also meant we could

handle everything in our release-tools project, which is responsible

for tagging a release and kicking off our packaging.

Even though we ended up not using a lot of the work that went into it, my

original spike was still valuable. It allowed us to see pain points early on,

refine the process, and find a better solution. It also gave me additional

experience interacting with Git repositories programmatically via Rugged, and

that would go on to be especially useful as we implemented the final tooling.

Building the building blocks

We knew there were several components that we'd need to build:

1. Something to read and represent the individual YAML data files

2. Something to compile individual entries into a Markdown list

3. Something to insert the compiled Markdown into the correct spot in an existing list of releases

4. Something to remove the files that had been compiled, and then commit the updated CHANGELOG.md file to the repository

All of these components were created in a single merge request and refined

through several code review cycles. The commits listed there are all fairly

atomic and may be interesting to read through on their own. The code review that

happened in the merge request was incredibly valuable, and allowed us to really

simplify some code that was hard to wrap one's head around, even for me as the

original author!

Automated testing

Of course, we wouldn't consider this solution complete until we had automated

tests guaranteeing the behavior and consistency of the automated compilation,

including reading from and writing to multiple branches across multiple

repositories.

I ended up using Rugged to create fixture repositories that would create a

repeatable testing environment, which we could then verify with [custom RSpec

matchers].

Hooking into the release process

At this point we were fairly confident the changelog compilation worked, so it

was time to hook it into our existing release process.

While testing this integration on a real release, we uncovered a pretty

hilarious (but dangerous) oversight. I'll let the commit that fixed it speak for

itself:

Protect against deleting everything when there are no changelog entries

On a stable branch with no changelog entry files, the resulting empty

array was passed to Rugged::Index#remove_all which, when given an

empty array, removes everything. This was not ideal.

Developer tooling

The final pieces of the puzzle were creating a tool to help developers create

valid changelog entries easily, and adding documentation. Both were handled in

this merge request.

This tool allows developers to run bin/changelog, passing it the title of

their change, to generate a valid changelog entry file. Additional options are

in the documentation.

Future plans

This changelog process has worked beautifully for us since it was introduced,

and we know it might be just as useful to other projects. We're [investigating a

way to make it more generic] so that it can remove a tedious chore for more

developers.

I worked on this project as part of our Edge team, now known as the [Quality

team]. If you're interested in this kind of internal tooling or other

automation, we're hiring! Check out our open positions.

Photo by Patrick Tomasso on Unsplash