Future-proofing Git repository maintenance

Users get the most from Gitaly, the service responsible for the storage and maintenance of all Git repositories in GitLab, when traffic hitting it is efficiently handled. Therefore, we must ensure our Git repositories remain in a well-optimized state. When it comes to Git monorepositories, this maintenance can be a complex task that can cause a lot of overhead by itself because repository housekeeping becomes more expensive the larger the repositories get. This blog post explains in depth what we have done over the past few GitLab releases to rework our approach to repository housekeeping for better scaling and to maintain an optimized state to deliver the best peformance for our users.

The challenge with Git monorepository maintenance

To ensure that Git repositories remain performant, Git regularly runs a set of maintenance tasks. On the client side, this usually happens by automatically running git-gc(1) periodically, which:

• Compresses revisions into a packed-refs file.
• Compresses objects into packfiles.
• Prunes objects that aren't reachable by any of the revisions and that have not been used for a while.
• Generates and updates data structures like commit-graphs that help to speed up queries against the Git repository.

Git periodically runs git gc --auto automatically in the background, which analyzes your repository and only performs maintenance tasks if required.

At GitLab, we can't use this infrastructure because it does not give us enough control over which maintenance tasks are executed at what point in time. Furthermore, it does not give us full control over exactly which data structures we opt in to. Instead, we have implemented our own maintenance strategies that are specific to how GitLab works and catered to our specific needs. Unfortunately, the way GitLab implemented repository maintenance has been limiting us for quite a while by now.

• It is unsuitable for large monorepositories.
• It does not give us the ability to easily iterate on the employed maintenance strategy.

This post explains our previous maintenance strategy and its problems as well as how we revamped the architecture to allow us to iterate faster and more efficiently maintain repositories.

Our previous repository maintenance strategy

In the early days of GitLab, most of the application ran on a single server. On this single server, GitLab directly accessed Git repositories. For various reasons, this architecture limited us, so we created the standalone Gitaly server that provides a gRPC API to access Git repositories.

To migrate to exclusively accessing Git repository data using Gitaly we:

• Migrated all the logic that was previously contained in the Rails application to Gitaly.
• Created Gitaly RPCs and updated Rails to not execute the logic directly, but instead invoke the newly-implemented RPC.

While this was the easiest way to tackle the huge task back then, the end result was that there were still quite a few areas in the Rails codebase that relied on knowing how the Git repositories were stored on disk.

One such area was repository maintenance. In an ideal world, the Rails server would not need to know about the on-disk state of a Git repository. Instead, the Rails server would only care about the data it wants to get out of the repository or commit to it. Because of the Gitaly migration path we took, the Rails application was still responsible for executing fine-grained repository maintenance by calling certain RPCs:

• Cleanup to delete stale, temporary files that have accumulated
• RepackIncremental and RepackFull to either pack all loose objects into a new packfile or alternatively to repack all packfiles into a single one
• PackRefs to compress all references into a single packed-refs file
• WriteCommitGraph to update the commit-graph
• GarbageCollect to perform various different tasks

These low-level details of repository maintenance were being managed by the client. But because clients didn't have any information on the on-disk state of the repository, they could not even determine which of these maintenance tasks had to be executed in the first place. Instead, we had a very simple heuristic: Every few pushes, we ran one of the above RPCs to perform one of the maintenance tasks. While this heuristic worked, it wasn't great for the following reasons:

• Repositories can be modified without using pushes at all. So if users only use the Web IDE to commit to repositories, they may not get repacked at all.
• Because repository maintenance is controlled by the client, Gitaly can't assume a specific repository state.
• The threshold for executing housekeeping tasks is set globally across all projects rather than on a per-project basis. Consequently, no matter whether you have a tiny repository or a huge monorepository, we would use the same intervals for executing maintenance tasks. As you may imagine though, doing a full repack of a Git repository that is only a few dozen megabytes in size is a few orders of magnitudes faster than repacking a monorepository that is multiple gigabytes in size.
• Specific types of Git repositories hosted by Gitaly need special care and we required Gitaly clients to know about these.
• Repository maintenance was inefficient overall. Clients do not know about the on-disk state of repositories. Consequently, they had no choice except to repeatedly ask Gitaly to optimize specific data structures without knowing whether this was required in the first place.

Heuristical maintenance strategy

It was clear that we needed to change the strategy we used for repository maintenance. Most importantly, we wanted to:

• Make Gitaly the single source of truth for how we maintain repositories. Clients should not need to worry about low-level specifics, and Gitaly should be able to easily iterate on the strategy.
• Make the default maintenance strategy work for repositories of all sizes.
• Make the maintenance strategy work for repositories of all types. A client should not need to worry about which maintenance tasks must be executed for what repository type.
• Avoid optimizing data structures that already are in an optimal state.
• Improve visibility into the optimizations we perform.

As mentioned in the introduction, Git periodically runs git gc --auto. This command inspects the repository's state and performs optimizations only when it finds that the repository is in a sufficiently bad state to warrant the cost. While using this command directly in the context of Gitaly does not give us enough flexibility, it did serve as the inspiration for our new architecture.

Instead of providing fine-grained RPCs to maintain various parts of a Git repository, we now only provide a single RPC OptimizeRepository that works as a black-box to the caller. This RPC call:

1. Cleans up stale data in the repository if there is any.
2. Analyzes the on-disk state of the repository.
3. Depending on this on-disk state, performs only these maintenance tasks that are deemed to be necessary.

Because we can analyze and use the on-disk state of the repository, we can be far more intelligent about repository maintenance compared to the previous strategy where we optimized some bits of the repository every few pushes.

Packing objects

In the old-style repository maintenance, the client would call either RepackIncremental or RepackFull. This would either: Pack all loose objects into a new packfile or repack all objects into a single packfile.

By default, we would perform a full repack every five repacks. While this may be a good default for small repositories, it gets prohibitively expensive for huge monorepositories where a full repack may easily take several minutes.

The new heuristical maintenance strategy instead scales the allowed number of packfiles by the total size of all combined packfiles. As a result, the larger the repository becomes, the less frequently we perform a full repack.

Pruning objects

In the past, clients would periodically call GarbageCollect. In addition to repacking objects, this RPC would also prune any objects that are unreachable and that haven't been accessed for a specific grace period.

The new heuristical maintenance strategy scans through all loose objects that exist in the repository. If the number of loose objects that have a modification time older than two weeks exceeds a certain threshold, it spawns the git prune command to prune these objects.

Packing references

In the past, clients would call PackRefs to repack references into the packed-refs file.

Because the time to compress references scales with the size of the packed-refs file, the new heuristical maintenance strategy takes into account both the size of the packed-refs file and the number of loose references that exist in the repository. If a ratio between these two figures is exceeded, we compress the loose references.

Auxiliary data structures

There are auxiliary data structures like commit-graphs that are used by Git to speed up various queries. With the new heuristical maintenance strategy, Gitaly now automatically updates these as required, either when they are deemed to be out-of-date, or when they are missing altogether.

Heuristical maintenance strategy rollout

We rolled out this new heuristical maintenance strategy to GitLab.com in March 2022. Initially, we only rolled it out for gitlab-org/gitlab, which is a repository where maintenance performed particularly poorly in the past. You can see the impact of the rollout in the following graph:

In this graph, you can see that:

1. Until March 19, we used the legacy fine-grained RPC calls. We spent most of the time in RepackFull, followed by RepackIncremental and GarbageCollect.
2. Because March 19 and 20 occurred on a weekend, nothing much happens with housekeeping.

3. Early on March 21 we switched gitlab-org/gitlab to use heuristical housekeeping using OptimizeRepository. Initially, there didn't seem to be much of an improvement. There wasn't much difference in how much time we spent maintaining this repository compared to the past.

   However, this was caused by an inefficient heuristic. Instead of only pruning objects when there were stale ones, we always pruned objects when we saw that there were too many loose objects.

4. We deployed a fix for this bug on March 22, which led to a significant drop in time spent optimizing this repository compared to before.

This demonstrated two things:

• We're easily able to iterate on the heuristics that we have in Gitaly.
• Using the heuristics saves a lot of compute time as we don't unnecessarily optimize anymore.

We have subsequently rolled this out to all of GitLab.com, starting on March 29, 2022, with similar improvements. With this change, we more than halved the CPU load when performing repository optimizations.

Observability

While it is great that OptimizeRepository has managed to save us a lot of compute power, one goal was to improve visibility into repository housekeeping. More specifically, we wanted to:

• Gain visibility on the global level to see what optimizations are performed across all of our repositories.
• Gain visibility on the repository level to know what state a specific repository is in.

In order to improve global visibility, we expose a set of Prometheus metrics that allow us to observe important details about our repository maintenance. The following graphs show the optimizations performed in a 30-minute window of our production systems on GitLab.com.

• The optimizations, which are being performed in general.

  

• The average latency it takes to perform each of these optimizations.

  

• What kind of stale data we are cleaning up.

  

To improve visibility into the state each repository is in we have started to log structured data that includes all the relevant bits. A subset of the information it exposes is:

• The number of loose objects and their sizes.
• The number of packfiles and their combined size.
• The number of loose references.
• The size of the packed-refs file.
• Information about commit-graphs, bitmaps and other auxiliary data structures.

This information is also exposed through Prometheus metrics:

These graphs expose important metrics of the on-disk state of our repositories:

• The top panel shows which data structures exist.
• The heatmaps on the left show how large specific data structures are.
• The heatmaps on the right show how many of these data structures we have.

Combining both the global and per-repository information allows us to easily observe how repository maintenance behaves during normal operations. But more importantly, it gives us meaningful data when rolling out new features that change the way repositories are maintained.

Manually enabling heuristical housekeeping

While the heuristical housekeeping is enabled by default starting with GitLab 15.9, it has already been introduced with GitLab 14.10. If you want to use the new housekeeping strategy before upgrading to 15.9, you can opt in by setting the optimized_housekeeping feature flag. You can do so via the gitlab-rails console:

Feature.enable(:optimized_housekeeping)

Future improvements

While the new heuristical optimization strategy has been successfully battle-tested for a while now for GitLab.com, at the time of writing this blog post, it still wasn't enabled by default for self-deployed installations. This has finally changed with GitLab 15.8, where we have default-enabled the new heuristical maintenance strategy.

We are not done yet, though. Now that Gitaly is the only source of truth for how repositories are optimized, we are tracking improvements to our maintenance strategy in epic 7443:

• Multi-pack indices and geometric repacking will help us to further reduce the time spent repacking objects.
• Cruft packs will help us to further reduce the time spent pruning objects and reduce the overall size of unreachable objects.
• Gitaly will automatically run housekeeping tasks when receiving mutating RPC calls so that clients don't have to call OptimizeRepository at all anymore.

So stay tuned!