Git 2.42 release: Here are four of our contributions in detail

Git 2.42

was officially released on August 21, 2023, and included some

improvements from GitLab's Git team. Git is the foundation of

repository data at GitLab. GitLab's Git team works on new features, performance improvements, documentation improvements,

and growing the Git community. Often our contributions to Git have a

lot to do with the way we integrate Git into our services at

GitLab.

We previously shared some of our improvements that were included in the Git 2.41 release. Here are some highlights from the Git 2.42 release, and a

window into how we use Git on the server side at GitLab.

1. Prevent certain refs from being packed

Write-ahead logging

In Gitaly, we

want to use a write-ahead log

to replicate Git operations on different machines.

This means that the Git objects and references that should be changed

by a Git operation are first kept in a log entry. Then, when all the

machines have agreed that the operation should proceed, the log entry

is applied so the corresponding Git objects and references are

actually added to the repositories on all the machines.

Need for temporary references

Between the time when a specific log entry is first written and when

it is applied, other log entries could be applied which could remove

some objects and references. It could happen that these objects and

references are needed to apply the specific log entry though.

So when we log an entry, we have to make sure that all the objects and

references that it needs to be properly applied will not be removed

until that entry is either actually applied or discarded.

The best way to make sure things are kept in Git is to create new Git

references pointing to these things. So we decided to use temporary

references for that purpose. They would be created when a log entry is

written, and then deleted when that entry is either applied or

discarded.

Packed-refs performance

Git can store references in "loose" files, with one reference per

file, or in the packed-refs file, which contains many of them. The

git pack-refs command is used to pack some references from "loose"

files into the packed-refs file.

For reading a lot of references, the packed-refs file is very

efficient, but for writing or deleting a single reference, it is not

so efficient as rewriting the whole packed-refs file is required.

As temporary references are to be created and then deleted soon after,

storing them in the packed-refs file would not be efficient. It

would be better to store them in "loose" files.

The git pack-refs command had no way to be told precisely which refs

should be packed or not though. By default it would repack all the

tags (which are refs in refs/tags/) and all the refs that are

already packed. With the --all option one could tell it to repack

all the refs except the hidden refs, broken refs, and symbolic refs,

but that was the only thing that could be controlled.

Improving git pack-refs

We decided to improve git pack-refs by adding two new options to it:

• --include <pattern> which can be used to specify which refs should be packed
• --exclude <pattern> which can be used to specify which refs should not be packed

John Cai, Gitaly:Git team engineering manager, implemented these options.

For example, if the refs managed by the write-ahead log are in

refs/wal/, it's now possible the exclude them from being moved into

the packed-refs file by using:

$ git pack-refs --exclude "refs/wal/*"

Details of the patch series, including discussions, can be found

here.

2. Get machine-readable output from git cat-file --batch

Efficiently retrieving Git object information

In GitLab, we often retrieve Git object information. For example, when a

user navigates into the files and directories in a repository, we need

to get the content of the corresponding Git blobs and trees so that

we can show it.

In Gitaly, we use git cat-file to retrieve Git object information

from a Git repository. As it's a frequent operation, it needs to be

performed efficiently, so we use the batch modes of git cat-file

available through the --batch, --batch-check and --batch-command

options.

In these modes, a pointer to a Git object can be repeatedly sent to

the standard input, called 'stdin', of a git cat-file command, while

the corresponding object information is read from the standard ouput,

called 'stdout' of the command. This way we don't need to launch a

new git cat-file command for each object.

GitLab can keep, for example, a git cat-file --batch-command process

running in the background while feeding it commands like

info <object> or contents <object> through its stdin to

get either information about an object or its content.

Newlines in stdin, stdout, and filenames

The commands or pointers to Git objects that are sent through stdin

should be delimited using newline characters, and in the same way `git

cat-file` will use newline characters to delimit the information from

different Git objects in its output. This is a common shell practice

to make it easy to chain commands together. For example, one can

easily get the size (in bytes) of the last three commits on the current

branch using the following:

$ git log -3 --format='%H' | git cat-file --batch-check='%(objectsize)'

285

646

428

Sometimes, though, the pointer to a Git object can contain a filename

or a directory name, as such a pointer is allowed to be in the form

<branch>:<path>. For example HEAD:Documentation is a valid

pointer to the blob or the tree corresponding to the Documentation

path on the current branch.

This used to be an issue because on some systems newline characters

are allowed in file or directory names. So the -z option was

introduced last year in Git 2.38 to allow users to change the input

delimiter in batch modes to the NUL character.

Error output

When the -z option was introduced, it wasn't considered useful to

change the output delimiter to be also the NUL character. This is

because only tree objects can contain paths and the internal format

of tree objects already uses NUL characters to delimit paths.

Unfortunately, it was overlooked that in case of an error the pointer

to the object is displayed in the error message:

$ echo 'HEAD:does-not-exist' | git cat-file --batch

HEAD:does-not-exist missing

As the error messages are printed along with the regular ouput of the

command on stdout, passing in an invalid pointer with a number of

newline characters in it could make it very difficult to parse the

output.

-Z comes to the rescue

Toon Claes, Gitaly senior engineer, initially worked on a

patch to just quote the pointer in the error message, but it was

decided in the Git mailing list discussions related to the patch that

it would be better to just create a new -Z option. This option would

change both the input and the output delimiter to the NUL character,

while the old -z option would be deprecated over time.

So Patrick Steinhardt, Gitaly staff engineer, implemented that new -Z option.

Details of the patch series, including discussions, can be found

here

and here.

3. Pass pseudo-options to git rev-list --stdin

Computing sizes

In GitLab, we need to have different ways to compute the size of Git

related content. For example, we need to know:

• how much disk space a repository is using
• how big a specific Git object is
• how much additional space on a repository is required by a specific set of revisions (and the objects they reference)

Knowing "how much disk space a repository is using" is useful to

enforce repository-related quotas and is easy to get using regular

shell and OS features.

Size information about a specific Git object is useful to enforce

quotas related to maximum file size. It can be obtained using, for

example, git cat-file -s <object> or

echo <object> | git cat-file --batch-check='%(objectsize)'

as already seen above.

Computing the space required by a set of revisions is useful, too, as

forks can share Git content in what we call

"pool repositories,"

and we want to discriminate how much content belongs to each forked

repository. Fortunately, git rev-list has a --disk-usage option

for this purpose.

Passing arguments to git rev-list

git rev-list can take a number of different arguments and has a lot

of different options. It's a fundamental command to traverse commit

graphs, and it should be flexible enough to fulfill a lot of different

user needs.

When repositories grow, they often store a lot of references and a lot

of files and directories, so there is often the need to pass a big

number of references or paths as arguments to the

command. References and paths can be quite long though.

To avoid hitting platform limits related to command line length, long

ago, a --stdin mode was added that allowed users to pass revisions

and paths through stdin, instead of as command line

arguments. However, when that was implemented, it was not considered

necessary to allow options or pseudo-options, like --not,

--glob=..., or --all to be passed through stdin.

This appeared to be a problem for GitLab, as for computing sizes for

forked repositories we needed some of the pseudo-options, and it would

have been intricate and possibly buggy to pass some of them and their

arguments as arguments on the command line while others were passed

through stdin.

Allowing pseudo-options

To fix this issue, Patrick Steinhardt implemented a small patch series to

allow pseudo-options through stdin.

With it, in Git 2.42, one can now pass pseudo-options, like --not,

--glob=..., or --all through stdin when the --stdin mode is used.

Details of the patch series, including discussions, can be found

here.

4. Code and test improvements

While looking at some Git code, we are often tempted to modify nearby

code, either to change only its style when the code is ancient and it

would look better using Git's current code style, or to refactor it to

make it cleaner. This is why we sometimes send small patch series that

don't have a real GitLab related purpose.

In Git 2.42, examples of style code improvements we made are the

part1

and

part2

test code modernization patches from John Cai.

And here is

an example of a refactoring to cleanup some code by Patrick Steinhardt.