What’s new in Git 2.51.0?

The Git project recently released Git 2.51. Due to summer in the Northern Hemisphere and slower progress, this release cycle was on the shorter side of 8 weeks (typically a release cycle lasts about 12 weeks). Let’s look at some notable changes in this release, including contributions from the Git team at GitLab and also the wider Git community.

Performance optimizations for git-push(1) and git-fetch(1)

The git-push(1) and git-fetch(1) commands allow users to synchronize local and remote repositories. Part of the operation involves updating references in the repository. In repositories with many references, this can take significant time, especially for users who work with large development environments, monorepos, or repositories with extensive CI/CD pipelines.

Git reference transactions can include multiple reference updates, but they follow an all-or-nothing approach. If any single update within the transaction fails, the entire transaction fails and none of the reference updates are applied. But reference updates as part of git-push(1) and git-fetch(1) are allowed to fail, which allows repositories to synchronize a subset of references even in the case where a different subset has diverged. To facilitate this behavior, Git creates a separate transaction for each reference update, allowing some transactions to fail while the rest succeed.

Creating a separate transaction per update incurs significant overhead, as each transaction includes an initiation and teardown phase and also checks for whether there are conflicting reference names. The “reftable” backend also performs auto-compaction at the end of a transaction, so multiple transactions would trigger multiple auto-compactions, which would drastically increase the latency of the command.

In Git 2.51.0, these commands now use batched updates instead of separate transactions. Batched updates allow updating multiple references under a single transaction, while still allowing some updates to fail. This removes the overhead and scales better with the number of references to be updated, since only a single transaction is used. This significantly improves the performance of the “reftable” backend, which now outperforms the “files” backend. Users can reap these performance improvements without needing to make any changes.

For git-fetch(1) we see a 22x performance improvement for the “reftable” backend and 1.25x improvement for the “files” backend when used in a repository with 10,000 references.

Benchmark 1: fetch: many refs (refformat = reftable, refcount = 10000, revision = master)
  Time (mean ± σ):      3.403 s ±  0.775 s    [User: 1.875 s, System: 1.417 s]
  Range (min … max):    2.454 s …  4.529 s    10 runs

Benchmark 2: fetch: many refs (refformat = reftable, refcount = 10000, revision = HEAD)
  Time (mean ± σ):     154.3 ms ±  17.6 ms    [User: 102.5 ms, System: 56.1 ms]
  Range (min … max):   145.2 ms … 220.5 ms    18 runs

Summary
  fetch: many refs (refformat = reftable, refcount = 10000, revision = HEAD) ran
   22.06 ± 5.62 times faster than fetch: many refs (refformat = reftable, refcount = 10000, revision = master)

Benchmark 1: fetch: many refs (refformat = files, refcount = 10000, revision = master)
  Time (mean ± σ):     605.5 ms ±   9.4 ms    [User: 117.8 ms, System: 483.3 ms]
  Range (min … max):   595.6 ms … 621.5 ms    10 runs

Benchmark 2: fetch: many refs (refformat = files, refcount = 10000, revision = HEAD)
  Time (mean ± σ):     485.8 ms ±   4.3 ms    [User: 91.1 ms, System: 396.7 ms]
  Range (min … max):   477.6 ms … 494.3 ms    10 runs

Summary
  fetch: many refs (refformat = files, refcount = 10000, revision = HEAD) ran
    1.25 ± 0.02 times faster than fetch: many refs (refformat = files, refcount = 10000, revision = master)

For git-push(1) we see a 18x performance improvement for the reftable backend and 1.21x improvement for the “files” backend when used in a repository with 10,000 references.

Benchmark 1: push: many refs (refformat = reftable, refcount = 10000, revision = master)
  Time (mean ± σ):      4.276 s ±  0.078 s    [User: 0.796 s, System: 3.318 s]
  Range (min … max):    4.185 s …  4.430 s    10 runs

Benchmark 2: push: many refs (refformat = reftable, refcount = 10000, revision = HEAD)
  Time (mean ± σ):     235.4 ms ±   6.9 ms    [User: 75.4 ms, System: 157.3 ms]
  Range (min … max):   228.5 ms … 254.2 ms    11 runs

Summary
  push: many refs (refformat = reftable, refcount = 10000, revision = HEAD) ran
   18.16 ± 0.63 times faster than push: many refs (refformat = reftable, refcount = 10000, revision = master)

Benchmark 1: push: many refs (refformat = files, refcount = 10000, revision = master)
  Time (mean ± σ):      1.121 s ±  0.021 s    [User: 0.128 s, System: 0.975 s]
  Range (min … max):    1.097 s …  1.156 s    10 runs

Benchmark 2: push: many refs (refformat = files, refcount = 10000, revision = HEAD)
  Time (mean ± σ):     927.9 ms ±  22.6 ms    [User: 99.0 ms, System: 815.2 ms]
  Range (min … max):   903.1 ms … 978.0 ms    10 runs

Summary
  push: many refs (refformat = files, refcount = 10000, revision = HEAD) ran
    1.21 ± 0.04 times faster than push: many refs (refformat = files, refcount = 10000, revision = master)

This project was led by Karthik Nayak.

Planning towards Git 3.0

11 years ago, Git 2.0 was released, which was the last major version release of Git. While we don’t have a specific timeline for the next major Git release, this release includes decisions made towards Git 3.0.

The Git 3.0 release planning allows us to plan for and implement breaking changes and communicate them to the extended Git community. Next to documentation, Git can also be compiled with these breaking changes for those who want to experiment with these changes. More information can be found in the BreakingChanges document.

The Git 2.51.0 release makes some significant changes towards Git 3.0.

Reftable as the default reference backend

In the Git 2.45.0 release, the “reftable” format was introduced as a new backend for storing references like branches or tags in Git, which fixes many of the issues with the existing "files" backend. Please read our beginner's guide to how reftables work for more insight into the “reftable” backend.

The Git 2.51.0 release marks the switch to using the "reftable" format as default in Git 3.0 for newly created repositories and also wires up the change behind a feature flag. The “reftable” format provides the following improvements over the traditional “files” backend:

• It is impossible to store two references that only differ in casing on case-insensitive filesystems with the "files" format. This issue is common on Windows and macOS platforms. As the "reftable" backend does not use filesystem paths to encode reference names this problem goes away.

• Similarly, macOS normalizes path names that contain unicode characters, which has the consequence that you cannot store two names with unicode characters that are encoded differently with the "files" backend. Again, this is not an issue with the "reftable" backend.

• Deleting references with the "files" backend requires Git to rewrite the complete "packed-refs" file. In large repositories with many references this file can easily be dozens of megabytes in size; in extreme cases it may be gigabytes. The "reftable" backend uses tombstone markers for deleted references and thus does not have to rewrite all of its data.

• Repository housekeeping with the "files" backend typically performs all-into-one repacks of references. This can be quite expensive, and consequently housekeeping is a tradeoff between the number of loose references that accumulate and slow down operations that read references, and compressing those loose references into the "packed-refs" file. The "reftable" backend uses geometric compaction after every write, which amortizes costs and ensures that the backend is always in a well-maintained state.

• Operations that write multiple references at once are not atomic with the "files" backend. Consequently, Git may see in-between states when it reads references while a reference transaction is in the process of being committed to disk.

• Writing many references at once is slow with the "files" backend because every reference is created as a separate file. The "reftable" backend significantly outperforms the "files" backend by multiple orders of magnitude.

• The “reftable” backend uses a binary format with prefix compression for reference names. As a result, the format uses less space compared to the "packed-refs" file.

This project was led by Patrick Steinhardt.

SHA-256 as the default hash function

The Git version control system stores objects in a content-addressable filesystem. This means it uses the hash of an object to address content such as files, directories, and revisions, unlike traditional filesystems, which use sequential numbers. Using a hash function has the following advantages:

• Easy integrity checks as a single bit flip would change the hash output completely.

• Fast object lookup as objects can be indexed by their hash.

• Object names can be signed and third parties can trust the hash to address the signed object and all objects it references.

• Communication using Git protocol and out of band communication methods have a short reliable string that can be used to reliably address stored content.

Since its inception, Git has used the SHA-1 hashing algorithm. However, security researchers have discovered some flaws in SHA-1, specifically the SHAttered attack, which shows a practical SHA-1 hash collision. We moved to using a hardened SHA-1 implementation by default since Git 2.13.0. However, SHA-1 is still a weak hashing algorithm and it is only a matter of time before additional attacks will further reduce its security.

SHA-256 was identified as the successor to SHA-1 in late 2018. Git 2.51.0 marks it as the default hash algorithm to be used in Git 3.0.

This project was led by brian m. carlson.

Removal of git-whatchanged(1)

The git-whatchanged(1) command shows logs with differences each commit introduces. While this is now succeeded by git log --raw, the command was kept around for historical reasons.

Git 2.51.0 requires users of the command to explicitly use the --i-still-use-this flag to capture any users who still use the deprecated command, and also marks the command for removal in Git 3.0.

This project was led by Junio C Hamano.

git switch and git restore are no longer experimental

The git-checkout(1) command can be used for multiple different use cases. It can be used for switching references:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

$ git checkout next
Switched to branch 'next'
Your branch is up to date with 'origin/next'.

Or for restoring files:

$ echo "additional line" >> git.c

$ git status
On branch master
Your branch is up to date with 'origin/master’.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   git.c

no changes added to commit (use "git add" and/or "git commit -a")

$ git checkout git.c
Updated 1 path from the index

$ git status
On branch master
Your branch is up to date with 'origin/master’.

nothing to commit, working tree clean

For new users of Git, this can cause a lot of confusion. So in Git 2.33.0, these were split into two new commands, git-switch(1) and git-restore(1).

The git-switch(1) command allows users to switch to a specific branch:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

$ git switch next
Switched to branch 'next'
Your branch is up to date with 'origin/next'.

And the git-restore(1) command allows users to restore working tree files:

$ echo "additional line" >> git.c

$ git status
On branch master
Your branch is up to date with 'origin/master’.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   git.c

no changes added to commit (use "git add" and/or "git commit -a")

$ git restore git.c

$ git status
On branch master
Your branch is up to date with 'origin/master’.

nothing to commit, working tree clean

While the two commands have existed since 2019, they were marked as experimental. The effect is that the Git project doesn’t guarantee backwards compatibility for those commands: the behavior may change at any point in time. While the intent originally was to stabilize those commands after a couple of releases, this hasn’t happened up to this point.

This has led to several discussions on the Git mailing list where users are unsure whether they can start using these new commands, or whether they might eventually go away again. But given that no significant changes have ever been proposed, and that some users are already using these commands, we have decided to no longer declare them as experimental in Git 2.51.

This project was led by Justin Tobler.

git for-each-ref(1) receives pagination support

The git for-each-ref command is used to list all references present in the repository. As it is part of the plumbing layer of Git, this command is frequently used for example by hosting forges to list references that exist in the repository in their UI. But as repositories grow, it becomes less realistic to list all references at once – after all, the largest repositories may contain millions of them! So instead, forges tend to paginate the references.

This surfaces an important gap: git-for-each-ref does not know to skip references from previous pages that have already been shown. Consequently, it may have to list a large number of uninteresting references before it finally starts to yield the references required for the current page. This is inefficient and leads to higher-than-necessary latency or even timeouts.

Git 2.51.0 supports a new --start-after flag for git for-each-ref, which allows paginating the output. This can also be combined with the --count flag to iterate over a batch of references.

$ git for-each-ref --count=10
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-001
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-002
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-003
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-004
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-005
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-006
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-007
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-008
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-009
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-010

$ git for-each-ref --count=10 --start-after=refs/heads/branch-010
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-011
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-012
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-013
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-014
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-015
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-016
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-017
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-018
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-019
9751243fba48b34d29aabfc9784803617a806e81 commit    refs/heads/branch-020

This project was led by Karthik Nayak.

What's next?

Ready to experience these improvements? Update to Git 2.51.0 and start using git switch and git restore in your daily workflow.

For GitLab users, these performance enhancements will automatically improve your development experience once your Git version is updated.

Learn more in the official Git 2.51.0 release notes and explore our complete archive of Git development coverage.