Speed up your monorepo workflow in Git

Monorepos have grown in popularity in recent years. For many of us, they are a

part of our daily Git workflows. The trouble is working with them can be slow. Speeding up

a developer's workflow can reap huge savings in the long run for any team.

First, a word about monorepos. What does it mean for a repository to be a

monorepo anyway? Well, it depends who you ask and the definition has become

more flexible over time, but here are a few.

Characteristics of monorepos

Monorepos have the following characteristics.

Multiple sub-projects

The typical definition of "monorepo" is a repository that contains multiple sub-projects. For instance, let's imagine a repository with a web-facing front end,

a backend, an iOS app directory, and an android app directory:

awesome-app/

|

|--backend/

|

|--web-frontend/

|

|--app-ios/

|

|--app-android/

awesome-app is a single repository:

git clone https://my-favorite-git-hosting-service.com/awesome-app.git

The Chromium repository is a good

example of this.

Large files

Repositories can also grow to be very large if large files are checked in. In

some cases, binaries or other large assets such as images are checked into the

repository to have their history tracked. Other times, large files are inadvertently

introduced into the repository. The way Git history works, even if these files are

immediately removed, the single version that was checked in remains.

Old projects with deep histories

While Git is very good at compressing text files, when a Git repository has a deep history,

the need to keep all versions of a file around can cause the size of the repository to be huge.

The Linux repository is a good example of this.

For instance, the Linux project's first Git commit is from April 2005.

And a git rev-list --all --count gives us 1,120,826 commits! That's a lot of

history! Getting into Git internals a little bit, Git keeps a commit object, and a

tree object for each commit, as well as a copy of the files at that snapshot

in history. This means a deep Git history means a lot of Git data.

Speeding up your Git workflow

Here are some features to help speed up your Git workflow.

Sparse checkout

git sparse checkout reduces the

number of files you check out to a subset of the repository. (NOTE: This feature

in Git is still marked experimental.) This is especially useful in the case of

many sub-projects in a repository.

Taking our example of a monorepo with multiple

sub-projects, let's say that as a front-end web developer I only need to make

changes to web-frontend/.

> git clone --no-checkout
https://my-favorite-git-hosting-service.com/awesome-app.git

> cd awesome-app

> git sparse-checkout set web-frontend

> git checkout

Your branch is up to date with 'origin/master'.

> ls

> web-frontend README.md

Or, if you've already checked out a worktree, sparse checkout can be used to remove

files from the worktree.

> git clone https://my-favorite-git-hosting-service.com/awesome-app.git

> cd awesome-app

> ls

> backend web-frontend app-ios app-android README.md

> git sparse-checkout set web-frontend

Updating files: 100% (103452/103452), done.

> ls

> web-frontend README.md

Sparse checkout will only include the directories indicated, plus all files

directly under the root repository directory.

This way, we only checkout the directories that we need, saving both space locally

and time since each time git pull is done, only files that are checked out will

need to be updated.

More information can be found in the docs

for sparse checkout.

Partial clone

git partial clone has a similar goal to sparse checkout in reducing the number

of files in your local Git repository. It provides the option to filter out

certain types of files when cloning.

Partial clone is used by passing the --filter option to git-clone.

git clone --filter=blob:limit=10m

This will exclude any files over 10 megabytes from being copied to the local

repository. A full list of supported filters are included in the

[docs for git-rev-list](https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt