Blog Engineering Speed up your monorepo workflow in Git
Published on: September 6, 2022
3 min read

Speed up your monorepo workflow in Git

Tap into the features that can reap huge savings in the long run for any developer team.

speedmonorepo.jpg

Monorepos have grown in popularity in recent years. For many of us, they are a part of our daily Git workflows. The trouble is working with them can be slow. Speeding up a developer's workflow can reap huge savings in the long run for any team.

First, a word about monorepos. What does it mean for a repository to be a monorepo anyway? Well, it depends who you ask and the definition has become more flexible over time, but here are a few.

Characteristics of monorepos

Monorepos have the following characteristics.

Multiple sub-projects

The typical definition of "monorepo" is a repository that contains multiple sub-projects. For instance, let's imagine a repository with a web-facing front end, a backend, an iOS app directory, and an android app directory:

awesome-app/
|
|--backend/
|
|--web-frontend/
|
|--app-ios/
|
|--app-android/

awesome-app is a single repository:

git clone https://my-favorite-git-hosting-service.com/awesome-app.git

The Chromium repository is a good example of this.

Large files

Repositories can also grow to be very large if large files are checked in. In some cases, binaries or other large assets such as images are checked into the repository to have their history tracked. Other times, large files are inadvertently introduced into the repository. The way Git history works, even if these files are immediately removed, the single version that was checked in remains.

Old projects with deep histories

While Git is very good at compressing text files, when a Git repository has a deep history, the need to keep all versions of a file around can cause the size of the repository to be huge.

The Linux repository is a good example of this.

For instance, the Linux project's first Git commit is from April 2005.

And a git rev-list --all --count gives us 1,120,826 commits! That's a lot of history! Getting into Git internals a little bit, Git keeps a commit object, and a tree object for each commit, as well as a copy of the files at that snapshot in history. This means a deep Git history means a lot of Git data.

Speeding up your Git workflow

Here are some features to help speed up your Git workflow.

Sparse checkout

git sparse checkout reduces the number of files you check out to a subset of the repository. (NOTE: This feature in Git is still marked experimental.) This is especially useful in the case of many sub-projects in a repository.

Taking our example of a monorepo with multiple sub-projects, let's say that as a front-end web developer I only need to make changes to web-frontend/.

> git clone --no-checkout https://my-favorite-git-hosting-service.com/awesome-app.git
> cd awesome-app
> git sparse-checkout set web-frontend
> git checkout
Your branch is up to date with 'origin/master'.
> ls
> web-frontend README.md

Or, if you've already checked out a worktree, sparse checkout can be used to remove files from the worktree.

> git clone https://my-favorite-git-hosting-service.com/awesome-app.git
> cd awesome-app
> ls
> backend web-frontend app-ios app-android README.md
> git sparse-checkout set web-frontend
Updating files: 100% (103452/103452), done.
> ls
> web-frontend README.md

Sparse checkout will only include the directories indicated, plus all files directly under the root repository directory.

This way, we only checkout the directories that we need, saving both space locally and time since each time git pull is done, only files that are checked out will need to be updated.

More information can be found in the docs for sparse checkout.

Partial clone

git partial clone has a similar goal to sparse checkout in reducing the number of files in your local Git repository. It provides the option to filter out certain types of files when cloning.

Partial clone is used by passing the --filter option to git-clone.

git clone --filter=blob:limit=10m

This will exclude any files over 10 megabytes from being copied to the local repository. A full list of supported filters are included in the [docs for git-rev-list](https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt

We want to hear from you

Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum. Share your feedback

Ready to get started?

See what your team could do with a unified DevSecOps Platform.

Get free trial

Find out which plan works best for your team

Learn about pricing

Learn about what GitLab can do for your team

Talk to an expert