The following page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features or functionality remain at the sole discretion of GitLab Inc.
Section | Core Platform |
Maturity | Non-marketable |
Content Last Reviewed | 2024-03-04 |
The Gitaly direction page belongs to the Systems Stage within the Core Platform section, and is maintained by Mark Wood. The Gitaly Engineering team and stable counterparts can be found on the Engineering team page.
This strategy is a work in progress, and everyone can contribute. Please comment and contribute in the linked issues and epics. Sharing your feedback directly on GitLab.com is the best way to contribute to our strategy and vision.
If you would like support from the Gitaly team, please see the team's page detailing How to contact the Gitaly team.
Gitaly is the service responsible for the storage and maintenance of all Git repositories in GitLab for both our GitLab.com customers, as well as self-managed customers. Git repositories are essential to GitLab, for Source Code Management, Wikis, Snippets, Design Management, and Web IDE. Every stage of the DevOps lifecycle to the right of Create - Verify, Package, Release, Configure, Govern, Monitor and Secure - depends on the project repositories. Because the majority of GitLab capabilities depend on information stored in Git repositories, performance and availability are of primary importance.
GitLab is used to store Git repositories by small teams of a few people all the way up to large enterprises with many terabytes of data. For this reason, Gitaly has been built to scale from small single server GitLab instances, to large high availability architectures.
Gitaly provides multiple interfaces to read and write Git data:
While GitLab is the largest user of the Gitaly project, it is important to note that Gitaly is a standalone project that can be adopted separately from GitLab. As such, we strive to ensure that all business specific decisions are made within the GitLab application. Our belief is that Gitaly should provide the ability for management interfaces, but not make any specific management decisions.
For example, some users may want the ability to move repositories between different storage nodes for either cost savings or performance reasons. While Gitaly should provide an easy to use interface to efficiently move repositories, the calling application should be making the decisions around which repositories to move where.
Processes requiring no business data or inputs should be fully contained within Gitaly. These types of processes include repository maintenance and storage maintenance type tasks. We believe that these types of features provide substantial value for projects utilizing Gitaly and provide a compelling reason to chose Gitaly as a repository storage architecture.
The Gitaly team is divided into two sub-teams which both strive to provide scalable and performant Git repository storage. These two teams are the Git team and the Cluster team, and represent two distinct focuses for the Gitaly group. (For additional details, see the team page.
Git Team
Gitaly is built on top of the open source Git project, and we feel very strongly that we should be contributing to this project. As users and ambassadors of Git, it is our goal to actively engage in the Git community through bug fixes, feature additions, and mentoring. One of this team's major goals is to ensure we continue to enhance the Git project.
In addition to contributing to upstream Git, the Gitaly Git team is responsible for integrating Git within the Gitaly product. This involves optimizing our internal usage of Git, performing housekeeping on repository data and generally ensuring that the repository storage for our customers is performing as well as possible for all shapes and sizes of data.
Cluster Team
The Cluster team's goal is to provide a durable, performant, and reliable Git storage layer for GitLab. This team will tightly collaborate with the Gitaly Git team as they will build directly upon the Git tooling. Areas where the Cluster team will focus on is High Availability Git storage (Gitaly Cluster), scalability of Gitaly storage mechanisms, and improved user and administrative experiences.
In order to support highly available Git repository storage, Gitaly Cluster has been released. This provides redundant storage benefits such as voted writes, read distribution, and data redundancy. For full documentation, please see the details on Configuring Gitaly Cluster.
To fully appreciate the use-case for Gitaly Cluster, we must first clarify the role of highly available repository storage. From the Gitaly perspective, highly available (HA) storage means that a fault-tolerant interface for repository data exists, such that the loss of a single storage node will not compromise the ability to read / write Git data. Gitaly cluster fulfills this role by providing an interface to define multiple Gitaly storage nodes, and set a replication factor for stored repositories (how many nodes each repository should be stored on). In the event of a storage node loss, read and write operations continue as before, and when the cluster is returned to full capacity, the data is re-replicated to the returning node.
What HA repository storage does not provide is improved performance. Though in some cases write performance improves (through read distribution), the general concept is that you are trading storage cost and potential performance impacts for fault tolerance.
The one year plan for Gitaly Group is founded on three main principals discussed below. As Gitaly is one of the foundational elements of GitLab, these principals were chosen ensure that Gitaly is ready to meet future business needs.
In the unlikely event that there is a repository data issue necessitating in data restoration, we want to ensure that there are sufficient tools in place to allow our customers to be up and running as efficiently and painlessly as possible. To this end, the Gitaly team are actively involved in the GitLab Disaster Recovery Working Group where we play a critical role in defining opportunities to improve overall repository data recovery time objectives (RTO) and recovery point objectives (RPO). We also meet regularly with customers to better understand our self-managed customer needs.
Our one year focus areas are:
As our customers deploy larger installations of GitLab, they start to hit more pain points around the scalability of Gitaly and its repository management. This is especially true in light of our intent to make cloud-native deployments of Gitaly a reality, where nodes may be spun up and down more regularly.
Our goals here are:
We recognize that administrators of self-managed instances want to more easily configure and manage repository storage. In order to assist in both of these user journeys, we plan to do the following:
Over the next quarter, the Gitaly team is focused on the following areas. While this is not an exhaustive list, it does give some insight into our major focus areas.
As our customers continue to grow their repositories (both in count and in size), it is critical that the underlying Gitaly services can scale appropriately to meet these demands. As a team, we have decided that the solution to several of the existing scalability issues surrounding Gitaly cluster is a paradigm shift in how we replicate Git data. As such, we have begun initial efforts in shifting to a decentralized architecture for Gitaly Cluster.
The highlights of this forward thinking approach are as follows:
While this is not an area we own as a team, we believe that it is crucial for us to support the teams working in these areas to ensure that Git backups and data management allows for rapid and accurate restoration of Git data. The team is heavily involved in leading the GitLab disaster recovery working group. As leaders in this working group, we're collaborating with a broad cross-functional team to help ensure our service's success long-term. As part of this effort, we're architecting a solution for implementing write-ahead logging on GitLab.com.
We are in the beginning stages of investigating the known limitations around Gitaly running well in Kubernetes. This is going to be an ongoing theme across the next year, but we wanted to begin by ensuring that we understood first and foremost what issues exist today. The team is taking a data driven approach involving load testing in an effort to ensure we have clearly identified functional gaps.
In order to best represent our Transparency Value, it is just as important to clarify what the Gitaly team cannot prioritize currently. This does not mean that we do not recognize the need for some of these features, simply that we have a finite team.
Better Support for Administrative User Journeys
We want to ensure that in the future, we support user journeys such as adding, removing, and replacing nodes cleanly, and provide a basic administrative dashboard to monitor node health.
BIC (Best In Class) is an indicator of forecated near-term market performance based on a combination of factors, including analyst views, market news, and feedback from the sales and product teams. It is critical that we understand where GitLab appears in the BIC landscape.
The version control systems market is expected to be valued at close to US$550mn in the year 2021 and is estimated to reach US$971.8md by 2027 according to Future Market Insights which is broadly consistent with revenue estimates of GitHub ($250mn ARR) and Perforce ($130mn ARR). The opportunity for GitLab to grow with the market, and grow it's share of the version control market is significant.
Git is the market leading version control system, demonstrated by the 2018 Stack Overflow Developer Survey where over 88% of respondents use Git. Although there are alternatives to Git, Git remains dominant in open source software, usage by developers continues to grow, it installed by default on macOS and Linux, and the project itself continues to adapt to meet the needs of larger projects and enterprise customers who are adopting Git, like the Microsoft Windows project.
According to a 2016 Bitrise survey of mobile app developers, 62% of apps hosted by SaaS provider were hosted in GitHub, and 95% of apps are hosted in by a SaaS provider. These numbers provide an incomplete view of the industry, but broadly represent the large opportunity for growth in SaaS hosting on GitLab.com, and in self hosted where GitLab is already very successful.
Support large repositories
As applications mature, the existing code base continues to grow. As such, average repository sizes are on the rise and version control systems must be able to handle these large repositories in a performant manner. Additionally, many development tasks may require version control of large files, which again, should be handled seamlessly.
Ensure data safety
Application code has a very high value to organizations. It is unacceptable to have a solution which does not make it easy to ensure the integrity of your data, as well as provide easy means of backing up and restoring your data should something go wrong. Ideally, these solutions should use efficient and cost effective storage to optimize your business infrastructure.
Important competitors are GitHub.com and Perforce which, in relation to Gitaly, compete with GitLab in terms of raw Git performance and support for enormous repositories respectively.
Customers and prospects evaluating GitLab (GitLab.com and self hosted) benchmark GitLab's performance against GitHub.com, including Git performance. The Git performance of GitLab.com for easily benchmarked operations like cloning, fetching and pushing, show that GitLab.com similar to GitHub.com.
Perforce competes with GitLab primarily on its ability to support enormous repositories, either from binary files or monolithic repositories with extremely large numbers of files and history. This competitive advantage comes naturally from its centralized design which means only the files immediately needed by the user are downloaded. Given sufficient support in Git for partial clone, and sufficient performance in GitLab for enormous repositories, existing customers are waiting to migrate to GitLab.
Git lies at the core of what Gitaly does and is of critical importance to GitLab. We are aiming to increase up our investment into the Git community to ensure that it stays healthy in the long term.
Direction for the GitLab contributions to the Git project is tracked separately from the Gitaly direction, even though they are closely tied to each other. This is done because not only is Git a standalone project, but more importantly it is a community-led open source project, and thus not under direct control of the Gitaly team.
While we can influence the direction of Git somewhat by being regular contributors to the project and building trust and credit with the community, it will never be (and should not be!) under our full control. This is only natural in an open source project and also by design, as the success of Git relies on a healthy and diverse community with interests that may at times be conflicting with each other. We want to foster that community and collaborate with it, including with our direct competitors.
There's no question, repository data sets are growing rapidly. With the prevalence of machine learning and artificial intelligence adoptions, we're seeing a steady increase in repository size as well as the number of files needing to be stored. The Git team needs to ensure that repository storage scales with these expanding data storage needs.
Monorepo performance is a complex topic that involves many different parts of Git.
As projects grow older and grow more commits, the repository's object graph naturally expands. This creates scalability issues over time regardless of the respective object sizes. We want to ensure that the object database of Git repositories can scale well when facing repositories with dozens of millions of objects.
Our goals here are:
Pluggable object databases: The object storage format and the transport format of Git are tightly coupled. Furthermore, access to the object format happens rather ad-hoc without a lot of abstraction. It is clear though that the current design of the object database has limited scalability, and the more features we pile on top of it the harder it becomes to continue scaling.
We need to get back to the drawing board and explore whether it is feasible to make the object database pluggable. This is a huge effort and will likely span over multiple years, but it is needed to ensure long-term scalability of Git.
Better support for bundle URIs: We have started to use bundle URIs to help reduce the load on Gitaly servers. This is especially important in the context of CI systems. The feature is still relatively young though, and there are issues with both the implementation and the UX. We need to invest more time to help polish this feature.
Despite issues with large object graphs, Git is also known to not scale well with large objects like binary files. These are becoming increasingly more important with the advent of AI, but are also critical to open up to additional markets that are heaviliy dependent on binary assets like game studios.
Our goals are:
Offloading blobs through promisor remotes: We need server side changes to enable a triangular workflow between Git server, client, and a promisor remote which will host blobs that we offload from the git server.
Allow easy access to promisor remotes: Make it easier for Git to access promisor remotes. If we have a lot of objects offloaded to object storage because they are rarely used, Git should be able to access the objects and metadata easily without having to pull the objects back into the repository.
Storing large objects more efficiently: Investigate whether we can store large objects more efficiently and allow for deduplication when such large objects are changed. One example would be to use rolling hash functions to split up large objects better, which would potentially allow us to store changes to such large objects incrementally. Furthermore, if we are able to expose those chunks via the transport layer, then this would potentially be able to provide a native replacement for Git LFS when combined with partial clones.
Sending large objects more efficiently: Next to the problem of storing large objects efficiently, we also need to investigate how to send incremental changes to large objects over the wire efficiently. This may require us to expose the above representation with rolling hash functions to clients to efficiently compute deltas. An alternative would be to introduce a new object type that allows us to serve large binaries more efficiently.
We have seen repeatedly that Git and Gitaly start to struggle in repositories that have millions of references. While one approach should be to reduce the number of references that we need to store, we also need to acknowledge that our customers have valid usecases for repositories with this many references. Adapting Git so that it can handle repositories with many references is thus important.
Our goals here are:
Implement reftable support in GitLab: Reftables are a new storage data structure within Git that we have the privilege of contributing to upstream Git. This back end for Git references will improve the performance of repositories with many references, as well as eliminate the current race condition around updating reference. This will translate into better support for monorepos and will help ensure continued stability for our GitLab.com offering.
Performance improvements for the reftable backend: While the reftable has been upstreamed and generally outperforms the files backend in most scenarios, there still are lots of performance optimizations feasible. To reap the full benefits of this new backend we should invest more time into this.
Partial-update transactions: For the reftable backend to operate efficiently, updates of multiple references should ideally happen in a single transaction. Tools like git-fetch(1) may want to update only a subset of those refs though where the expected state matches the actual state of a reference. This is done via separate transactions. With the introduction of partial-update transactions, we allow evicting references from such a transaction, while still continuing with remaining references.
Many commands in Git scale with the size of the file tree, and large monorepos tend to have file trees that are both deep and broad. This creates bottlenecks in commands that scale with tree size. Adapting those commands to handle such large trees better is thus another angle for ensuring future scalability.
Our goals here are:
Better UI for sparse checkouts (client side): Sparse checkouts help users scale by not requiring a fully checked out worktree. For one, this saves disk space. Second, this allows users to skip fetching many blobs for partial clones. The sparse checkout tool in Git is hardly used because of usability challenges. We can vastly improve this command to make it more easily usable for monorepo cases.
git-replay(1): Improve git-replay(1) to support additional usecases for cherry-picks, rebases, rebasing merges and reverts.
git-blame-tree(1): Upstream git-blame-tree(1) to improve performance when
computing the last commit entries in a specific tree have been changed. This
should speed up RPCs like ListLastCommitsForTree()
significantly.
Many of our customers have expressed a big interest in cloud-native deployments of Gitaly. Our official stance is that this is not supported, where a huge contributor to this stance is that Git does not handle constrained environments well.
Our goals here are:
Git can be an intimidating tool with very complex syntax. This poses a direct threat because it allows a contender in the same space to take away market share if they provide a significantly better user experience. This has been showing with the recent advent of Jujutsu, which is claimed by many to provide a superior user experience compared to Git.
Our goals here are:
More streamlined interfaces: The user interfaces provided by Git have grown organically over time and are often redundant, inconsistent, confusing and/or hard to discover. We want to invest the time to make the interfaces more consistent and thus easier to use. We want to help the community to adopt a set of guidelines to help adapt existing and design future user interfaces.
Better user journey with large repositories: Provide a better user journey for interacting with very large repositories (in collaboration with the Source Code group).
Observe challengers like Jujutsu: We want to observe development of challengers like Jujutsu (JJ), a new version control system funded by Google for improved user experience with monorepos. We have noticed that many users praise its usability, especially compared to Git. As JJ can use Git repositories as backend, we may want to investigate whether it may be a good addition to the toolchain of users.
Furthermore, we may want to help Google with the libification of Git in this context. If the Git backend used by JJ plays well with Git due to this effort, then we ensure that JJ will stay compatible with repositories hosted on GitLab.
By staying close to developments in JJ, we position ourselves such that we can start to pivot to JJ if this ever becomes necessary.
As we have started to invest more into the Git community, we need to treat it with the same focus on maintainability as we treat projects that are owned by GitLab. We want to ensure that the codebase remains accessible to existing and new contributors and readily extensible. As we entrust Git with handling our customer's most important data, we also need to ensure that the Git project is safe.
Our goals here are:
Integer overflows: During the audit of X41 D-Sec we found many integer overflows in Git that can lead to exploitable out-of-bounds writes. The worst issues were able to trigger remote code execution bugs in Gitaly. While known instances have been fixed, Git is full of implicit integer conversions that can lead to this kind of bug. We should invest the time to harden Git against this kind of vulnerability and fix this whole class of bugs. This requires lots of fixes throughout the codebase so that we can enable relevant compiler warnings.
Memory leaks: While many parts of Git are leak free, others aren't. This is reflected in Git's test suite, where only a subset of tests run with the leak sanitizer enabled. This makes it easy to introduce new memory leaks that go unnoticed. The effect is that we are working against a moving target when trying to reduce memory usage of Git overall. We want to help the Git project such that the whole test suite is free of memory leaks.
Libification: The Git project provides the command line tools as the only
interface to Git repositories, which makes it hard to embed Git-related
functionality into applications directly. There are ongoing efforts to also
provide a linkable library that exposes parts of Git's functionality to such
applications. This requires a large effort to make Git play nice for such a
usecases: removal of global state, removing calls to exit()
and abort()
and
general refactorings of the to-be-exposed interfaces. We want to help this
effort to eventually have a proper Git library, which would also open up the
ability for ourselves to for example provide long-running Git daemons.
Structured error handling: Improve error handling such that Git can return information about the root cause of failures in a structured way. We currently have to rely on parsing error messages to figure out what exactly has gone wrong, which is fragile and often leads to bugs.
Improved tracking of direction While core developers of the Git community are typically aware of ongoing projects, this is less true for the wider community. This makes it hard to pick up work as a new contributor, requires the community to often re-hash similar discussions, and doesn't allow outside parties to learn about the direction. We want to help the Git project to track direction better to improve the flow of information.
Partial Clone is built-in to Git and available in GitLab 13.0 or newer. Scalar is compatible with partial clone, and Microsoft is contributing to its improvement based on their learnings from the VFS for Git protocol.
Divergent solution for CDN Offloading
While we recognize that a lot of good work has gone into independent solutions, we are committed to work with the Git community on a CDN approach. We intend to support, implement, and contribute to this solution as it be comes available. This is currently being explored in our Support Git CDN offloading epic.
This section contains messaging, questions, and resources for our sales counterparts to successfully position and sell Gitaly Cluster. It is important to note that Gitaly Cluster is not perfect for every installation. Our goal is to provide options for our customers so they can choose the best repository storage mechanism for their particular business needs.
Gitaly is a centralized service which handles all access to files to file storage for GitLab. Gitaly services Git requests from the GitLab web application, command line, and via the API. Gitaly is highly configurable and can utilize one or more storage locations to read / write repository data.
The Gitaly service is required for all GitLab installs, and is a separate product from Gitaly Cluster. While Gitaly handles accessing repository storage, Gitaly Cluster provides a highly available repository storage solution for our customers.
Gitaly Cluster was built to address the industry-wide difficulty around expanding Git repository storage in addition to the lack of high availability (HA) Git storage for critical applications. A prominent theme in industry is the idea of an ever expanding NFS storage location for repository storage. While this can work, over time performance degrades, and management becomes increasingly complex. Additionally, while the NFS file system is ideal for many types of files, it's well documented that the types of file accesses created by Git repository access can cause performance issues.
Our goal with Gitaly cluster is to build a Git repository storage system capable of scaling with our users needs, and providing a configurable level of redundancy to keep businesses operating, iterating, and growing.
Gitaly Cluster is a unique open-core project aimed at providing a scalable and high availability platform for Git repository storage. Gitaly Cluster enable horizontal scalability, allowing our customers to grow their storage in a simple, and well defined manner. We also capitalize on the redundant copies of data needed for HA by increasing read performance through read-distribution.
Customers should utilize Gitaly Cluster in a few key situations:
Customers may not desire to utilize Gitaly Cluster for the following reasons:
Documentation Resources
Enablement Presentation (Internal GitLab Only)
As Gitaly and Gitaly Cluster evolve, it is sometimes necessary to deprecate features. When this occurs, we will follow the documented Deprecations, removals and breaking changes procedure. This ensures that all stable counterparts within GitLab are informed, and that the GitLab Documentation is also updated to keep our customers informed.
In addition, we will track all deprecations throughout the 16.x milestones, and breaking changes occurring in the 17.0 milestone on the following epic:
Gitaly is a non-marketable category, and is therefore not assigned a maturity level.
Systems Administrators directly interact with Gitaly when installing, configuring, and managing a GitLab server, particularly when high availability is a requirement. In the past, systems administrators needed to create and manage an NFS cluster to configure a high availability GitLab instance, and manually manage the failover to new Gitaly nodes mounted on the same NFS cluster. In order to scale such a solution individual storage nodes needed to be re-sized, or a sharded Gitaly approach was required. Now that Gitaly Cluster is available, is possible to eliminate the NFS cluster from architecture and rely on Gitaly for replication. Gitaly Cluster brings with it automatic failover, horizontal scaling, and read access across replicas will deliver 99.999% uptime (five 9's) and improved performance without regular intervention. Systems administrators will have fewer applications to manage, as the last projects are migrated to GitLab and other version control systems are retired.
Developers will benefit from increasing performance of repositories of all shapes and sizes, on the command line and in the GitLab application, as performance improvements continue. Once support for monolithic repositories reaches minimal and continues maturing, developers will no longer be split between Git and legacy version control systems, as projects consolidate increasingly on Git. Developers that heavily use binary assets, like Game Developers, will at long last be able to switch to Git and eliminate Git LFS by adopting native large file support in Git.