How to migrate GitLab groups and projects more efficiently

Aug 2, 2023 · 8 min read
Magdalena Frankiewicz GitLab profile

Migrating groups and projects using direct transfer enables you to easily move GitLab resources between GitLab instances using either the UI or API. In a previous blog post, we announced the release of migrating projects as a beta feature available to everyone. We described the benefits of the method and steps to try it out.

Since then, we have made further improvements, especially focusing on efficient and reliable migrations for large projects. In this blog, we'll elaborate on these improvements, as well as their impact on the overall process and speed of migrations. We'll also discuss estimating the duration of migrations.

Imports of CI/CD pipelines

Problem: Timing out

We received a bug report about imports of CI/CD pipelines timing out and realized we needed to refine the underlying migration process. We considered the root cause of the problem and possible solutions, and ran proofs of concept. We concluded that we should tackle the problem of having one massive archive file for a project with a large number of a certain relation types (for example, pipelines).

What we improved

To fix the problem of timeouts, we decided to introduce batching to the process of exporting and importing relations (for example, merge requests or pipelines).

Before we could fully complete the epic for introducing the batching, we had to introduce a couple of other optimizations to the process of exporting CI/CD pipelines.

In GitLab 15.10, we started:

With these optimizations, exporting CI/CD pipelines sped up considerably. That allowed for a large number of pipelines in a project to be successfully exported to an archive file and then imported on the destination instance. However, because we were finally importing the pipelines, the overall duration of the migration increased.

In GitLab 16.3, we are introducing exporting and importing relations in batches. This has two benefits:

This improvement is already available by default on

Can we estimate the duration of a migration?

This question has been asked time and again. The answer is that duration of migration with direct transfer depends on many different factors. Some of them are:

There’s no exact formula to reliably estimate a migration. However, we checked the duration of each job importing a project relation to share with you the average numbers, so you can get an idea of how long importing your projects might take. Here is what we found:

You can find more project relations and the average duration to import them in the table below.

Project resource type Average time (in seconds) to import a single record
Empty project 2.4
Repository 20
Project attributes 1.5
Members 0.2
Labels 0.1
Milestones 0.07
Badges 0.1
Issues 0.1
Snippets 0.05
Snippet repositories 0.5
Boards 0.1
Merge requests 1
External pull requests 0.5
Protected branches 0.1
Project feature 0.3
Container expiration policy 0.3
Service desk setting 0.3
Releases 0.1
CI/CD pipelines 0.2
Commit notes 0.05
Wiki 10
Uploads 0.5
LFS objects 0.5
Design 0.1
Auto DevOps 0.1
Pipeline schedules 0.5
References 5
Push rule 0.1

How can we migrate efficiently?

We also know what is needed to achieve the most efficient migration possible.

A single direct transfer migration runs up to five entities (groups or projects) per import at a time, independent of the number of Sidekiq workers available on the destination instance. Importing five concurrent entities is the maximum allowed per migration by direct transfer. This limit has been set to not overload the source GitLab instance, because we saw network timeouts from source instances when we removed this limitation.

That doesn't mean that if more than five Sidekiq workers are available on the destination instance that they won't be utilized during a migration. On the contrary, more Sidekiq workers help speed up the migration by decreasing the time it takes to import each entity. Import of relations is distributed across multiple jobs and a single project entity has over 30 relations to be migrated. Exporting and importing relations in batches mentioned above results in even more jobs to be processed by the Sidekiq workers.

Increasing the number of Sidekiq workers on the destination instance helps speed up the migration until the source instance hardware resources are saturated. For more information on increasing the number of Sidekiq workers (increasing concurrency), see Set up Sidekiq instance.

The number of Sidekiq workers on the source instance should at least be enough to export the five concurrent entities in parallel (for each running import). Otherwise, there will be delays and potential timeouts as the destination is waiting for exported data to become available.

Distributing projects in different groups helps to avoid timeouts. If several large projects are in the same group, you can:

  1. Move large projects to different groups or subgroups.
  2. Start separate migrations each group and subgroup.

The GitLab UI can only migrate top-level groups. Using the API, you can also migrate subgroups.

What's next for migrating by direct transfer?

Of course, we're not done yet! We will continue to improve the direct transfer method, aiming towards coming out of beta and into general availability. Next, we are working on:

More details can be found on our migrating by direct transfer roadmap direction page. We are excited about this roadmap and hope you are too!

We want to hear from you. What's the most important missing piece for you? What else can we improve? Let us know in the feedback issue or schedule time with the Import and Integrations group product manager, and we'll keep iterating!

Disclaimer: This blog contains information related to upcoming products, features, and functionality. It is important to note that the information in this blog post is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. As with all projects, the items mentioned in this blog and linked pages are subject to change or delay. The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab.

Cover image by Adrien VIN on Unsplash

“Learn about performance improvements to GitLab migration by direct transfer and what's next.” – Magdalena Frankiewicz

Click to tweet

Edit this page View source