Published on: January 20, 2021
4 min read
GitLab uses parallel jobs to help long-running jobs run faster.
At GitLab, we must verify simultaneous changes from the hundreds of people that contribute to GitLab each day. How can we help them contribute efficiently using our pipelines?
The pipelines that we use to build and verify GitLab have more than 90 jobs. Not all of those jobs are equal. Some are simple tasks that take a few seconds to finish, while others are long-running processes that must be optimized carefully.
At the time of this writing, we have more than 700 pipelines running. Each of these pipelines represent changes from team members and contributors from the wider community. All GitLab contributors must wait for the pipelines to finish to make sure the change works and integrates with the rest of the product. We want our pipelines to finish as fast as possible to maintain the productivity of our teams.
This is why we constantly monitor the duration of our pipelines. For example, in December 2020, successful merge request pipelines had a duration of 53.8 minutes:
The average pipeline took 53.8 minutes to finish in December.
Given that we run around 500 merge request pipelines per day, we want to know: Can we optimize our process to change how long-running jobs run?
The frontend-fixtures
job uses rspec
to generate mock data files, which are then saved as files called "fixtures". These files are loaded by our frontend tests, so the frontend-fixtures
must finish before any of our frontend tests can start.
As not all of our tests need these frontend fixtures, many jobs use the
needs
keyword to start before thefrontend-fixtures
job is done.
In our pipelines, this job looked like this:
Inside the frontend fixtures job.
This job had a normal duration of 20 minutes, and each individual fixture could be generated independently, so we knew there was an opportunity to run this process in parallel.
The next step was to configure our pipeline to split the job into multiple batches that could be run in parallel.
Fortunately, GitLab CI provides an easy way to run a job in parallel using the parallel
keyword. In the background, this creates "clones" of the same job, so that multiple copies of it can run simultaneously.
Before:
frontend-fixtures:
extends:
- .frontend-fixtures-base
- .frontend:rules:default-frontend-jobs
After:
rspec-ee frontend_fixture:
extends:
- .frontend-fixtures-base
- .frontend:rules:default-frontend-jobs
parallel: 2
You will notice two changes. First, we changed the name of the job, so our job is picked up by Knapsack (more on that later), and then we add the keyword parallel
, so the job gets duplicated and runs in parallel.
The new jobs that are generated look like this:
The new jobs that are picked up by Knapsack and run in parallel.
As we used a value of parallel: 2
, actually two jobs are generated with the names:
rspec-ee frontend_fixture 1/2
rspec-ee frontend_fixture 2/2
Our two "generated" jobs, now take three and 17 minutes respectively, giving us an overall decrease of about three minutes.
The parallel jobs that are running in the pipeline.
As we use Knapsack to distribute the test files among the parallel jobs, we were able to make more improvements by reducing the time it takes our longest-running fixtures-generator file to run.
We did this by splitting the file into smaller batches and optimizing it, so we have more tests running in parallel, which shaved off an additional ~3.5 minutes.
If you want to ramp up your productivity you can leverage parallel
on your pipelines by following these tips:
parallel
keyword, while measuring the outcome over the next few running pipelines.We discuss how running jobs in parallel improved the speed of pipelines on GitLab Unfiltered.
And here are links to some of the resources we used to run pipelines in parallel:
parallel
to fixtures.And many thanks to Rémy Coutable, who helped me implement this improvement.