Companies in all industries and sectors have significantly invested in digital transformation and increased their software development capabilities. GitLab delivers modern DevOps with a complete DevOps platform. However, some organizations require self-managed GitLab and GitLab Runners, which creates added costs for hosting and running GitLab infrastructure. Our latest cost analysis and performance benchmarks show that customers can realize cost savings of up to 23% and performance gains of up to 36% by deploying the GitLab application and GitLab Runner on the Arm-based Graviton2 when compared to the x86 based M5/C5 EC2 instances.
Arm is a leading provider of silicon intellectual property (IP) for intelligent systems-on-chip (SoC) that power billions of devices. GitLab and Arm have collaborated closely to make GitLab tools available for devices based on Arm architecture. AWS is the first major public cloud provider to offer Arm-based EC2 compute instances powered by Graviton2 processors built upon Arm Neoverse N1 IP cores.
Performance benchmarks for GitLab 10,000 reference architecture
GitLab is a highly scalable and modular application and can accomodate 10 users to 10,000 as a business scales. Today, the GitLab 10,000 reference architecture provides users with a blueprint for hosting GitLab on x86_64-backed compute on leading cloud platforms providers. Building upon our collaboration from last year, the next step was to include Arm64 backed compute in the reference architecture.
For the research, we first ran the performance benchmarks comparing the cost of hosting GitLab's Reference Architecture for up to 10,000 users on Arm64 and x86 environments on AWS. We found that GitLab customers can realize up to 23% cost savings on their AWS bill by deploying GitLab on Graviton2-based EC2 instances over comparable x86-based EC2 instances for about the same level of performance. See the monthly AWS cost for running this scenario on the Arm64 environment and x86 environment.
The figure below shows the components that make up the GitLab 10,000 reference architecture:
An example of components that make up a 10,000 user GitLab architecture.
Read more about the components required to set up the 10,000 architecture.
For testing, we used the GitLab Performance Tool developed in-house by GitLab to test the performance of GitLab. Below is a high-level view of the different kinds of tests generated by GPT.
The different types of tests created by the GitLab Performance Tool.
All data was generated under a group named gpt and split into two areas: vertical and horizontal. The vertical area consists of one or more large projects that are considered a good and representative size for performance testing. The horizontal area consists of a large number of subgroups that have a large number of projects. All of these subgroups are saved under the parent subgroup gpt/many_groups_and_projects.
We also used the GitLab Environment Toolkit (GET), a provisioning and configuration toolkit, for deploying GitLab's Reference Architectures with Terraform and Ansible.
About performance benchmarking for the self-managed GitLab Runner
GitLab Runner is the open source application that runs GitLab CI/CD jobs on various computing platforms and operating systems. The GitLab Runner has supported Arm architecture since GitLab 12.6, which allows users to run CI/CD jobs natively on Arm.
We ran the performance benchmark results for the GitLab Runner by compiling a standard Linux kernel on M6g and M5 instances. In this case, we demonstrated 36% performance gain on M6g instances under 100% CPU utilization. For example, it took 7 minutes and 53 seconds to compile Linux kernel on M5.xlarge (4 core) instances, whereas it only took 5 minutes 47 seconds on M6g.xlarge (4 core) instances.
The figure below shows that the architecture of the test setup for the GitLab runner we used to benchmark the performance based on the GitLab Runner stress test repository.
The architecture of the GitLab Runner used to benchmark performance.
We used Prometheus and Grafana to obtain the CPU utilization graphs for both M6g and M5 instances from the Runner. The diagrams below show we have 100% CPU utilization on both Arm and x86 environments, and we are still able to achieve a 36% performance gain with the GitLab Runners on Arm-based M6g instances.
Inside CPU use on Arm environment.
CPU use on x86 environments.
Users can benefit from the 36% performance gain for CI job execution and the roughly 23% per month in cost savings for executing CI jobs. The savings can be significant for customers that consume about 500,000 CI compute minutes per month.
GitLab customers increase performance and decrease cost by moving to Arm
GitLab enterprise customers can gain 36% in performance improvements and 23% cost savings by deploying GitLab and GitLab Runner on AWS Graviton2-based EC2 instance. If your company's cloud infrastructure is on AWS, then you should consider whether moving workloads to Arm-based Graviton2 instances is suitable for your organization.
Check out this repository for resources for getting started with AWS Graviton processors and information on supported operating systems and software. Feel free to open an issue if you have questions or need more help.
Join us at Arm DevSummit 2021 to learn more about GitLab performance benchmarking and other topics.
