Production Architecture

Our core infrastructure is currently hosted on several cloud providers, all with different functions. This document does not cover servers that are not integral to the public facing operations of GitLab.com.

On this page

Diagram of the Architecture

Source, GitLab internal use only

Internal Networking Scheme

A visualisation of the whole address space can be found here (GitLab internal use only).

Production

Virtual Network Name: GitLabProd

Resource Group: GitLabProd

IP space: 10.64.0.0/11

Resource Group Availability Set Subnet Name Subnet Range Friendly group name
ExternalLBProd ExternalLBProd ExternalLBProd 10.65.1.0/24 Load balancers
InternalLBProd InternalLBProd InternalLBProd 10.65.2.0/24 Load balancers
DBProd DBProd DBProd 10.66.1.0/24 Databases
RedisProd RedisProd RedisProd 10.66.2.0/24 Databases
ConsulProd ConsulProd ConsulProd 10.67.1.0/24 Support Services
VaultProd VaultProd VaultProd 10.67.2.0/24 Support Services
DeployProd DeployProd DeployProd 10.67.3.0/24 Support Services
MonitoringProd MonitoringProd MonitoringProd 10.68.1.0/24 Monitoring and logs
LogProd LogProd LogProd 10.68.2.0/24 Monitoring and logs
APIProd APIProd APIProd 10.69.2.0/23 Services
GitProd GitProd GitProd 10.69.4.0/23 Services
SidekiqProd SidekiqProd SidekiqProd 10.69.6.0/23 Services
WebProd WebProd WebProd 10.69.8.0/23 Services
StorageProd StorageProd StorageProd 10.70.2.0/23 Storage

Canary

Virtual Network Name: GitLabCanary

Resource Group: GitLabCanary

IP space: 10.192.0.0/13

Resource Group Availability Set Subnet Name Subnet Range Friendly group name
ExternalLBStaging ExternalLBStaging ExternalLBStaging 10.192.1.0/24 Load balancers
InternalLBStaging InternalLBStaging InternalLBStaging 10.192.2.0/24 Load balancers
DBStaging DBStaging DBStaging 10.193.1.0/24 Databases
RedisStaging RedisStaging RedisStaging 10.193.2.0/24 Databases
ConsulStaging ConsulStaging ConsulStaging 10.194.1.0/24 Support Services
VaultStaging VaultStaging VaultStaging 10.194.2.0/24 Support Services
DeployStaging DeployStaging DeployStaging 10.194.3.0/24 Support Services
MonitoringStaging MonitoringStaging MonitoringStaging 10.195.1.0/24 Monitoring and logs
LogStaging LogStaging LogStaging 10.195.2.0/24 Monitoring and logs
APIStaging APIStaging APIStaging 10.196.2.0/23 Services
GitStaging GitStaging GitStaging 10.196.4.0/23 Services
SidekiqStaging SidekiqStaging SidekiqStaging 10.196.6.0/23 Services
WebStaging WebStaging WebStaging 10.196.8.0/23 Services
StorageStaging StorageStaging StorageStaging 10.197.2.0/23 Storage

Staging

Virtual Network Name: GitLabStaging

Resource Group: GitLabStaging

IP space: 10.128.0.0/12

Resource Group Availability Set Subnet Name Subnet Range Friendly group name
ExternalLBStaging ExternalLBStaging ExternalLBStaging 10.128.1.0/24 Load balancers
InternalLBStaging InternalLBStaging InternalLBStaging 10.128.2.0/24 Load balancers
DBStaging DBStaging DBStaging 10.129.1.0/24 Databases
RedisStaging RedisStaging RedisStaging 10.129.2.0/24 Databases
ConsulStaging ConsulStaging ConsulStaging 10.130.1.0/24 Support Services
VaultStaging VaultStaging VaultStaging 10.130.2.0/24 Support Services
DeployStaging DeployStaging DeployStaging 10.130.3.0/24 Support Services
MonitoringStaging MonitoringStaging MonitoringStaging 10.131.1.0/24 Monitoring and logs
LogStaging LogStaging LogStaging 10.131.2.0/24 Monitoring and logs
APIStaging APIStaging APIStaging 10.132.2.0/23 Services
GitStaging GitStaging GitStaging 10.132.4.0/23 Services
SidekiqStaging SidekiqStaging SidekiqStaging 10.132.6.0/23 Services
WebStaging WebStaging WebStaging 10.132.8.0/23 Services
StorageStaging StorageStaging StorageStaging 10.133.2.0/23 Storage

Azure

The main portion of GitLab.com is hosted on Microsoft Azure. We have the following servers there.

Note that these numbers can fluctuate to adapt to the platform needs.

We also use availability sets to ensure that a minimum number of servers in each group are available at any given time. This ensures that Azure will not reboot all instances in the same availability set at the same time for anything that is planned.

All our servers run the latest Ubuntu LTS unless there is a specific need to do otherwise. Every server is configured with a fully fledged set of firewall rules for increased security.

Load Balancers

We utilize Azure load balancers in front of our HAProxy nodes. This allows us to leverage on the Azure infrastructure for HA as well as taking advantage of the power of HAProxy.

Additionally, we utilize an Azure load balancer to manage PostgreSQL failovers.

Service nodes

Different services have different resource utilization patterns so we use a variety of instance types across our service nodes that are consistent for each group. We have recently isolated traffic by type on dedicated pools of nodes. We hope you noticed the performance improvement.

Digital Ocean

Digital Ocean houses several servers that do not need to directly interact with our main infrastructure. There are many of these that do a variety of things, however not all will be listed here.

The primary things on Digital Ocean at this time are:

AWS

We host our DNS with route53 and we have several EC2 instances for various purposes. The servers you will interact with most are listed Below

Google Cloud

We are currently investigating Google Cloud.

Monitoring

See how it's doing, for more information on that, visit the monitoring handbook.

Technology at GitLab

We use a lot of cool (but boring) technologies here at GitLab. Below is a non-exhaustive list of tech we use here.