A number of people have asked about the infrastructure of GitLab.com. Our passionate and curious Twitter followers inquired specifically about how many servers we use for GitLab.com. Given the number of questions we've gotten on this topic, we wanted to go ahead and offer an inside look at our GitLab.com infrastructure. In this post, you'll find out just how many servers we use. You'll gain some perspective on what those servers are up to.
Baseline
For running GitLab.com as an application we have:
- 5 HAProxy load balancers that are handling GitLab.com HTTP, HTTPS, and SSH
- 2 HAProxy load balancers that are handling "alternative SSH" (altssh.GitLab.com) so they do redirection from 443 to 22
- 2 HAProxy load balancers that are handling https://pages.gitlab.io HTTP and HTTPS
- 20 workers running GitLab EE application stack (Nginx, Workhorse, Unicorn + Rails, Redis + Sidekiq)
- 2 NFS servers for the storage
- 2 Redis servers
- 2 PostgreSQL servers
- 3 Elasticsearch servers
Those are servers that we manage directly. With that, the server count is at 38.
Next
We also use 6 of Azure's "Availability Sets": 3 for load balancers, 1 for Redis HA, 1 for PostgreSQL HA, and 1 for Elasticsearch HA. Each of these Availability Sets has its own "internal" load balancer that is managing the HA traffic. If we count them as a GitLab.com servers, then we need to add 6 servers (now, the count is 44).
We also have 3 servers for GitLab Runners in autoscale mode. Two of them are managing autoscaling of runners for GitLab CE/EE projects (so they are used only by GitLab and I will not count them). But the third is used to manage autoscaling for Shared Runners at GitLab.com. So +1 for the "Shared Runners manager."
We also have some servers that are specific for GitLab as a company (Runners for building Omnibus packages, etc.) but I wouldn't count that as a part of GitLab.com.
And at the end, we have 45 servers that are used to make GitLab.com a usable application for our users.
But wait, there's more
Ah! Don't forget about autoscaled Docker machines! Each user's builds are running on Docker hosts created "on demand" by the autoscaling mode of the Runner. Last week, I looked at a diagram of the machine utilization and it showed that we had:
- minimum 12 machines running at once,
- maximum 150 machines running at once,
- an average of 54 machines running at once.
Because Shared Runners can be used by all GitLab.com users then I would count them as well!
Final count
So, the answer is, GitLab.com is currently running on 45 servers. However, if we also count the build hosts for Shared Runners, then GitLab.com is using 60 to 200 servers!
We appreciate the question and the curiosity. As always, keep the questions coming! You can also visit our Operations issue tracker for a live look at what the team is working on.