Blog Engineering An inside look at the infrastructure of GitLab.com
April 29, 2016
3 min read

An inside look at the infrastructure of GitLab.com

In this post, you'll find out just how many servers we use. You'll gain some perspective on what those servers are up to.

goal.jpg

A number of people have asked about the infrastructure of GitLab.com. Our passionate
and curious Twitter followers inquired specifically about how many servers we use for
GitLab.com. Given the number of questions we've gotten on this topic, we wanted to go
ahead and offer an inside look at our GitLab.com infrastructure. In this post,
you'll find out just how many servers we use. You'll gain some perspective on what
those servers are up to.

Baseline

For running GitLab.com as an application we have:

  • 5 HAProxy load balancers that are handling GitLab.com HTTP, HTTPS, and SSH
  • 2 HAProxy load balancers that are handling "alternative SSH" (altssh.GitLab.com) so they do redirection from 443 to 22
  • 2 HAProxy load balancers that are handling https://pages.gitlab.io HTTP and HTTPS
  • 20 workers running GitLab EE application stack (Nginx, Workhorse, Unicorn + Rails, Redis + Sidekiq)
  • 2 NFS servers for the storage
  • 2 Redis servers
  • 2 PostgreSQL servers
  • 3 Elasticsearch servers

Those are servers that we manage directly. With that, the server count is at 38.

Next

We also use 6 of Azure's "Availability Sets": 3 for load balancers, 1 for Redis HA, 1 for
PostgreSQL HA, and 1 for Elasticsearch HA. Each of these Availability Sets has its own "internal"
load balancer that is managing the HA traffic. If we count them as a GitLab.com servers, then
we need to add 6 servers (now, the count is 44).

We also have 3 servers for GitLab Runners in autoscale mode. Two of them are managing autoscaling
of runners for GitLab CE/EE projects (so they are used only by GitLab and I will not count them).
But the third is used to manage autoscaling for Shared Runners at GitLab.com. So +1 for
the "Shared Runners manager."

We also have some servers that are specific for GitLab as a company (Runners for building
Omnibus packages, etc.) but I wouldn't count that as a part of GitLab.com.

And at the end, we have 45 servers that are used to make GitLab.com a usable application for our
users.

But wait, there's more

Ah! Don't forget about autoscaled Docker machines! Each user's builds are running on Docker hosts
created "on demand" by the autoscaling mode of the Runner. Last week, I looked at a diagram of the
machine utilization and it showed that we had:

  • minimum 12 machines running at once,
  • maximum 150 machines running at once,
  • an average of 54 machines running at once.

Because Shared Runners can be used by all GitLab.com users then I would count them as well!

Final count

So, the answer is, GitLab.com is currently running on 45 servers. However, if we also
count the build hosts for Shared Runners, then GitLab.com is using 60 to 200 servers!

We appreciate the question and the curiosity. As always, keep the questions coming!
You can also visit our Operations issue tracker for a live look at what
the team is working on.

We want to hear from you

Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum. Share your feedback

Ready to get started?

See what your team could do with a unified DevSecOps Platform.

Get free trial

New to GitLab and not sure where to start?

Get started guide

Learn about what GitLab can do for your team

Talk to an expert