Blog Engineering GitLab Infrastructure Update
September 26, 2016
2 min read

GitLab Infrastructure Update

Hear how we're working through infrastructure challenges as we scale.

infrastructure.jpeg

As Infrastructure Lead, my job is to make GitLab.com fast and highly available.

Lately, it's been a challenge. Why? We are hitting our threshold where scale starts to matter. For example, over 2,000 new repos are being created during peak hours, and CI runners are requesting new builds 3,000,000 times per hour. It's an interesting problem to have. We have to store this information somewhere and make sure that while we're gaining data and users, GitLab.com keeps working fine.

A large part of the issue we're running into as we scale is that there is little or no documentation on how to tackle this kind of problem. While there are companies that have written high-level posts, almost none of them have shared how they arrived at their solutions.

One of our main issues in the past six months has been around storage. We built a CephFS cluster to tackle both the capacity and performance issues of using NFS appliances. Another more recent issue is around PostgreSQL vacuuming and how it affects performance locking up the database given the right kind of load.

As outlined in our values, we believe we have a responsibility to document this so other companies know what to do when they reach this point. Last Thursday, I gave a GitLab.com infrastructure status report during our daily team call. Watch the recording or download the slides to see how we're working through our challenges with scaling.

Recording & Slides

Recording


Slides


We want to hear from you

Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum. Share your feedback

Ready to get started?

See what your team could do with a unified DevSecOps Platform.

Get free trial

New to GitLab and not sure where to start?

Get started guide

Learn about what GitLab can do for your team

Talk to an expert