In the past 8 months gitlab-workhorse, a 'weekend project' written in Go
instead of our preferred Ruby, grew from a tiny program that addressed
git clone timeouts into a critical component that touches almost all
HTTP requests to GitLab. In this blog post I will reflect on how we got
here.
Technical and personal motivations
GitLab is a Ruby on Rails web application that uses the
Unicorn Ruby web server. I am a fan of
Unicorn because it makes application resource leaks manageable and
because it has served GitLab well for a long time by patching up
problems which we found or find too hard to solve 'properly'. (I am
known to growl at people who suggest swapping out Unicorn for another
web server in GitLab.)
At the same time, the design of Unicorn is incompatible with one of
GitLab's main functions, namely Git repository access (git clone,
git push, etc.) via HTTP(S). The reason it is incompatible is that
Unicorn heavily relies on (relatively) short request timeouts. If you
configure Unicorn to time out slowly rather than quickly then it starts
to become a lot less pleasant to work with. A git clone on the other
hand may take quite a long time if you are fetching a large Git
repository. In my previous role as a service engineer at GitLab I
regularly had to explain this tension to customers. The only solution we
could offer was 'use Git over SSH instead'.
Another factor that led to gitlab-workhorse was my unfulfilled curiosity
about the Go programming language. Go is
sometimes credited with (or discredited for) having a strong marketing
push behind it. The marketing worked on me: I have had a plush Go mascot
staring at me on my desk for almost three years now.
A weekend project gets merged into master
So one weekend in July last year I found myself with an itch to build
something in Go and a lack of imagination which led me to ask myself:
could I rewrite
gitlab-grack, the GitLab
component that responds to Git HTTP clients, in Go? For the record I am
not proud of having used my own (non-work) time to drive development of
this project for the first few months, I think it sets a bad example for
myself and others. But that is how it went.
The result was a very short and mostly correct Go program called
'gitlab-git-http-server' that suffered from none of the timeout issues
that git clone via Unicorn had. It integrated with GitLab in a sneaky
way by letting NGINX divert Git HTTP requests away from Unicorn to
gitlab-git-http-server. The required changes in the GitLab Rails
codebase were so minor that I could easily hide them behind a feature
flag. To top it off I
announced gitlab-git-http-server to the team on a day the CTO was on
vacation: the ultimate sneak attack. Just kidding, but I thought it was
a funny coincidence about Dmitriy's
vacation.
The team somehow let me 'try gitlab-git-http-server out' (read: merge it
into master and deploy it on our staging server) and so it got started.
Less than a month later GitLab 7.14 shipped with gitlab-git-http-server
behind a feature flag. This allowed us to test it on GitLab.com for a
month. Between our staging environment and GitLab.com we were able to
catch the worst bugs. In GitLab 8.0 (released the month after 7.14)
we made gitlab-git-http-server an official (and required!) component of
GitLab.
The acceptance of gitlab-git-http-server by the team was probably helped
by a shared understanding that GitLab's Git-over-HTTP solution was just
not quite cutting it, and by the fact that we already used Go for
gitlab-ci-multi-runner.
But there was no up-front decision to solve the problem of Git-over-HTTP
at this particular time, or using these means.
Feature creep
Until now gitlab-git-http-server was a one-trick pony: it only handled
Git HTTP clients. But as usually happens when you have a new hammer,
other things started looking like nails. In GitLab 8.1 we changed
gitlab-git-http-server to also handle the 'download zip' button from the
GitLab web interface. In retrospect this seems obvious but at the time
it felt like a big leap: in our minds, gitlab-git-http-server was a
daemon that understood (stateless) HTTP Basic
Authentication
but not the session cookies used by GitLab to identify individual users.
But a session cookie is just an HTTP header so nothing stopped
gitlab-git-http-server from 'impersonating' a logged-in user and
generating a zip file for them on the fly. I have a bit of a hard time
explaining now why but we thought this was very neat at the time.
Time for a new name
In GitLab 8.2 we wanted to ship two new features for which we expected
the same sort of Unicorn timeout problems that had plagued Git-over-HTTP
in the past: Git LFS support (developed
by Marin) and CI build
artifacts
(developed by Kamil). Both of these
features depended on users uploading and downloading arbitrarily large
files.
This development brought many improvements to gitlab-git-http-server.
First of all, the more people had to say or write
'gitlab-git-http-server', the more obvious it became that the name was
too awkward. And with all these new features it was also no longer
appropriate, because the program did more than dealing with
Git-over-HTTP. We have Marin to thank for coming up with
'gitlab-workhorse' which I especially like because it pokes fun at
Unicorn.
It was also a great development for gitlab-workhorse to be getting
attention from Kamil because he is our resident Go expert. This was very
welcome: I felt confident enough that gitlab-workhorse functioned
correctly, but I am not an experienced Go programmer. Having Kamil in
the game helped us make gitlab-workhorse a better Go program.
For a short while, Marin and I were on the one hand trying to implement
file uploads/downloads in gitlab-workhorse, while Kamil on the other
hand was implementing the same thing for CI artifacts using NGINX
plugins. Luckily we spotted the duplication of efforts before the code
went out the door so we were able to implement this in gitlab-workhorse
together for GitLab 8.2.
We ended up with an especially nice solution for file downloads in
gitlab-workhorse, inspired by the mechanism Kamil intended to use in
NGINX: X-Sendfile headers. Most of the time when you want to use
gitlab-workhorse to make something faster or more robust in GitLab you
have to write both Ruby code and Go code. But because Ruby on Rails
understands X-Sendfile
already,
GitLab developers can reap the benefits of gitlab-workhorse for file
downloads without writing any Go code!
Betting the farm
By this time the success with which we could build new GitLab features
by doing part of the work in gitlab-workhorse started to cause problems
of its own. Each time we added a feature to gitlab-workhorse that meant
diverting more HTTP requests to gitlab-workhorse in the NGINX
configuration. This complexity was hidden from people who installed
GitLab using our Omnibus packages
but I could tell from the gitlab-workhorse issue tracker that this was a
recurring source of problems for installations from source.
Prior to gitlab-workhorse, NGINX served static files or forwarded
requests to Unicorn:
+----------+ +-------------+
| | | |
| NGINX +----> | Unicorn |
| | | |
+------+---+ +-------------+
|
|
| +------------+
| | |
+--------> | static |
| files |
| |
+------------+
Now with gitlab-workhorse in the picture, NGINX had to know which
requests to send to Unicorn, which to gitlab-workhorse, and which to
static files.
+--------------------+
| |
+------> | gitlab-workhorse |
| | |
| +---------+----------+
| |
| v
|
+------+---+ +-------------+
| | | |
| NGINX +----> | Unicorn |
| | | |
+------+---+ +-------------+
|
|
| +------------+
| | |
+--------> | static |
| files |
| |
+------------+
Kamil half-jokingly suggested at one point that we could route all HTTP
traffic to GitLab through gitlab-workhorse. Over time I started to
believe this was a good idea: it would radically simplify the NGINX
configuration for GitLab, and consequently make it easier to deploy
GitLab behind other web servers (like Apache) which some people prefer
strongly to using NGINX. It seems the idea grew on Kamil too because we
soon saw a huge merge request from him to gitlab-workhorse which turned
it into a 'smart proxy' that serves static files, injects error pages,
implements Git-over-HTTP plus other features, and proxies traffic to
GitLab.
+-------------+ +---------------------+ +------------+
| | | | | |
| NGINX +-------> | gitlab-workhorse +------> | Unicorn |
| | | | | |
+-------------+ +---------------------+ +------------+
This change went out in GitLab 8.3. In little over four months
gitlab-workhorse went from being a little helper daemon on the side to a
traffic cop that routes all HTTP requests going into GitLab.
This work immediately paid off in GitLab 8.4 when
Grzegorz added the CI artifact browsing
feature and GitLab 8.5 where we started serving 'raw' Git blobs via
gitlab-workhorse. Neither of these changes forced GitLab administrators
to update their NGINX configuration.
What to do next
The most important next step for gitlab-workhorse is to make it less
dependent on me by getting more of my team members to contribute to the
code. I am trying to do this by purposely leaving some nice
changes for
others to implement.
Because of the gradual (and sneaky :) ) way gitlab-workhorse was added
to GitLab we still have some technical debt in GitLab in the Git HTTP
authentication / authorization
code. It would be
nice to clean this up.
Finally, it is sub-optimal that we still buffer Git pushes in NGINX
before forwarding them to gitlab-workhorse. We could avoid this
unncessary delay and give people who use Apache instead of NGINX a
better experience if we implement selective request buffering in
gitlab-workhorse.
If you want to know what is coming next in GitLab, check out our
Direction page, or follow developments on
our latest milestones for 8.7 CE
and EE.
