Scaling down: How we shrank image transfers by 93%

The Memory team recently shipped an improvement to our image delivery functions
that drastically reduces the amount of data we serve to clients. Learn here how we went from knowing nothing about
Golang and image scaling to a working on-the-fly image scaling solution built into
Workhorse.

Introduction

Images are an integral part of GitLab. Whether it is user and project avatars, or images embedded in issues
and comments, you will rarely load a GitLab page that does not include images in some way shape or form.
What you may not be aware of is that despite most of these images appearing fairly small when presented
on the site, until recently we were always serving them in their original size.
This meant that if you would visit a merge request, then all user avatars that appeared merely as thumbnails
in sidebars or comments would be delivered by the GitLab application in the same size they were uploaded in,
leaving it to the browser rendering engine to scale them down as necessary. This meant serving
megabytes of image data in a single page load, just so the frontend would throw most of it away!

While this approach was simple and served us well for a while, it had several major drawbacks:

• Perceived latency suffers. The perceived latency is the time that passes between a user
  requesting content, and that content actually becoming visible or being ready to engage with.
  If the browser has to download several megabytes of image data, and then has to furthermore
  scale down those images to fit the cells they are rendered into, the user experience unnecessarily suffers.
• Egress traffic cost. On gitlab.com, we store all images in object storage, specifically GCS
  (Google Cloud Storage). This means that our Rails app first needs to resolve an image entity to
  a GCS bucket URL where the binary data resides, and have the client
  download the image through that endpoint. This means that for every image served, we cause
  traffic from GCS to the user that we have to pay for, and the more data we serve, the higher the cost.

We therefore took on the challenge to both improve rendering performance and reduce traffic costs
by implementing an image scaler that would downscale images
to a requested size before delivering them to the client.

Phase 1: Understanding the problem

The first problem is always: understand the problem! What is the status quo exactly? How does it work?
What is broken about it? What should we focus on?

We had a pretty good idea of the severity of the problem, since we regularly run performance tests
through sitespeed.io that highlight performance problems on our site.
It had identified images sizes as one of the most severe issues:

To better inform a possible solution, an essential step was to collect enough data
to help identify the areas we should focus on. Here are some of the highlights:

• Most images requested are avatars. We looked at the distribution of requests for certain types of images.
  We found that about 70% of them were for avatars, while the remaining 30% accounted for embedded images.
  This suggested that any solution would have the biggest reach if we focused on avatars first. Within the
  avatar cohort we found that about 62% are user avatars, 22% are project avatars, and 16% are group avatars,
  which isn't surprising.
• Most avatars requested are PNGs or JPEGs. We also looked at the distribution of image formats. This is partially
  affected by our upload pipeline and how images are processed (for instance, we always crop user avatars and store them as PNGs)
  but we were still surprised to see that both formats made up 99% of our avatars (PNGs 76%, JPEGs 23%). Not much
  love for GIFs here!
• We serve 6GB of avatars in a typical hour. Looking at a representative window of 1 hour of GitLab traffic, we saw
  almost 6GB of data move over the wire, or 144GB a day. Based on experiments with downscaling a representative user avatar,
  we estimated that we could reduce this to a mere 13GB a day on average, saving 130GB of bandwidth each day!

This was proof enough for us that there were significant gains to be made here. Our first intuition was: could this
be done by a CDN? Some modern CDNs like Cloudflare already support image resizing
in some of their plans. However, we had two major concerns about this:

1. Supporting our self-managed customers. While gitlab.com is the largest GitLab deployment we know of, we have hundreds of thousands
   of customers who run their own GitLab installation. If we were to only resize images that pass through a CDN in front of gitlab.com,
   none of those customers would benefit from it.
2. Pricing woes. While there are request budgets based on your CDN plan, we were worried about the operational cost this would
   add for us and how to reliably predict it.

We therefore decided to look for a solution that would work for all GitLab users, and that would be more under
our own control, which led us to phase 2: experimentation!

Phase 2: Experiments, experiments, experiments!

A frequent challenge for our team (Memory)
is that we need to venture into parts of GitLab's code base
that we are unfamiliar with, be it with the technology, the product area, or both. This was true in this
case as well. While some of us had some exposure to image scaling services, none of us had ever built or
integrated one.

Our main goal in phase 2 was therefore to identify what the possible approaches to image scaling were,
explore them by researching existing solutions or even building proof-of-concepts (POCs), and grade
them based on our findings. The questions we asked ourselves along the way were:

• When should we scale? Upfront during upload or on-the-fly when an image is requested?
• Who does the work? Will it be a dedicated service? Can it happen asynchronously in Sidekiq?
• How complex is it? Whether it's an existing service we integrate, or something we build ourselves,
  does implementation or integration complexity justify its relatively simple function?
• How fast is it? We shouldn't forget that we set out to solve a performance issue. Are we sure that
  we are not making the server slower by the same amount of time we save in the client?

With this in mind, we identifed multiple architectural approaches to consider,
each with their own pros and cons. These issues also doubled as a form of architectural decision log
so that decisions for or against an approach are recorded.

The major approaches we considered are outlined next.

Static vs. dynamic scaling

There are two basic ways in which an image scaler can operate: it can either create thumbnails of
an existing image ahead of time, e.g. during the original upload as a background job. Or it can
perform that work on demand, every time an image is requested. To make a long story short: while
it took a lot of back and forth, and even though we had a working POC,
we eventually discarded the idea of scaling statically, at
least for avatars. Even though CarrierWave (the Ruby uploader
we employ) has an integration
with MiniMagick and is able to perform that kind of work, it suffered from several issues:

1. Maintenance heavy. Since image sizes may change over time, a strategy is needed to backfill sizes
   that haven't been computed yet. This raised questions especially for self-managed customers where
   we do not control the GitLab installation.
2. Statefulness. Since thumbnails are created alongside the original image, it was unclear how to perform
   cleanups should they become necessary, since CarrierWave does not store these as separate database
   entities that we could easily query.
3. Complexity. The POC we created turned out to be more complex than anticipated and felt like we
   were shoehorning this feature onto existing code. This was exacerbated by the fact that at the time
   we were running a very old version of CarrierWave that was already a maintenance liability, and upgrading it
   would have added scope creep and delays to an already complex issue.
4. Flexibility. The actual scaler implementation in CarrierWave is buried three layers down the Ruby dependency stack,
   and it was difficult to replace the actual scaler binary (which would become a
   problem when trying to secure this solution as we will see in a moment.)

For these reasons we decided to scale images on-the-fly instead.

Dynamic scaling: Workhorse vs. dedicated proxy

When scaling images on-the-fly the question becomes: where? Early on there was a suggestion to use
imgproxy, a "fast and secure standalone server for resizing and converting remote images".
This sounded tempting, since it is a "batteries included" offering, it's free to use, and it is a great
way to isolate the task of image scaling from other production work loads, which has benefits around
security and fault isolation.

The main problem with imgproxy was exactly that, however: a standalone server.
Introducing a new service to GitLab
is a complex task, since we strive to appear as a single application to the end user,
and documenting, packaging, configuring, running and monitoring a new service just for rescaling images seemed excessive.
It therefore wasn't in line with our prerogative of focusing on the
minimum viable change.
Moreover, imgproxy had significant overlap with existing architectural components at GitLab, since we already
run a reverse proxy: Workhorse.

We therefore decided that the fastest way to deliver an MVC was to build out this functionality in Workhorse
itself. Fortunately we found that we already had an established pattern for dealing with special, performance
sensitive workloads, which meant that we could
learn from existing solutions for similar problems (such as image delivery from remote storage), and we could
lean on its existing integration with the Rails application for request authentication and running business
logic such as validating user inputs, which helped us tremendously to focus on the actual problem: scaling images.

There was a final decision to make, however: scaling images is a very different kind of workload from
serving ordinary requests, so an open question was how to integrate a scaler into Workhorse in a way
that would not have knock-on effects on other tasks Workhorse processes need to execute.
The two competing approaches discussed were to either shell out to an executable that performs the scaling,
or run a sidecar process
that would take over image scaling work loads from the main Workhorse process.

Dynamic scaling: Sidecar vs. fork-on-request

The main benefit of a sidecar process is that it has its own life-cycle and memory space, so it can be tuned
separately from the main serving process, which improves fault isolation. Moreover, you only pay the
cost for starting the process once. However, it also comes with
additional overhead: if the sidecar dies, something has to restart it, so we would have to look at
process supervisors such as runit to do this for us, which again comes with a significant amount
of configuration overhead. Since at this point we weren't even sure how costly it would be to serve
image scaling requests, we let our MVC principle guide us and decided to first explore the simpler
fork-on-request approach, which meant shelling out to a dedicated scaler binary on each image scaling
request, and only consider a sidecar as a possible future iteration.

Forking on request was explored as a POC
first, and was quickly made production ready and deployed
behind a feature toggle. We initially ended up settling on GraphicsMagick
and its gm binary to perform the actual image scaling for us, both because it is a battle tested library, but also
because there was precedent at GitLab to use it for existing features, which allowed us to ship
a solution even faster.

The overall request flow finally looked as follows:

sequenceDiagram
    Client->>+Workhorse: GET /image?width=64
    Workhorse->>+Rails: forward request
    Rails->>+Rails: validate request
    Rails->>+Rails: resolve image location
    Rails-->>-Workhorse: Gitlab-Workhorse-Send-Data: send-scaled-image
    Workhorse->>+Workhorse: invoke image scaler
    Workhorse-->>-Client: 200 OK

The "secret sauce" here is the Gitlab-Workhorse-Send-Data header synthesized by Rails. It carries
all necessary parameters for Workhorse to act on the image, so that we can maintain a clean separation
between application logic (Rails) and serving logic (Workhorse).
We were fairly happy with this solution in terms of simplicity and ease of maintenance, but we
still had to verify whether it met our expectations for performance and security.

Phase 3: Measuring and securing the solution

During the entire development cycle, we frequently measured the performance of the various appraoches
we tested, so as to understand how they would affect request latency and memory use.
For latency tests we relied on Apache Bench, since
recalling our initial mission, we were mostly interested in reducing the request latency a user might experience.

We also ran benchmarks encoded as Golang tests that specifically compared different scaler implementations
and how performance changed with different image formats and image sizes. We learned a lot from these
tests, especially where we would typically lose the most time, which often was in encoding/decoding
an image, and not in resizing an image per se.

We also took security very seriously from the start. Some image formats such as SVGs are notorious
for remote code execution attacks, but there were other concerns such as DOS-ing the service with
too many scaler requests or PNG compression bombs. We therefore
put very strict requirements in place around what sizes (both dimensionally but also in bytes) and
formats we will accept.

Unfortunately one fairly severe issue remained that turned out to be a deal breaker with our simple
solution: gm is a complex piece of software, and shelling out to a 3rd party binary written in C still
leaves the door open for a number of security issues. The decision was to sandbox the binary
instead, but this turned out
to be a lot more difficult than anticipated. We evaluated but discarded multiple approaches to sandboxing
such as via setuid, chroot and nsjail, as well as building a custom binary on top of seccomp.
However, due to performance, complexity or other concerns we discarded all of them in the end.
We eventually decided to sarifice some performance for the sake of protecting our users as best we can and
wrote a scaler binary in Golang, based on an existing imaging
library, which had none of these issues.

Results, conclusion and outlook

In roughly two months we took an innocent sounding but in fact complex topic, image scaling, and went
from "we know nothing about this" to a fully functional solution that is now running on gitlab.com.
We faced many headwinds along the way, in part because we were unfamiliar with both the topic and
the technology behind Workhorse (Golang), but also because we underestimated the challenges of delivering
an image scaler that will be both fast and secure, an often difficult trade-off. A major lesson learned
for us is that security cannot be an afterthought; it has to be part of the design from day one and
must be part of informing the approach taken.

So was it a success? Yes! While the feature didn't have as much of an impact on overall perceived client
latency as we had hoped, we still dramatically improved a number of metrics. First and foremost, the
dreaded "properly size image" reminder that topped our sitepeed metrics reports is resolved. This is also evident
in the average image size processed by clients, which for image heavy pages fell off a cliff (that's good -- lower is
better here):

Site-wide we saw a staggering 93% reduction in image transfer size of page content delivered to clients.
These gains also translate into savings for GCS egress traffic, and hence Dollar cost savings, by an equivalent amount.

A feature is never done of course, and there are a number of things we are looking to improve in the future:

• Improving metrics and observability
• Improving performance through more aggressive caching
• Adding support for WebP and other features such as image blurring
• Supporting content images embedded into GitLab issues and comments

The Memory team meanwhile will slowly step back from this work, however, and hand it over to product teams
as product requirements evolve.