Blog Update: Elasticsearch lessons learnt for Advanced Global Search 2020-04-28
April 28, 2020
8 min read

Update: Elasticsearch lessons learnt for Advanced Global Search 2020-04-28

This extends on previous blog posts that showcase the work done and lessons learnt while building our Advanced Global Search features.


For some time now, GitLab has been working on enabling the Elasticsearch
integration on to allow as many users as possible access
to the Advanced Global


This article follows up with yet more lessons learned on our road to Enabling
Elasticsearch for You can read the first article from
the second article from
to see where we've come

Selective indexing is the largest known installation of GitLab and as such has a lot of
projects, code, issues, merge requests and other things that need to be indexed
in Elasticsearch in order for us to support Advanced Global Search. Given the
volume of data and given that we didn't have much experience running
Elasticsearch at any scale in the company, it made sense for us to think of a
way to gradually index our data and learn new lessons at each scaling
milestone. In order to do this we built in the ability to enable indexing and
searching of individual groups and projects so we could stagger out the

Back in June 24, 2019, we enabled the integration for the gitlab-org group on, and after doing this, we learnt a lot before we were able
to expand to other groups.

Today, we have enabled Advanced Global Search for over 900 groups already, and
we are still increasing this rollout. In order to get to this large number, we
needed to automate the gradual increase in groups by allowing operators to
roll out to percentages rather than just one group at a
. This has been used
now to get us to 15% of bronze tier customers and will hopefully be useful for
getting us all the way to 100% of all paying customers.

This post will detail a few key lessons learnt since enabling our first group.
At the very least it will serve as a reminder of the things we wished we knew
before we ran into them.

Defense in depth

One of the first and least expected problems we ran into after enabling this
feature was a continuous stream of security vulnerabilities caused by the
complexity of replicating our authorization model in Elasticsearch. We really
thank our HackerOne community for how quickly they were able to notice the
mistakes we made in our authorization logic and appreciate the impact they've
made on ensuring people's data is secure.

Basically, our Elasticsearch integration needs to cover all permutations of user
permissions, project visibilities, group inheritance, confidential resources,
and other features that GitLab supports for determining whether or not a user
can view a specific document. All documents live in a single index, so the only
way to ensure that searches return the correct set of results a user can see is
to include the permissions checks in the search query itself. However, this breaks
one of the first programming rules we've learnt, which is D.R.Y. (Do not Repeat
Yourself). And that rule in this case manifested in the expected outcome of
bugs caused by our permission logic being wrong in our Elasticsearch queries.

After fixing bug after bug which fell into this same formula of incorrect
permission logic, we soon realized we just needed to apply a 2nd layer of
permission filtering of search
using the same
code we use to check permissions in GitLab. Once we implemented this, the
vulnerabilities mostly disappeared despite a few lower severity and harder
to accomplish attacks. This new logic we now call "redacting" search results
also allowed for an additional threat/bug detection mechanism in GitLab by
setting up alerts when the redacting logic was

Sorted sets

GitLab initially followed the common path for indexing our database models by
simply queueing a background job on every created/updated/deleted record in the
database that also needed to be in Elasticsearch. This is very easy to
implement in Rails using Sidekiq, but it
comes with the downside that you end up sending many small updates to
Elasticsearch. These very frequent writes to ElasticSearch cause performance
issues at

After reading Keeping Elasticsearch in
decided to implement a buffered queue
based on Redis
sorted sets
. This approach allowed us to
easily batch updates using the Elasticsearch bulk API every minute, and it also
had the advantage of automatically de-duplicating updates due to the sorted set
data structure.

This has now been running for a few months without any major issues. The
de-duplication turns out to be very powerful given our specific workloads, as
over a 24hr period of time, 95% of updates are duplicates. These details will
obviously vary based on the specifics of your application, but in GitLab these
duplicates happen very close together as people edit issue descriptions, MR
descriptions and other objects in quick succession, and so de-duplication will
allow us to reach a considerably larger scale by reducing load on Elasticsearch

Some index settings can cause certain queries to break

A very important lesson we
is that it's very
easy to break your Elasticsearch queries if you change the
and don't test all the types of queries you are sending. In particular, the
documentation does mention that "Positions can be used for proximity or phrase
queries", but it does not clearly mention that if you try to do those queries
(for example, in the form of simply query
without the positions index options, then your queries will fail.

Of course the other important lesson included in our
was to
provide an adequate integration test suite of the different types of queries we
allow our users to perform to avoid these regressions.

Just because Elasticsearch allows you to update index mappings doesn't mean it will work

Elasticsearch does make it clear that updating existing mappings in an index
is usually not an
but there seem to be some cases where you are able to change the mapping without
triggering any error at the time of changing the mapping, but later no documents
will be able to write to the index. We learnt

when trying to update
The lesson was to properly test that you can change the setting and keep writing
to the index after.

Remote reindex has some limitations that make it difficult to do quickly

A key part of running Elasticsearch is in production is coming up with a
strategy for schema changes or other data migrations. We have experience doing
these kinds of changes with Postgres, and most web application developers will
have similar experience with migrating relational databases. But Elasticsearch
doesn't allow for many in-place migrations of schema or data, and as such, it
generally requires users to
all of their data in order to make changes to the schema or data.

In the case of GitLab, we've been making considerable improvements to the
storage used by our Elasticsearch
which is a
key step in rolling out the feature on But changes to the index
options which reduce storage have meant we needed to reindex the entire
dataset to see the benefits.

Initially, it seemed like the best way to do this would be to use the reindex

feature in Elasticsearch. This seemed appealing as it would mean that the high
volume of writes which happen during the reindex would not put any load on our
primary cluster during the reindex process. This benefit was valid, but it
turned out that we couldn't get the reindexing process to go quick

due to a limitation in this API where it won't allow you to configure the
buffer size

Aliases are really a good idea to do from the start

When reading about reindexing in Elasticsearch, you will likely learn that
using an index

is almost always at the heart of these solutions. Without this, it's very
difficult to perform zero downtime reindexing, because Elasticsearch provides no
way to rename an index.

As such, we learnt that we probably should have been using aliases from the
beginning and plan to implement this functionality

There are several settings to speed up reindexing

After realizing that reindex to remote was not going to work well for us, we did
learn about a bunch of ways to make the in-cluster reindexing happen a lot
faster. Most of these come from Elastic's excellent guide to tune for indexing

The changes we used when
speed things up were:

  1. Set number_of_replicas to 0 on the destination index. The default is 1
    which means 1 replica which means all the data written to the index needs to be
    written to 2 nodes. Disabling means basically half the amount of writing
  2. Set refresh_interval to -1 on the destination index. The default
    behaviour is to refresh the indices every second. Because we aren't reading from
    this new indexing during the reindexing, we can then disable it as we don't mind
    that searches won't return results.
  3. Set
    to async on the destination index. This will allow the cluster to sync the
    updates to disk asynchronously. It does come with some risk of data loss if
    a hard disk fails during reindexing, but does come with the advantage of
    speeding up the process, and when we finish, we can switch this back to the

You can make the replica recovery go faster by changing settings too

As mentioned above, we decided to disable replication during the reindexing, but
after it finishes, we need to re-enable replication and allow replicas to
catch up before switching to the new index. This process may take a while but
you can speed it up by increasing

Use routing to speed up queries but note the limit on number of routing ids in the request

GitLab routes all documents based on the project. This allowed us the
opportunity to route searches to the correct nodes when searching within
. This
did speed up our searches by around

We did however learn when making this optimization to be careful to not send
too many routing ids in the request, or you may exceed the 4096 bytes limit on
Elasticsearch HTTP

We want to hear from you

Enjoyed reading this blog post or have questions or feedback? Share your thoughts by creating a new topic in the GitLab community forum. Share your feedback

Ready to get started?

See what your team could do with a unified DevSecOps Platform.

Get free trial

New to GitLab and not sure where to start?

Get started guide

Learn about what GitLab can do for your team

Talk to an expert