Update: Elasticsearch lessons learnt for Advanced Global Search 2020-04-28

For some time now, GitLab has been working on enabling the Elasticsearch integration on GitLab.com to allow as many GitLab.com users as possible access to the Advanced Global Search features.

This article follows up with yet more lessons learned on our road to Enabling Elasticsearch for GitLab.com. You can read the first article from Mario and the second article from Nick to see where we've come from.

Selective indexing

GitLab.com is the largest known installation of GitLab and as such has a lot of projects, code, issues, merge requests and other things that need to be indexed in Elasticsearch in order for us to support Advanced Global Search. Given the volume of data and given that we didn't have much experience running Elasticsearch at any scale in the company, it made sense for us to think of a way to gradually index our data and learn new lessons at each scaling milestone. In order to do this we built in the ability to enable indexing and searching of individual groups and projects so we could stagger out the release.

Back in June 24, 2019, we enabled the integration for the gitlab-org group on GitLab.com, and after doing this, we learnt a lot before we were able to expand to other groups.

Today, we have enabled Advanced Global Search for over 900 groups already, and we are still increasing this rollout. In order to get to this large number, we needed to automate the gradual increase in groups by allowing operators to roll out to percentages rather than just one group at a time. This has been used now to get us to 15% of bronze tier customers and will hopefully be useful for getting us all the way to 100% of all paying customers.

This post will detail a few key lessons learnt since enabling our first group. At the very least it will serve as a reminder of the things we wished we knew before we ran into them.

Defense in depth

One of the first and least expected problems we ran into after enabling this feature was a continuous stream of security vulnerabilities caused by the complexity of replicating our authorization model in Elasticsearch. We really thank our HackerOne community for how quickly they were able to notice the mistakes we made in our authorization logic and appreciate the impact they've made on ensuring people's data is secure.

Basically, our Elasticsearch integration needs to cover all permutations of user permissions, project visibilities, group inheritance, confidential resources, and other features that GitLab supports for determining whether or not a user can view a specific document. All documents live in a single index, so the only way to ensure that searches return the correct set of results a user can see is to include the permissions checks in the search query itself. However, this breaks one of the first programming rules we've learnt, which is D.R.Y. (Do not Repeat Yourself). And that rule in this case manifested in the expected outcome of bugs caused by our permission logic being wrong in our Elasticsearch queries.

After fixing bug after bug which fell into this same formula of incorrect permission logic, we soon realized we just needed to apply a 2nd layer of permission filtering of search results using the same code we use to check permissions in GitLab. Once we implemented this, the vulnerabilities mostly disappeared despite a few lower severity and harder to accomplish attacks. This new logic we now call "redacting" search results also allowed for an additional threat/bug detection mechanism in GitLab by setting up alerts when the redacting logic was triggered.

Sorted sets

GitLab initially followed the common path for indexing our database models by simply queueing a background job on every created/updated/deleted record in the database that also needed to be in Elasticsearch. This is very easy to implement in Rails using Sidekiq, but it comes with the downside that you end up sending many small updates to Elasticsearch. These very frequent writes to ElasticSearch cause performance issues at scale.

After reading Keeping Elasticsearch in Sync we decided to implement a buffered queue approach based on Redis sorted sets. This approach allowed us to easily batch updates using the Elasticsearch bulk API every minute, and it also had the advantage of automatically de-duplicating updates due to the sorted set data structure.

This has now been running for a few months without any major issues. The de-duplication turns out to be very powerful given our specific workloads, as over a 24hr period of time, 95% of updates are duplicates. These details will obviously vary based on the specifics of your application, but in GitLab these duplicates happen very close together as people edit issue descriptions, MR descriptions and other objects in quick succession, and so de-duplication will allow us to reach a considerably larger scale by reducing load on Elasticsearch writes.

Some index settings can cause certain queries to break

A very important lesson we learnt is that it's very easy to break your Elasticsearch queries if you change the index_options and don't test all the types of queries you are sending. In particular, the documentation does mention that "Positions can be used for proximity or phrase queries", but it does not clearly mention that if you try to do those queries (for example, in the form of simply query string) without the positions index options, then your queries will fail.

Of course the other important lesson included in our fix was to provide an adequate integration test suite of the different types of queries we allow our users to perform to avoid these regressions.

Just because Elasticsearch allows you to update index mappings doesn't mean it will work

Elasticsearch does make it clear that updating existing mappings in an index is usually not an option, but there seem to be some cases where you are able to change the mapping without triggering any error at the time of changing the mapping, but later no documents will be able to write to the index. We learnt this when trying to update index_options. The lesson was to properly test that you can change the setting and keep writing to the index after.

Remote reindex has some limitations that make it difficult to do quickly

A key part of running Elasticsearch is in production is coming up with a strategy for schema changes or other data migrations. We have experience doing these kinds of changes with Postgres, and most web application developers will have similar experience with migrating relational databases. But Elasticsearch doesn't allow for many in-place migrations of schema or data, and as such, it generally requires users to reindex all of their data in order to make changes to the schema or data.

In the case of GitLab, we've been making considerable improvements to the storage used by our Elasticsearch index which is a key step in rolling out the feature on GitLab.com. But changes to the index options which reduce storage have meant we needed to reindex the entire dataset to see the benefits.

Initially, it seemed like the best way to do this would be to use the reindex from remote feature in Elasticsearch. This seemed appealing as it would mean that the high volume of writes which happen during the reindex would not put any load on our primary cluster during the reindex process. This benefit was valid, but it turned out that we couldn't get the reindexing process to go quick enough due to a limitation in this API where it won't allow you to configure the buffer size.

Aliases are really a good idea to do from the start

When reading about reindexing in Elasticsearch, you will likely learn that using an index alias is almost always at the heart of these solutions. Without this, it's very difficult to perform zero downtime reindexing, because Elasticsearch provides no way to rename an index.

As such, we learnt that we probably should have been using aliases from the beginning and plan to implement this functionality soon.

There are several settings to speed up reindexing

After realizing that reindex to remote was not going to work well for us, we did learn about a bunch of ways to make the in-cluster reindexing happen a lot faster. Most of these come from Elastic's excellent guide to tune for indexing speed.

The changes we used when reindexing to speed things up were:

1. Set number_of_replicas to 0 on the destination index. The default is 1 which means 1 replica which means all the data written to the index needs to be written to 2 nodes. Disabling means basically half the amount of writing needed.
2. Set refresh_interval to -1 on the destination index. The default behaviour is to refresh the indices every second. Because we aren't reading from this new indexing during the reindexing, we can then disable it as we don't mind that searches won't return results.
3. Set translog.durability to async on the destination index. This will allow the cluster to sync the updates to disk asynchronously. It does come with some risk of data loss if a hard disk fails during reindexing, but does come with the advantage of speeding up the process, and when we finish, we can switch this back to the default.

You can make the replica recovery go faster by changing settings too

As mentioned above, we decided to disable replication during the reindexing, but after it finishes, we need to re-enable replication and allow replicas to catch up before switching to the new index. This process may take a while but you can speed it up by increasing indices.recovery.max_bytes_per_sec.

Use routing to speed up queries but note the limit on number of routing ids in the request

GitLab routes all documents based on the project. This allowed us the opportunity to route searches to the correct nodes when searching within projects. This did speed up our searches by around 5x.

We did however learn when making this optimization to be careful to not send too many routing ids in the request, or you may exceed the 4096 bytes limit on Elasticsearch HTTP lines.