Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Blueprint :: [Elasticsearch integration for one namespace]

Idea/Problem Statement

Problem:

Why:

Scope:

Summary of past efforts

see elastic at gitlab blueprint

Outline of options

At the moment of writing we do not have self-managed ES clusters, we only use Elastic Cloud. We use it for logs on gitlab.com and indexing on ops.gitlab.net. We could use it for indexing staging.gitlab.com and gitlab.com in the exact same way as it is used for ops.gitlab.net.

Additional considerations

Dependencies

fixes to a number of bugs, e.g. search using elastic integration even when disabled

design doc for this blueprint

elastic at gitlab blueprint

Costs

Total

total: ~1100USD/month

Production

prod spec:

cost estimate

Production monitoring

prod monitoring spec:

cost estimate

Staging

stg spec:

cost estimate:

Staging monitoring

stg monitoring spec:

cost estimate

Security

Are we going to store sensitive customer data? yes, see this dicussion for the security review

Do we trust our SaaS provider to secure the OS and the service? yes, we already trust them with our application logs.

Recommendations

Considering the time scale, for a single namespace we should continue using Elastic Cloud.

We should start evaluating our options for having ES clusters that will be able to handle hundreds of TB of data (which is not within the scope of this KR).