Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Category Direction - Logging


UPDATE: As of 2020-09-01: The Logging category at GitLab has presently been reprioritized and we are not actively progressing this category. The vision you see displayed on this page represents the direction we will pursue if this becomes a priority again in the future.

Introduction and how you can help

Thanks for visiting this category strategy page on Logging in GitLab. This category belongs to and is maintained by the Health group of the Monitor stage.

This page is maintained by Kevin Chu, group product manager. You can connect with him via Zoom or Email. If you're a GitLab user and have direct knowledge of your Logging usage, we'd especially love to hear your use case(s)

The Challenge

A fundamental requirement for running applications is to have a centralized location to manage and review the logs. While manually reviewing logs could work with just a single node app server, once the deployment scales beyond one you need solution which can aggregate and centralize them for review. In the distributed nature of cloud-native applications, it is crucial and critical to collect logs across multiple services and infrastructure, present them in an aggregated view, so users could quickly search through a list of logs that originate from multiple pods and containers.

Target Audience and Experience

Being able to capture and review logs are an important tool for all users across the DevOps spectrum. From pure developers who may need to troubleshoot their application when it is running in a staging or review environment, as well as pure operators who are responsible for keeping production services online.

The target workflow includes a few important use cases:

  1. Aggregating logs from multiple pods and containers from all namespaces
  2. Filtering by host, container, service, timespan, regex, and other criteria. These filtering options should align with the filter options and tags/labels of our other monitoring tools, like metrics.
  3. Log alerts should also be able to be created, triggering alerts under specific user defined scenarios.

Log aggregation use case in GitLab

Before the 12.8 release, existing Monitor stage users already could view pod logs directly from within the GitLab UI. However, this was done only through the available Kubernetes APIs. Viewing logs with the Kubernetes APIs is limited to allowing a log-tailing experience on a specific pod from multiple environments only.

With the 12.8 release, any user can deploy Elastic stack - a specific flavor of Elasticsearch to your Kubernetes cluster with the push of a button (similar to the way we deploy Prometheus). Once deployed, it automatically starts collecting all logs that are coming from the cluster and applications across the available environments (production, staging, testing, etc.), and they will surface in the Log explorer within the GitLab UI. With it the user can select whether they'd like to view logs from all pods vs individual pods, conduct a full-text search or go back in time. In addition, users can also navigate directly from the metric chart to the log explorer while preserving the context.


What's Next & Why

What is not planned right now

GitLab's log aggregation solution works only when an environment is defined on a managed Kubernetes cluster (where Elasticsearch and filebeat deployed into), working on external Elasticsearch instance which surface logs in GitLab UI is out of scope for now.

Maturity Plan

Competitive Landscape

Splunk and Elastic are the top two competitors in this space.

Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license