The following page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features or functionality remain at the sole discretion of GitLab Inc.
This is the product direction for Monitor:Observability. If you'd like to discuss this direction directly with the product managers for Monitor:Observability, feel free to reach out to Sebastien Pahl (Principal PM of Monitor:Observability) (GitLab, Email).
Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. In the (recent) past, simply monitoring your application and related systems may have been sufficient, e.g. watching for specific metrics to track whether performance and health are within expectations. Today, with the rapid adoption of cloud-native technologies and distributed systems, it is no longer sufficient to only look for known indicators. An observable system is one that enables operators to efficiently drill in and understand what went wrong even for previously unencountered problems.
GitLab's Observability solution is powered by two key platform capabilities, GitLab Observability Platform and the GitLab Observability UI. Please visit their respective direction pages for more details.
Make data useful for every stage of the DevOps process.
GitLab users and customers use GitLab Observability to improve the outcome of their software investment. They process, store, manage, provide access to, query, visualize, collaborate on, and operationalize on data collected throughout the entire DevOps process.
The practice of observability is typically done for only production applications. Our vision is broader. How teams build software, how the software operates, and ultimately how users interface and use our software, all generate meaningful data that can help software teams to better infer the states of a system and thus make meaningful improvements. As such, we are aiming to build an observability capability that not only helps enable teams to ensure the health and performance of production systems, but also enable teams to continuously improve outcome in each stage of the DevOps process.
On Dec 14, 2021, GitLab announced the acquisition of Opstrace. Opstrace is an observability distribution that will become integrated and usable out-of-the-box for all GitLab users. Using Opstrace, users gain the benefit of having a full observability platform, starting with Metrics, powered by Prometheus.
On March 29, 2021 we pubished a blog post outlining our vision for a new UI to be fully integrated with the devops platform.
Over the coming months, we will focus on integrating Opstrace into GitLab. Observability will be available, by default, for both GitLab SaaS and Self-Managed users, starting at the free tier. Using GitLab’s observability capability, you will be using a completely open-sourced platform. You do not have to worry about vendor lock-in from instrumentation to alerting.
Organizations often have the choice of using an observability vendor or building their own observability platform. With the former, teams outsource the problems of operating a system that needs to be scaled and consistently updated but are locked-in to proprietary software and APIs. With the latter, teams have to manage the complexity of an observability platform, figure out how to make all the open-source components work together and keep them running up to date. We want that choice to be easy to make.
We are different because we will focus on making the toil of operating open-source observability tools easy. Instead of staffing a team to keep toolings up to date, you can easily and confidently upgrade with the observability distribution. Furthermore, this observability tool is integrated with the rest of GitLab so you have a single tool to build, test, collaborate, deploy, and monitor your applications.
Over time, we’ll add additional delightful experiences such as:
We'll pursue our vision based on the following principles:
We want developers to have immediate access to a robust observability platform as a first principle. That means we will ensure it is available by default, starts with convention over configuration, and is easy to utilize.
We believe that observability data is most valuable when it is easy to glean insights across data types of metrics, logs, traces and errors. As a result we're working to store all types of observability data in a unified datastore - Clickhouse.
A common interface, and most importantly a common collaboration space is key to bringing observability to where developers work. We'll create an integrated UI that allows for collaboration on observability data directly within GitLab.
June 2022 Ops Engineering Showcase (AMER/EMEA)
The Monitor:Observability will work to help all GitLab customers Monitor their apps by providing a simple on-by-default observability stack. We'll focus first on more users, and more contributors' adoption in our dual flywheel strategy. We'll know if we are trending in the right direction by looking at the Monitor:Observability Monthly Active Users adoption, which will grow if we are providing value to our users.
While SaaS is the first goal to be able to have a good testing ground and enable quick on-boarding, we want to quickly focus on self-managed enterpise users.
The Observability group will be focused on bringing observability, including Metrics, Tracing, Error tracking and Logging to market within the GitLab platform.
Now that Opstrace is part of GitLab, the first thing we’ll do is integrate Opstrace into GitLab. For additional details on the integration progress, follow this GitLab epic.
We are currently focused on integrating the first pieces with the Gitlab platform.
We are coming to an end with Milestone 3 and are targeting to have error tracking and tracing in open beta for 15.4. This will allow using a sentry client (errors) or opentelemetry/jaeger agent (traces) to send data to a gitlab.com instance for a specific project and group (provided features flag is enabled at first). Data will be queryable in our observability UI based on a grafana fork.
As we move on from our deliverables with milestone 3, we will start designing out vision milestones linked to this issue. First priorities will be designing and implementing metrics backed by clickhouse (follow here).
Planning happens in the planning board where issues are arranged by release. Over time this will become more populated with a further look into the upcoming releases.
There are located in this project.
Our goal is to make Open-source Observability as easy as SaaS offerings. Having a fully usable observability Saas that one can also deploy in a dedicated manner.
Build an alternative Open Source UI for observability with different paradigms and goals than the current single offering (grafana). More here.
Open Source observability has many popular APIs such as Prometheus, Opentelemetry, … But metrics, logs, traces and other observability events all go into separate databases. Treating them as separate systems prevents building new higher level applications. We are working on getting all the data in a single backend based on clickhouse to enable richer user experience in the future.
While it’s true that we built a lot of kubernetes features, all that is needed is compatibility with APIs such as prometheus, not running kubernetes oneself.
We do use kubernetes to deploy opstrace itself but that is not something the user has to manage and it’s separate from their infrastructure.
As mentioned above, all we require is a cloud account. We deploy and configure everything with our terraform installer.
Kubernetes is used to run the system, but that is transparent to the user, operators are deployed on the kubernetes cluster to maintain the rest of the stack.
Terraform will be used to rollout new upgrades.
For omnibus users, since this kind of stack is bigger than what a single machine/VM can provide, this will also be available through the same manner. They would deploy it into one of their cloud accounts.
At first, there will be a separate UI. This is because we are using Grafana as a base and we need to use its technologies (react…) to build our UI. To bring data into the GitLab UI we will use iframes. The simplest thing to start with first would be to Iframe our entire UI (this keeps the feeling for the user of not leaving GitLab) and later allow framing and more directly embedding single dashboards & more.
There are two types of datastores.
Clickhouse for storing the observability data.
Postgres is used for our settings and users. This will disappear over time as we could use the existing GitLab one. No timelines yet as we first focus on other parts.
We are going to move away from supporting Cortex. We will be finding another Prometheus API and we are going to use Clickhouse as our metrics storage backend.
We are mapping to the GitLab ones.
We are going to archive the name "Opstrace" as a standalone product, and instead focus on Observability features on the GitLab DevOps platform. Timeline and work that needs to be done are tracked in this issue.