The following page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features or functionality remain at the sole discretion of GitLab Inc.
This is the product direction for Monitor:Observability. If you'd like to discuss this direction directly with the product managers for Monitor:Observability, feel free to reach out to Kevin Chu (GitLab, Email).
Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. In the (recent) past, simply monitoring your application and related systems may have been sufficient, e.g. watching for specific metrics to track whether performance and health are within expectations. Today, with the rapid adoption of cloud-native technologies and distributed systems, it is no longer sufficient to only look for known indicators. An observable system is one that enables operators to efficiently drill in and understand what went wrong even for previously unencountered problems.
Make data manageable, understandable, and feedback-ready for every stage of the DevOps process.
GitLab will empower users with continuous insight into the state and behavior of their software systems and software delivering system by leveraging ubiquitous telemetry data in a unique fashion–not just in production, but across every stage of the DevOps process.
On Dec 14, 2021, GitLab acquired Opstrace to jumpstart the plan to offer an observability solution in the One DevOps platform.
Opstrace added two foundational capabilities to the GitLab platform: the GitLab Observability Backend for data management and the GitLab Observability UI for data visualization. We plan to continue to invest in and mature these two areas. They enable the table stakes in any observability solution by enabling users to manage data all in one place and to access and visualize data through a single interface.
Beyond building on the foundation, we plan to lean into well-used areas of GitLab to bring users into GitLab Observability. These capabilities are our key differentiators and the key to our Go-To-Market strategy.
The table below is inspired by the Strategy/Metric/Tactic Lockup
| Strategy | Metric | Tactic / Project |
|---|---|---|
| Data all in one place | Percent of customers with two different data types | Integration of Observability Backend |
| Data is accessible from a single interface | Percent of customers who creates a dashboard | Integration of Observability UI |
| Deep GitLab integration | MAU of features enabled by Observability | GTM JTBD |
We plan to work toward our vision with the following principles:
We want developers to have immediate access to an available observability solution. That means we will ensure it is on by default, starts with convention over configuration, and is easy to use.
Data is most valuable when it is easy to glean insights across data types. As a result, we're working to store all types of data in a unified datastore - ClickHouse from metrics, logs, traces, and errors, to analytics data and other data types across the entire DevOps process.
We plan to create an integrated UI that enables visualization and collaboration of data directly within GitLab.
There is a vibrant open-source community working on observability and related toolings. The GitLab observability solution is built on top of the shoulder of the giants in this space. We intend to continue to leverage, augment, and work with what people already use and love. For example, we intend to adopt OpenTelemetry as the primary way to instrument, generate, collect, and export telemetry data. We also rely on Prometheus APIs and metric labeling as conventions.
We plan to follow our FY23 investment theme of GitLab Hosted First in bringing our observability solution first as a SaaS solution. We plan to offer the same solution, specifically designed to ease the toil of operation, to our self-managed customers. As we mature the product, maintainability and operability will continue to be built into the product based on our learnings operating GitLab SaaS.
Organizations have the choice of using an observability vendor or building their own observability platform. With the former, teams outsource the problems of operating a system that needs to be scaled and consistently updated but are locked into proprietary software and APIs. With the latter, teams not only have to manage the complexity of operating an observability platform at scale, but they have to also figure out how to make the components work together, and keep them up to date. We want that choice to be easy to make by offering a completely open, yet managed platform.
With each GitLab release you can confidently upgrade and maintain your open-source observability tooling. Furthermore, this observability solution is integrated with the rest of GitLab so you have a single tool to build, test, collaborate, deploy, and monitor your applications.
The decisions we made in architecting GitLab observability also enable a level of flexibility that other solutions don't have. GitLab customers can decide how they want to operate to maintain data ownership and governance. We plan to enable the following capabilities in the future:
We have an ambitious vision, but we're not going to get there overnight. As outlined in our strategy, we intend to leverage GitLab's strength as a comprehensive DevOps platform that our users already trust and use to create, build, test, and deploy software, to bring users into our observability solution. The identified jobs to be done are outlined below:
GitLab previously relied on the wrong tool (relational database) to store errors. The spiky nature of errors caused unfortunate outages to our service. We are working to move errors to the GitLab Observability Platform, and create dedicated visualization in the GitLab Observability UI. We plan to use GitLab error tracking internally and make it available to our users and customers.
One of the issues today with the GitLab CI/CD is there is no out-of-the-box solution to visualize where bottlenecks are in the pipeline. Inspired by the OpenTelemetry Jenkins plugin we should enable runners to instrument pipeline jobs using OpenTelemetry, provide traces to the GitLab Observability Platform, and help users understand what is happening in the GitLab Observability UI. Given that GitLab CI/CD is one of the most used features, we believe we can drive observability usage by providing a solution for this job.
Focusing on this job, we will also architect for the generic tracing use case. We want to make it easy for our users and customers to port over existing tracing jobs. We also want to make it easier for all GitLab users to start gaining the benefits of tracing.
Kubernetes is the operating platform or the future platform for many organizations. As such, we want to lower the cost to operate in Kubernetes via GitLab. We can create a solution for the job by creating a Kubernetes Dashboard. By building the Kubernetes Dashboard on top of GitLab observability, we can help the Configure and Release team to have less code to maintain, and build generic plumbing for generic data that changes frequently that should be displayed in dashboards.
Details TBD
Currently, Error Tracking and tracing is available in Open Beta.
For Error Tracking, we've replaced the PostgreSQL-based error backend with the GitLab Observability Backend to enable integrated error tracking in GitLab. With the capability, users can send errors to us via the Sentry SDK and users can use the legacy Error UI in GitLab Rails to interact with Errors stored in the GitLab Observability Backend. Because it is in Open Beta, error tracking is not ready to use for business-critical applications in production yet.
For tracing, users can send traces to GitLab using OpenTelemetry or Jaeger. However, there is no way to interface with the data directly within GitLab yet.
We plan to complete an end-to-end flow, with error tracking as the first completed use case.
To start, we plan to complete the integration of the GitLab Observability UI into GitLab. We will also enable users to add any Sentry instance as a data source. This way, users will have access to the visualization tool within GitLab, be able to connect to external data sources, and can have a way to interact with both errors and traces directly within GitLab.
Simultaneously, we need to migrate the GitLab Observability infrastructure to AWS. More details are in the epic.
Next, we will build out an end-to-end experience for error tracking using both the GitLab Observability Backend and GitLab Observability UI. This will establish GOUI as the single UI for users to interface with observability UI, and we will subsequently remove the legacy errors UI in GitLab rails.
Planning happens in the planning board where issues are arranged by release. Over time this will become more populated with a further look into the upcoming releases.
There are located in this project.
June 2022 Ops Engineering Showcase (AMER/EMEA)