Monitoring of GitLab.com

1. You are here:
2. Handbook
3. Engineering
4. Monitoring of GitLab.com

On this page

• Other Related Pages
• Monitoring
  • Pingdom Statistics
  • Main Monitoring Dashboards
    • Prometheus
    • Public Monitoring Infrastructure
    • Private Monitoring Infrastructure
    • Adding Dashboards
  • Selection of Useful Dashboards from the Monitoring
    • Blackbox Monitoring
    • Public Whitebox Monitoring
    • Private Whitebox Monitor
• Logs
  • Adding dashboards
• GitLab Profiler
• Instrumenting Ruby to Monitor Performance

Other Related Pages

• Production Architecture
• GitLab.com Settings
• GitLab Performance Monitoring Documentation
• Performance of the Application

Monitoring

Pingdom Statistics

For a quick view of the availability and performance history of GitLab.com, we use https://stats.pingdom.com. Specifically, this has the availability and latency of reaching

• a GitLab.com issue. For reference, it is the first gitlab-ce issue.
• GitLab.com "plain and simple" called the GitLab public check.

Main Monitoring Dashboards

We collect data using InfluxDB and Prometheus, leveraging available exporters like the node or the postgresql exporters, and we build whatever else is necessary. The data is visualized in graphs and dashboards that are built using Grafana. There are two interfaces to track this, as described in more detail below.

Prometheus

We have 3 prometheus clusters: main prometheus, prometheus-db, and prometheus-app. They provide an interface to query metrics using PromQL. Each prometheus cluster collects a set of related metrics:

• main prometheus: It collects various metrics such as consul, haproxy, etc.
• prometheus-db: It collects Postgresql database related metrics such as pg-bouncer exporter and postgres-exporter.
• prometheus-app: It collects application related metrics.

Public Monitoring Infrastructure

• No authentication is required
• Automatically syncs from the private monitoring infrastructure on every chef client execution. Don't change dashboards here, they will be overwritten.
• Refer to this interface by default; only use the private one for those cases where the public dashboard is not available.
• Data gathered in InfluxDB is not currently available in public, since InfluxDB does not scrub access tokens from URLs that are measured.

Private Monitoring Infrastructure

• Private GitLab account is required to access
• Highly Available setup
• Alerting feeds from this setup
• All dashboards here also display on the public monitoring infrastructure.
• Separated from the public for security and availability reasons, they should have exactly the same graphs after we deprecate InfluxDB.

Adding Dashboards

To learn how to set up a new graph or dashboard using Grafana, take a look at the following resources

• Guide to setting up Grafana dashboards by Grafana
• YouTube video showing how to set up a dashboard
• The Grafana repo where we keep an archive of InfluxDB dashboards created in Grafana. Use these to see details in the file structure, but note that the repo is truly an archive (nothing populates from it) and can be out of date.
• Need access to add a dashboard? Ask any team lead within the infrastructure team.

Selection of Useful Dashboards from the Monitoring

Blackbox Monitoring

• GitLab Web Status: front end perspective of GitLab. Useful to understand how GitLab.com looks from the user perspective. Use this graph to quickly troubleshoot what part of GitLab is slow.
• GitLab Git Status: front end perspective of GitLab ssh access.

Public Whitebox Monitoring

• Fleet overview: useful to see the fleet status from the inside of GitLab.com. Use this graph to quickly see if the workers or the database are under heavy load, and to check load balancer bandwidth.
• PostgresSQL Overview: useful to understand how is the database behaving in depth. Use this graph to review if we have spikes of exclusive locks, active or idle in transaction processes
• PostgresSQL Queries use this dashboard to understand if we have blocked or slow queries, dead tuples, etc.
• Storage Stats use this dashboard to understand storage use and performance.

Private Whitebox Monitor

• Host Stats: useful to dive deep into a specific host to understand what is going on with it. Select a host from the dropdown on the top.
• Business Stats: shows many pushes, new repos and CI builds.
• Daily overview: shows endpoints with amount of calls and performance metrics. Useful to understand what is slow generally.

Logs

Network, System, and Application logs are processed, stored, and searched using the ELK stack. For monitoring system performance and metrics Grafana is still the preferred interface. However, for investigating errors and incidents raw logs are available via Kibana.

Kibana dashboards are used to monitor application activity, spam events, transient errors, system and network authentication events, security events, etc. Commonly used dashboards are the Abuse, SSH, and Rack Attack dashboards.

One can view how we log our infrastructure as outlined by our runbook

Adding dashboards

To learn how to create Kibana dashboards use the following resources:

• Kibana Dashboard tutorial from Elastic.co
• Building a dashboard
• Using TimeLion for time series visualization

GitLab Profiler

GitLab profiler data, accessible using your GitLab Google credentials, is a dashboard with links to request profiles and SQL queries run when loading pages on GitLab.com.

To add a page to this dashboard, create a merge request to the gitlab-com/gitlab-profiler project.

Instrumenting Ruby to Monitor Performance

Blocks of Ruby code can be "instrumented" to measure performance.

• Documentation of instrumentation with more detail on how to implement this
• An example of how this is used for GitLab itself, can be found in this initializer.