Monitoring of GitLab.com

Monitoring

Pingdom Statistics

For a quick view of the availability and performance history of GitLab.com, we use https://stats.pingdom.com. Specifically, this has the availability and latency of reaching

a GitLab.com issue. For reference, it is the first gitlab-ce issue.
a GitHub.com issue. For reference, it is the first issue in the gitlabhq project.
GitLab.com "plain and simple" called the GitLab public check.

Main Monitoring Dashboards

We collect data using InfluxDB and Prometheus, leveraging available exporters like the node or the postgresql exporters, and we build whatever else is necessary. The data is visualized in graphs and dashboards that are built using Grafana. There are two interfaces to track this, as described in more detail below.

Public monitoring infrastructure

No authentication is required
Automatically syncs from the private monitoring infrastructure on every chef client execution. Don't change dashboards here, they will be overwritten.
Refer to this interface by default; only use the private one for those cases where the public dashboard is not available.
Data gathered in InfluxDB is not currently available in public, since InfluxDB does not scrub access tokens from URLs that are measured.

Private monitoring infrastructure

Private GitLab account is required to access
Highly Available setup
Alerting feeds from this setup
All dashboards here also display on the public monitoring infrastructure.
Separated from the public for security and availability reasons, they should have exactly the same graphs after we deprecate InfluxDB.

Adding Dashboards

To learn how to setup a new graph or dashboard using Grafana, take a look at the following resources

Guide to setting up Grafana dashboards by Grafana
YouTube video showing how to set up a dashboard
The Grafana repo where we keep an archive of InfluxDB dashboards created in Grafana. Use these to see details in the file structure, but note that the repo is truly an archive (nothing populates from it) and can be out of date.
Need access to add a dashboard? Ask any team lead within the infrastructure team.

Selection of Useful Dashboards from the Monitoring

Blackbox Monitoring

GitLab Web Status: front end perspective of GitLab. Useful to understand how GitLab.com looks from the user perspective. Use this graph to quickly troubleshoot what part of GitLab is slow.
GitLab Git Status: front end perspective of GitLab ssh access.

Public Whitebox Monitoring

Fleet overview: useful to see the fleet status from the inside of GitLab.com. Use this graph to quickly see if the workers or the database are under heavy load, and to check load balancer bandwidth.
Postgres Stats: useful to understand how is the database behaving in depth. Use this graph to review if we have spikes of exclusive locks, active or idle in transaction processes
Postgres Queries use this dashboard to understand if we have blocked or slow queries, dead tuples, etc.
Storage Stats use this dashboard to understand storage use and performance.

Private Whitebox Monitor

Host Stats: useful to dive deep into a specific host to understand what is going on with it. Select a host from the dropdown on the top.
Business Stats: shows many pushes, new repos and CI builds.
Daily overview: shows endpoints with amount of calls and performance metrics. Useful to understand what is slow generally.

GitLab Profiler

GitLab profiler data is a dashboard with links to request profiles and SQL queries run when loading pages on GitLab.com.

To add a page to this dashboard, create a merge request to the gitlab-com/gitlab-profiler project.

Instrumenting Ruby to monitor performance

Blocks of Ruby code can be "instrumented" to measure performance.

Documentation of instrumentation with more detail on how to implement this
An example of how this is used for GitLab itself, can be found in this initializer.