Capacity Planning for GitLab.com

1. You are here:
2. Engineering
3. Infrastructure
4. Capacity Planning for GitLab.com

Maintained by:

S

On this page

• Introduction
• Tools
  • Source Data
  • Forecasting with Tamland
• Workflow
  • Reviewing Tamland Data
  • Due Dates
  • Workflow Status Labels
  • Saturation Labels
  • Capacity Planning is a shared activity
  • Prioritization Framework
  • Relaying Capacity Incidents to Engineering
    • Identifying the DRI for a Capacity Issue
• Examples of Capacity Issues
  • redis-cache / redis_primary_cpu potential saturation
    • Timeline
  • monitoring / go_memory potential saturation
  • redis-cache / redis_memory potential saturation
  • redis-cache / node_schedstat_waiting potential saturation
  • git / kube_pool_max_nodes potential saturation

Introduction

GitLab.com's capacity planning is based on a forecasting model which is populated with the same saturation and utilization data that is used for short-term monitoring of GitLab.com.

The forecasting tool generates warnings which are converted to issues and these issues are raised in various status meetings.

Tools

At present, we use Facebook's Prophet library for forecasting. The model is used to generate a report, which is published weekly to https://gitlab-com.gitlab.io/gl-infra/tamland.

GitLab's Capacity Planning strategy is based on the following technologies:

1. Saturation Monitoring Jsonnet Configuration - for saturation monitoring definition, recording rule generation, short term alerting configuration generation.
2. Prometheus - capturing and processing utilization and saturation metrics over the short-term.
3. Thanos - long-term storage of utilization and saturation metrics.
4. Tamland - the forecasting program.
5. GitLab CI - running weekly forecast reports.
6. Jupyter Book - static site generation for generating forecast content from Python based notebooks.
7. Facebook Prophet - forecasting.
8. GitLab Pages Tamland Report Site - static site hosting of the generated forecasts.
9. Prometheus Pushgateway - accepting key metrics from Tamland forecasts.
10. GitLab Capacity Planning Issue Tracker - GitLab project used for tracking capacity planning forecast warnings. Issues are created by Tamland directly.
11. GitLab Slack Integration - Slack notifications of new issues in the capacity planning project, to the #infra_capacity-planning channel.

Source Data

The forecasting model uses the same saturation and utilization data model that we use to monitor GitLab.com over the short-term. This ensures that anything that we feel is worth monitoring as a potential saturation point will automatically be included in the forecasting model.

Because of this, all services used on GitLab.com are automatically included in the model.

The short-term saturation metric model used on GitLab.com models each resource as a percentage, from 0% to 100%, where 100% is completely saturated. Each resource has an alerting threshold. If this threshold is breached, alerts will fire and the engineer-on-call will be paged.

The thresholds are decided on a case-by-case basis and vary between resources. Some are near 100% while others are much lower, depending on the nature of the resource, the failure modes on saturation of the resource and the required time-to-mediation. Resources are classed as being either horizontally scalable or not. Horizontally scalable resources are generally considered lower priorities from a capacity planning point-of-view, whereas non-horizontally scalable resources (such as CPU on the primary Postgres instance, for example) require much longer-term strategies for remediation and are therefore considered higher priorities in the capacity planning process.

Forecasting with Tamland

Tamland relies on Facebook Prophet for generating a forecasting model. Prophet performs analysis of hourly, daily, weekly and monthly trends to forecast a future trend in the data.

Even the most skilled engineer would struggle to predict future saturation, so it's unlikely that a model could do it either. We do not expect it to be totally accurate. Instead, with hundreds of resources on GitLab.com that could potentially become saturated, Tamland's forecasts are a bellweather for changes in trends, particularly upward changes, drawing the attention of engineers who review the data to specific issues.

Tamland will attempt to predict a range of outcomes. For saturation, we focus on the median prediction (50th percentile) and only the upper 80th percentile prediction. The lower 80th percentile is not as important for saturation purposes.

The forecast process, Tamland, runs as a GitLab CI job on ops.gitlab.net. This job will run on a schedule defined in the scheduled pipeline (set to weekly). The process starts by reading the historical short-term saturation metric data from Thanos, up to 1-year period, using an hourly resolution.

Workflow

Reviewing Tamland Data

1. The Tamland report is generated.
2. If Tamland predicts that a resource will exceed its alerting threshold (as defined in the metrics catalog) within the next 90 days, it creates an issue.
3. On a weekly basis, an engineer reviews all open issues in the Capacity Planning tracker following the process described on the Scalability:Projections team page.
   1. Not all forecasts are always accurate: a sudden upward trend in the resource saturation metric may be caused by a factor that is known to be temporary - for example, a long running migration. The engineer will evaluate based on all information on-hand and determine whether the forecast is accurate and if the issue requires investigation.
4. If the engineer observes an unexplained trend they will note down their findings and find an appropriate DRI to investigate further and remediate. We use the Infradev Process and the Engineering Allocation Process to assist with the prioritization of these capacity alerts.
   1. These issues can be the catalyst for other issues to be created in the 'gitlab-org/gitlab' tracker by the stage groups for further investigation. These issues must be connected to the capacity planning issues as related issues.
5. Any issue concerning resource saturation or capacity planning in any tracker should have the "GitLab.com Resource Saturation' label applied.

Due Dates

We use the due date field to track when the next action is due. For example: the date we expect the issue to drop off the report, or the date we expect the DRI to have taken action. We do this because we want to use the capacity planning issue board as the single source of truth. The due date is visible on this board and it is easy to see which issues need attention.

The DRI is responsible for maintaining the due date and adding status information each time the due date is adjusted.

Workflow Status Labels

Capacity Planning issues are created without a state. After the initial assessment, one of the following labels should be applied.

1. capacity-planning::investigate - this alert requires further active assessment before deciding on a course of action
2. capacity-planning::monitor - we need to wait for time to pass to gather further data on this issue to make a decision on how to proceed
3. capacity-planning::in-progress - there is a mitigation in progress for this alert
4. capacity-planning::verification - we have completed work on this issue and are verifying the result

Saturation Labels

Each issue has saturation labels, indicating which thresholds it exceeds and how. An issue can have multiple saturation labels; for instance, any issue with saturation will, by definition, also have the other three.

1. saturation - this issue is predicted (by the median line) to reach 100% saturation in the next 90 days.
2. violation - this issue is predicted (by the median line) to reach the saturation threshold (which varies by component) in the next 90 days.
3. saturation-80%-confidence - this issue is predicted (by the upper end of the 80% confidence interval) to reach 100% saturation in the next 90 days.
4. violation-80%-confidence - this issue is predicted (by the upper end of the 80% confidence interval) to reach the saturation threshold (which varies by component) in the next 90 days.

Capacity Planning is a shared activity

The Scalability:Projections team owns the Capacity Planning process and we aim to enable others to take responsibility for the capacity demands of their features and services.

Prioritization Framework

The Scalability:Frameworks team uses capacity planning issues to drive prioritization. By taking saturation data as an input into the planning process, Frameworks team can identity potential projects to balance proactive and reactive work streams.

The prioritization framework uses an Eisenhower Matrix, a 2x2 matrix based on urgency and importance:

Urgent is based on forecast threshold (e.g. 100% saturation vs. hard SLO violation) and important is based on scalable resources (e.g. non_horizontal vs. horizontal). The following resources are available for prioritization:

• Quadrant board
• Issues sorted by priority
• Scoped prioritized labels

Relaying Capacity Incidents to Engineering

The forecasts are reviewed in the weekly Engineering Allocation meeting and any required corrective actions are prioritized according to the timeframes for saturation predicted by the forecast, and the criticality of the resources.

Practically, this is done by:

1. From the previous week's agenda, review all new and carry-on items listed.
   1. For each item, confirm the latest status with the DRI and update the agenda to match.
   2. If there is not movement on an issue, confirm the urgency of the problem and raise awareness in the meeting. Another approach is to ask the DRI for when the next update will be available.
2. In the current week's agenda, add all newly created issues with the purpose of raising awareness and finding a DRI.

Actions described above can also take place asynchronously at any time - we should not wait for the Engineering Allocation meeting to update issue status or find DRIs for issues.

The Scalability:Projections team will triage the capacity alerts by labeling them with the relevant severity/priority labels and assign them to the appropriate owner. We rely on the Infradev Process to assist with prioritization of these capacity issues. We remain available for guidance and review support.

Identifying the DRI for a Capacity Issue

There are three scenarios that can occur:

1. Wholly-owned component: An alert fires for a component that is owned by a specific group. Projections will triage the issue, and assign the alert to the Engineering Manager of that group. Tracking of the issue forms part of the Infradev Process. (Where Infrastructure owns the issue, the motivation for addressing the problem comes from wanting to prevent alerts to the on-call engineers).
2. Shared-ownership but clear cause: An alert fires for a general component. During triage, the Projections team can identify a single cause and a clear owner. Projections will triage the issue and assign the alert to the relevant Engineering Manager. Tracking of the issue forms part of the Infradev Process.
3. Shared-ownership, alert caused by multiple factors: The Projections team will remain the DRI for these issues and may also own the remediation tasks unless a specific DRI can be identified during the investigation.

The severity and priority assigned to these Infradev issues will be based on the alert information. If the alert continues to fire, the severity and priority will be raised appropriately.

When the DRI has been identified, the issue will be assigned to them and a comment will be added to describe how we identified them as the owner as well as what is expected from them to resolve the issue. The DRI should keep the issue updated by maintaining the due date and adding status information each time the due date is adjusted.

If the issue belongs to a specific team, the team label will also be applied.

Examples of Capacity Issues

In this section, we discuss a few capacity planning issues and describe how we applied the process above when addressing them.

redis-cache / redis_primary_cpu potential saturation

gitlab-com/gl-infra/capacity-planning#364

We split out some operations from redis-cache to a new redis-repository-cache instance to reduce our CPU utilisation on redis-cache. We had planned this for weeks in advance due to a capacity planning warning, and we were able to roll this out the day after we had a production page about CPU saturation for redis-cache.

If we hadn't had the capacity planning step in there, we may have noticed this problem much later, and had to scramble to implement mitigations in a high-pressure environment. Instead, we just accelerated our existing timeline slightly, and resolved it with a clean solution.

Timeline

1. 2022-09-21: our capacity planning process creates redis-cache / redis_primary_cpu potential saturation to warn about potential future saturation. At this time we noted that it was a risk, but not guaranteed to saturate in the next three months.
2. 2022-10-20: we create Introduce Redis Cluster for redis-ratelimiting (leading to Horizontally Scale redis-cache using Redis Cluster), representing the long-term solution to this.
3. 2022-12-05: having analyzed the workload for Redis Cache, we decide to work on Functional Partitioning for Repository Cache to mitigate Redis saturation forecasts. This is because Introduce Redis Cluster for redis-ratelimiting will take longer to deliver, and we might not be able to deliver it before the forecast saturation date.
4. 2022-12-06: we create Migrate repository cache from redis-cache to new shard on VMs to capture the core work in moving data from one Redis instance to another.
5. 2022-12-14: the new instance is available for testing in our pre-production environment.
6. 2023-01-11: we confirm that the current capacity forecast does not predict saturation, but we expect this is due to the end-of-year holidays.
7. 2023-01-27: we roll out the new redis-repository-cache on the pre and gstg environments in gitlab-com/gl-infra/scalability#2050 and gitlab-com/gl-infra/scalability#2052.
8. 2023-01-31: we have a production incident warning of imminent redis-cache CPU saturation in gitlab-com/gl-infra/production#8335.
9. 2023-01-31: we bring in the work to roll out to gprd as a result, from gitlab-com/gl-infra/production#8309. This was intended to happen the following week due to staff availability.
10. 2023-02-10: we have gone from CPU utilisation above 90% to below 70%, and the capacity planning issue (redis-cache / redis_primary_cpu potential saturation) issue automatically closes once the data updates for this partitioning.

monitoring / go_memory potential saturation

gitlab-com/gl-infra/capacity-planning#42

• The Tamland report showed that this component would likely saturate within the next 30 days.
• The engineer reviewing the issue saw that the trend lines indicated a problem.
• The engineer contacted the Engineering Manager for the team responsible for this component.
• The responsible team worked to correct the problem.
• When the team was satisfied with their changes, we confirmed that this component was no longer showing in the report.
• We also confirmed through source metrics that this component was no longer likely to saturate.

redis-cache / redis_memory potential saturation

gitlab-com/gl-infra/capacity-planning#45

• The Tamland report showed that this component might saturate in the next few months.
• The engineer reviewing the issue determined that this saturation point was an artificial limit. It is expected for this component to hover around its maximum without causing problems.
• The team worked to exclude these components from the Tamland process.
• The issue was resolved.

redis-cache / node_schedstat_waiting potential saturation

gitlab-com/gl-infra/capacity-planning#144

• The Tamland report showed potential saturation.
• The engineer reviewing this problem could see that outliers in the data (due to an incident) impacted the forecast.
• The engineer silenced the alerts and explained why this issue could be closed.

git / kube_pool_max_nodes potential saturation

gitlab-com/gl-infra/capacity-planning#31 and gitlab-com/gl-infra/capacity-planning#108

• The Tamland report showed potential saturation.
• The responsible team was contacted and they made changes to address the problem.
• They believed they had done enough to prevent the saturation from occuring so they closed the issue.
• When Tamland produced its next report, the item was still included.
• The responsible team picked up the issue again and found that the metrics reporting the problem had been broken with the previous change.
• The team confirmed that the saturation problem was definitely fixed and corrected the metrics to reflect the change.
• This example shows that Tamland will continue to notify us of a capacity issue until the metrics show that it is resolved.