Gitlab hero border pattern left svg Gitlab hero border pattern right svg

Category Direction - Continuous Integration Scaling

Continuous Integration Scaling

In FY22, we have committed to SaaS First. In the Verify Stage, this means prioritizing the scale of Continuous Integration and ensuring our users on GitLab.com are leveraging a reliable and available service. We are focused on a goal of 20M builds per day as it represents not only a target to drive our future architecture, but a volume that we expect to achieve within several quarters.

Investment Allocation

Currently, we are staffing this Category with 10% of the Verify output, or equivalent to ~3 engineers in Verify. After we establish the load testing effort, we will evaluate if additional headcount will be required to support this effort.

We are expecting to on-ramp additional headcount in FY22-Q3 per product#2178, raising the percentage to ~24% of the Verify headcount.

Additional Resources

For specific information and features related to authoring/defining pipelines, check out Pipeline Authoring. You may also be looking for one of the following related product direction pages: GitLab Runner.

What's Next & Why

Now that we have delivered the Next CI/CD scale target: 20M builds per day by 2024 blueprint and related scope on queuing mechanisms via gitlab&5909, primary key capacity via gitlab#325618, and handling large amounts of data via [gitlab&6009](https://gitlab.com/groups/gitlab-org/-/epics/6009, and have delivered our CICD Time Decay blueprint, we are ready to begin Phase II of CI Scaling: implementing CICD Time Decay.

The first track of effort will be focused on partitioning CI tables via gitlab&5417 and the second track is to partition queuing tables via gitlab#347027.

Roadmap View

Queue Improvements PI

As the new queuing mechanism rolls out, performance changes will be visible in the [Duration of the builds queue retrieval using the big query SQL](https://dashboards.gitlab.net/d/stage-groups-pipeline_execution/stage-groups-group-dashboard-verify-pipeline-execution?

Vision Items

Our top vision items we have defined include:

  1. Improve GitLab CI/CD data model
  2. Improve Runner job queuing
  3. GitLab Runner Autoscaler architecture
  4. Create the capability to load testing the GitLab CI system
Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license